Which prediction method is best for making

The representation for LVQ is a collection of codebook vectors. These are selected randomly in the beginning and adapted to best summarize the training dataset over a number of iterations of the learning algorithm. After learned, the codebook vectors can be used to make predictions just like K-Nearest Neighbors.

The most similar neighbor best matching codebook vector is found by calculating the distance between each codebook vector and the new data instance. The class value or real value in the case of regression for the best matching unit is then returned as the prediction.

Best results are achieved if you rescale your data to have the same range, such as between 0 and 1. If you discover that KNN gives good results on your dataset try using LVQ to reduce the memory requirements of storing the entire training dataset. Support Vector Machines are perhaps one of the most popular and talked about machine learning algorithms.

A hyperplane is a line that splits the input variable space. In SVM, a hyperplane is selected to best separate the points in the input variable space by their class, either class 0 or class 1. The SVM learning algorithm finds the coefficients that results in the best separation of the classes by the hyperplane. The distance between the hyperplane and the closest data points is referred to as the margin.

The best or optimal hyperplane that can separate the two classes is the line that has the largest margin. Only these points are relevant in defining the hyperplane and in the construction of the classifier. These points are called the support vectors. They support or define the hyperplane. In practice, an optimization algorithm is used to find the values for the coefficients that maximizes the margin. SVM might be one of the most powerful out-of-the-box classifiers and worth trying on your dataset.

Random Forest is one of the most popular and most powerful machine learning algorithms. It is a type of ensemble machine learning algorithm called Bootstrap Aggregation or bagging. The bootstrap is a powerful statistical method for estimating a quantity from a data sample.

Such as a mean. You take lots of samples of your data, calculate the mean, then average all of your mean values to give you a better estimation of the true mean value. In bagging, the same approach is used, but instead for estimating entire statistical models, most commonly decision trees.

Multiple samples of your training data are taken then models are constructed for each data sample. When you need to make a prediction for new data, each model makes a prediction and the predictions are averaged to give a better estimate of the true output value.

Random forest is a tweak on this approach where decision trees are created so that rather than selecting optimal split points, suboptimal splits are made by introducing randomness. The models created for each sample of the data are therefore more different than they otherwise would be, but still accurate in their unique and different ways. Combining their predictions results in a better estimate of the true underlying output value.

If you get good results with an algorithm with high variance like decision trees , you can often get better results by bagging that algorithm. Boosting is an ensemble technique that attempts to create a strong classifier from a number of weak classifiers. This is done by building a model from the training data, then creating a second model that attempts to correct the errors from the first model.

Models are added until the training set is predicted perfectly or a maximum number of models are added. AdaBoost was the first really successful boosting algorithm developed for binary classification.

It is the best starting point for understanding boosting. Modern boosting methods build on AdaBoost, most notably stochastic gradient boosting machines. AdaBoost is used with short decision trees. After the first tree is created, the performance of the tree on each training instance is used to weight how much attention the next tree that is created should pay attention to each training instance. Training data that is hard to predict is given more weight, whereas easy to predict instances are given less weight.

This technique requires considerably more computer time for each item and, at the present time, human attention as well. Until computational shortcuts can be developed, it will have limited use in the production and inventory control area.

However, the Box-Jenkins has one very important feature not existing in the other statistical techniques: the ability to incorporate special information for example, price changes and economic data into the forecast. The reason the Box-Jenkins and the X are more costly than other statistical techniques is that the user must select a particular version of the technique, or must estimate optimal values for the various parameters in the models, or must do both.

For example, the type and length of moving average used is determined by the variability and other characteristics of the data at hand. We expect that better computer methods will be developed in the near future to significantly reduce these costs.

In some instances where statistical methods do not provide acceptable accuracy for individual items, one can obtain the desired accuracy by grouping items together, where this reduces the relative amount of randomness in the data. Forecasters commonly use this approach to get acceptable accuracy in situations where it is virtually impossible to obtain accurate forecasts for individual items.

Also, it is sometimes possible to accurately forecast long-term demands, even though the short-term swings may be so chaotic that they cannot be accurately forecasted. We found this to be the case in forecasting individual items in the line of color TV bulbs, where demands on CGW fluctuate widely with customer schedules.

In this case, there is considerable difficulty in achieving desired profit levels if short-term scheduling does not take long-term objectives into consideration. For this reason, and because the low-cost forecasting techniques such as exponential smoothing and adaptive forecasting do not permit the incorporation of special information, it is advantageous to also use a more sophisticated technique such as the X for groups of items.

This technique is applied to analyze and forecast rates for total businesses, and also to identify any peculiarities and sudden changes in trends or patterns. This information is then incorporated into the item forecasts, with adjustments to the smoothing mechanisms, seasonals, and the like as necessary.

Frequently one must develop a manual-override feature, which allows adjustments based on human judgment, in circumstances as fluid as these. Granting the applicability of the techniques, we must go on to explain how the forecaster identifies precisely what is happening when sales fluctuate from one period to the next and how such fluctuations can be forecast.

A trend and a seasonal are obviously two quite different things, and they must be handled separately in forecasting. Consider what would happen, for example, if a forecaster were merely to take an average of the most recent data points along a curve, combine this with other, similar average points stretching backward into the immediate past, and use these as the basis for a projection.

The forecaster might easily overreact to random changes, mistaking them for evidence of a prevailing trend, mistake a change in the growth rate for a seasonal, and so on. To avoid precisely this sort of error, the moving average technique, which is similar to the hypothetical one just described, uses data points in such a way that the effects of seasonals and irregularities are eliminated.

Furthermore, the executive needs accurate estimates of trends and accurate estimates of seasonality to plan broad-load production, to determine marketing efforts and allocations, and to maintain proper inventories—that is, inventories that are adequate to customer demand but are not excessively costly. Before going any further, it might be well to illustrate what such sorting-out looks like.

Part A presents the raw data curve. Part B shows the seasonal factors that are implicit in the raw data—quite a consistent pattern, although there is some variation from year to year. In the next section we shall explain where this graph of the seasonals comes from.

Part C shows the result of discounting the raw data curve by the seasonals of Part B; this is the so-called deseasonalized data curve. We might further note that the differences between this trend-cycle line and the deseasonalized data curve represent the irregular or nonsystematic component that the forecaster must always tolerate and attempt to explain by other methods. In sum, then, the objective of the forecasting technique used here is to do the best possible job of sorting out trends and seasonalities.

Unfortunately, most forecasting methods project by a smoothing process analogous to that of the moving average technique, or like that of the hypothetical technique we described at the beginning of this section, and separating trends and seasonals more precisely will require extra effort and cost. Still, sorting-out approaches have proved themselves in practice. We can best explain the reasons for their success by roughly outlining the way we construct a sales forecast on the basis of trends, seasonals, and data derived from them.

This is the method:. In special cases where there are no seasonals to be considered, of course, this process is much simplified, and fewer data and simpler techniques may be adequate. We have found that an analysis of the patterns of change in the growth rate gives us more accuracy in predicting turning points and therefore changes from positive to negative growth, and vice versa than when we use only the trend cycle.

The main advantage of considering growth change, in fact, is that it is frequently possible to predict earlier when a no-growth situation will occur. The graph of change in growth thus provides an excellent visual base for forecasting and for identifying the turning point as well.

The reader will be curious to know how one breaks the seasonals out of raw sales data and exactly how one derives the change-in-growth curve from the trend line. One of the best techniques we know for analyzing historical data in depth to determine seasonals, present sales rate, and growth is the X Census Bureau Technique, which simultaneously removes seasonals from raw information and fits a trend-cycle line to the data.

The output includes plots of the trend cycle and the growth rate, which can concurrently be received on graphic displays on a time-shared terminal. Although the X was not originally developed as a forecasting method, it does establish a base from which good forecasts can be made. One should note, however, that there is some instability in the trend line for the most recent data points, since the X, like virtually all statistical techniques, uses some form of moving average.

It has therefore proved of value to study the changes in growth pattern as each new growth point is obtained. In particular, when recent data seem to reflect sharp growth or decline in sales or any other market anomaly, the forecaster should determine whether any special events occurred during the period under consideration—promotion, strikes, changes in the economy, and so on. The X provides the basic instrumentation needed to evaluate the effects of such events.

Generally, even when growth patterns can be associated with specific events, the X technique and other statistical methods do not give good results when forecasting beyond six months, because of the uncertainty or unpredictable nature of the events.

For short-term forecasts of one to three months, the X technique has proved reasonably accurate. We have used it to provide sales estimates for each division for three periods into the future, as well as to determine changes in sales rates. The forecasts using the X technique were based on statistical methods alone, and did not consider any special information. The division forecasts had slightly less error than those provided by the X method; however, the division forecasts have been found to be slightly biased on the optimistic side, whereas those provided by the X method are unbiased.

This suggested to us that a better job of forecasting could be done by combining special knowledge, the techniques of the division, and the X method.

This is actually being done now by some of the divisions, and their forecasting accuracy has improved in consequence. The X method has also been used to make sales projections for the immediate future to serve as a standard for evaluating various marketing strategies. This has been found to be especially effective for estimating the effects of price changes and promotions. As we have indicated earlier, trend analysis is frequently used to project annual data for several years to determine what sales will be if the current trend continues.

Regression analysis and statistical forecasts are sometimes used in this way—that is, to estimate what will happen if no significant changes are made. Then, if the result is not acceptable with respect to corporate objectives, the company can change its strategy.

However, the development of such a model, usually called an econometric model, requires sufficient data so that the correct relationships can be established.

During the rapid-growth state of color TV, we recognized that economic conditions would probably effect the sales rate significantly. However, the macroanalyses of black-and-white TV data we made in for the recessions in the late s and early s did not show any substantial economic effects at all; hence we did not have sufficient data to establish good econometric relationships for a color TV model.

A later investigation did establish definite losses in color TV sales in due to economic conditions. In Corning decided that a better method than the X was definitely needed to predict turning points in retail sales for color TV six months to two years into the future. Adequate data seemed to be available to build an econometric model, and analyses were therefore begun to develop such a model for both black-and-white and color TV sales. Our knowledge of seasonals, trends, and growth for these products formed a natural base for constructing the equations of the models.

The economic inputs for the model are primarily obtained from information generated by the Wharton Econometric Model, but other sources are also utilized. Using data extending through , the model did reasonably well in predicting the downturn in the fourth quarter of and, when data were also incorporated into the model, accurately estimated the magnitude of the drop in the first two quarters of Because of lead-lag relationships and the ready availability of economic forecasts for the factors in the model, the effects of the economy on sales can be estimated for as far as two years into the future.

In the steady-state phase, production and inventory control, group-item forecasts, and long-term demand estimates are particularly important. The interested reader will find a discussion of these topics on the reverse of the gatefold. Finally, through the steady-state phase, it is useful to set up quarterly reviews where statistical tracking and warning charts and new information are brought forward.

At these meetings, the decision to revise or update a model or forecast is weighed against various costs and the amount of forecasting error. In a highly volatile area, the review should occur as frequently as every month or period. In concluding an article on forecasting, it is appropriate that we make a prediction about the techniques that will be used in the short- and long-term future. As we have already said, it is not too difficult to forecast the immediate future, since long-term trends do not change overnight.

Many of the techniques described are only in the early stages of application, but still we expect most of the techniques that will be used in the next five years to be the ones discussed here, perhaps in extended form. The costs of using these techniques will be reduced significantly; this will enhance their implementation. We expect that computer timesharing companies will offer access, at nominal cost, to input-output data banks, broken down into more business segments than are available today.

The continuing declining trend in computer cost per computation, along with computational simplifications, will make techniques such as the Box-Jenkins method economically feasible, even for some inventory-control applications. Computer software packages for the statistical techniques and some general models will also become available at a nominal cost. At the present time, most short-term forecasting uses only statistical methods, with little qualitative information.

Where qualitative information is used, it is only used in an external way and is not directly incorporated into the computational routine.

We predict a change to total forecasting systems, where several techniques are tied together, along with a systematic handling of qualitative information. Econometric models will be utilized more extensively in the next five years, with most large companies developing and refining econometric models of their major businesses. Marketing simulation models for new products will also be developed for the larger-volume products, with tracking systems for updating the models and their parameters.

Heuristic programming will provide a means of refining forecasting models. While some companies have already developed their own input-output models in tandem with the government input-output data and statistical projections, it will be another five to ten years before input-output models are effectively used by most major corporations. Within five years, however, we shall see extensive use of person-machine systems, where statistical, causal, and econometric models are programmed on computers, and people interacting frequently.

As we gain confidence in such systems, so that there is less exception reporting, human intervention will decrease. Basically, computerized models will do the sophisticated computations, and people will serve more as generators of ideas and developers of systems. For example, we will study market dynamics and establish more complex relationships between the factor being forecast and those of the forecasting system.

Further out, consumer simulation models will become commonplace. The models will predict the behavior of consumers and forecast their reactions to various marketing strategies such as pricing, promotions, new product introductions, and competitive actions.

Probabilistic models will be used frequently in the forecasting process. Finally, most computerized forecasting will relate to the analytical techniques described in this article. Computer applications will be mostly in established and stable product businesses. Although the forecasting techniques have thus far been used primarily for sales forecasting, they will be applied increasingly to forecasting margins, capital expenditures, and other important factors. This will free the forecaster to spend most of the time forecasting sales and profits of new products.

Doubtless, new analytical techniques will be developed for new-product forecasting, but there will be a continuing problem, for at least 10 to 20 years and probably much longer, in accurately forecasting various new-product factors, such as sales, profitability, and length of life cycle.

With an understanding of the basic features and limitations of the techniques, the decision maker can help the forecaster formulate the forecasting problem properly and can therefore have more confidence in the forecasts provided and use them more effectively. The forecaster, in turn, must blend the techniques with the knowledge and experience of the managers.

The need today, we believe, is not for better forecasting methods, but for better application of the techniques at hand. See Harper Q. North and Donald L. See John C. Chambers, Satinder K. Mullick, and David A. See Graham F. Stone and R. You have 1 free article s left this month. You are reading your last free article for this month. Subscribe for unlimited access. Create an account to read 2 more.

Financial analysis. How to Choose the Right Forecasting Technique. Forecasting can help them […] by John C. Mullick, and Donald D. Basic Forecasting Techniques. Inventory Control While the X method and econometric or causal models are good for forecasting aggregated sales for a number of items, it is not economically feasible to use these techniques for controlling inventories of individual items.

Some of the requirements that a forecasting technique for production and inventory control purposes must meet are these: It should not require maintenance of large histories of each item in the data bank, if this can be avoided. Computations should take as little computer time as possible. The technique should identify seasonal variations and take these into account when forecasting; also, preferably, it will compute the statistical significance of the seasonals, deleting them if they are not significant.

It should be able to fit a curve to the most recent data adequately and adapt to changes in trends and seasonals quickly. It should be applicable to data with a variety of characteristics. It also should be versatile enough so that when several hundred items or more are considered, it will do the best overall job, even though it may not do as good a job as other techniques for a particular item. Group-Item Forecasts In some instances where statistical methods do not provide acceptable accuracy for individual items, one can obtain the desired accuracy by grouping items together, where this reduces the relative amount of randomness in the data.

Long-Term Demands Also, it is sometimes possible to accurately forecast long-term demands, even though the short-term swings may be so chaotic that they cannot be accurately forecasted. Does the surrogate have similar sensitivities to model inputs? Does the model discrepancy function if there is one adequately capture uncertainty for these predictions? Should the same model discrepancy function transfer to the QOI?

Exactly how best to use multiple sources of physical data to improve the quality and accuracy of predictions is an active VVUQ research area. In cases where the validation effort will call for additional experiments, the methodologies of validation and prediction can be used to help assess the value of additional experiments and might also suggest new types of experiments to address weaknesses in the assessment.

Ideas from the design of experiments from statistics Wu and Hamada, are relevant here, but the design of validation experiments involves additional complications that make this an open research topic. The computational demands of the computational model are a complicating factor, as is the issue of dealing with model discrepancies. Also, some of the key requirements for additional experiments—such as improving the reliability of the assessment or improving communication to stakeholders or decision makers—are not easily quantified.

The experimental planning enterprise is considered from a broader perspective in Chapter 6. Of primary interest is the performance of the thermal protection system TPS , which protects the vehicle from the extreme thermal environment arising from travel through the atmosphere at speeds of Mach 20 or higher, depending on the trajectory.

Vehicles that use ablative heat shields e. TPS consumption is a critical issue in the design and operation of a reentry vehicle—if the entire heat shield is consumed, the vehicle will burn up. TPS consumption is governed by a range of physical phenomena, including high speed and turbulent fluid flow, high-temperature aero-thermo-chemistry, radiative heating, and the response of complex materials the ablator.

Thus, a numerical simulation of reentry vehicles requires models of these phenomena. The reentry vehicle simulations share a number of complicating characteristics with many other high-consequence computational science applications.

These complicating characteristics include the following:. These characteristics greatly complicate the assessment of prediction reliability and the application of VVUQ techniques. As described above, there are two components of the verification of computer simulation: 1 ensuring that the computer code used in the simulation correctly implements the intended numerical discretization of the model code verification and 2 ensuring that the errors introduced by the numerical discretization are sufficiently small solution verification.

There are many aspects of ensuring the correct implementation of a mathematical model in a computer code. Many of these are just good software engineering practices, such as exhaustive model development and user documentation, modern software design, configuration control, and continuous unit and regression testing. Commonly understood to be important but less commonly practiced, these processes are an integral part of the PECOS software environment.

To ensure that an implementation is actually producing correct solutions, one wants to compare results to known, preferably analytic, solutions. Although MMS is a widely recognized approach, it is not commonly used. One reason is that it is much more difficult to implement for complex problems than it appears. First, even for systems of moderate complexity e. Thus, constructing analytic solutions is itself a software engineering and reliability challenge.

Second, the introduction of the source terms into the code being tested must be done with minimal preferably no changes to the code, so that the tests are relevant to the code as it will be used. Unfortunately, this introduction of the source terms may not be possible in codes that have not been designed for it. Finally, it is necessary that manufactured solutions have characteristics similar to those of the problems that the codes will be used to solve.

This is important so that bugs are not masked by the fact that the terms in which they occur may be insignificant in a manufactured solution that is too simple. The theoretical convergence is second order.

These manufactured solutions have been imported into MASA. MASA and associated solutions have been publicly released. An example is shown in Figure 5.

In the initial test, the solution error did not converge to zero with uniform grid refinement, which led to the discovery of a bug in the implementation of the SA equations. When this bug was fixed, the error did reduce with refinement, but not at the theoretically expected rate of h 2. Generally, the models are used to predict certain output QOIs, and one wants to ensure that these quantities are within some tolerance of those from the exact solution of the models.

Accessed March 19, Accessed March 24, An acceptable numerical error tolerance depends on the circumstances. At the PECOS Center, since the numerical discretization errors are under the control of the analyst, the view is taken that they should be made sufficiently small to be negligible compared to other sources of uncertainty. This avoids the need to model the uncertainty arising from such errors. It is important to identify the QOIs for which predictions are being made because the numerical discretization requirements for predicting some quantities e.

Solution verification, then, requires that the discretization error in the QOIs be estimated. The common practice of comparing solutions on two grids to check how much they differ is not sufficient. In simple situations it is possible to refine the discretization uniformly e. Once one has an estimate of the errors in the QOIs, it may be necessary to refine the discretization to reduce this error.

At the PECOS Center, the simulation codes used to make predictions of the ablator consumption rate the QOI have been developed to perform adjoint-based error estimation and goal-oriented refinement. Adaptivity is used to reduced the estimated error in the QOIs to below specified tolerances, thereby accomplishing solution verification. Data and associated models of data uncertainty are critical to predictive simulation. They are needed for the calibration of physical models and inadequacy models and for the validation of these models.

At the PECOS Center, the calibration, validation, and prediction processes are closely related, interdependent, and at the heart of uncertainty quantification in computational modeling. A number of complications arise from the need to pursue validation in the context of a QOI.

First, note that in most situations the QOI in the prediction scenario is not accessible for observation, since otherwise, a prediction would generally not be needed. This inability to observe the QOI can arise for many reasons, such as legal or ethical restrictions, lack of instrumentation, limitations of laboratory facilities to reproduce the prediction scenario, cost, or that the prediction is about the future.

It is experimentally unobservable because the conditions are not accessible in the laboratory and because flight tests are expensive, making it impractical to test every trajectory of interest. Validation tests are of course posed by comparing to observations the outputs of the model for some observable quantity. The central challenge is to determine what the mismatch between observations and the model, and the relevant prediction uncertainties, imply about predictions of unobserved QOIs.

Because the QOIs cannot be observed, the only access that one has to them is through the model, and so this assessment can be done only in the context of the model. Another complication arises when the system being modeled is complicated with many parts or encompasses many interacting physical phenomena.

In this case, the validation process is commonly hierarchical, with validation tests of models for subcomponents or individual physical phenomena based on relatively simple inexpensive experiments. As an example, in the reentry vehicle problem being pursued at the PECOS Center, the individual physical phenomena include aero-chemistry, turbulence, thermal radiation, surface chemistry, and ablator material response. Combinations of subcomponents or physical phenomena are then tested against more complicated, less-abundant multiphysics experiments.

Finally, in the best circumstances, one has some experimental observations available for the complete system, allowing a validation test for the complete model. The hierarchical validation process can be envisioned as a validation pyramid shown in Figure 5. The hierarchical nature of multiphysics validation poses further challenges. First, the QOIs are generally accessible only through the model of the full system, so that single-physics models do not have access to the QOI, making QOI-aware validation difficult.

For example, in the validation of boundary-layer turbulence models for the reentry vehicle simulations pursued at the PECOS Center, the turbulent wall heat flux is identified as a surrogate QOI, since it is directly related to, and is a driver for, the ablation rate.

Multiphysics validation tests performed at higher levels of the pyramid are important because they generally test the models for the coupling between the single-physics models. But the fact that data are generally scarce at these higher levels means that these coupling models are commonly not as rigorously tested as the simpler models are, affecting the overall quality of the final prediction.

Large-scale computational models play a role in the assessment and mitigation of rare, high-consequence events. By definition, such events occur very infrequently, which means that there is little measured data from them. Thus, the issues that complicate extrapolative predictions are almost always present in predictions involving rare events. Still, computational models play a key role in safety assessments for nuclear reactors by the Nuclear Regulatory Commission Mosleh et al.

Computational models also play a role in characterizing the causes and consequences of potential natural disasters such as earthquakes, tsumanis, severe storms, avalanches, fires, or even meteor impacts. The behavior of engineered systems e. In many cases, such as probabilistic risk assessment Kumamoto and Henley, applied to nuclear reactor safety, computational models are used to evaluate the consequences of identified scenarios, helping to quantify the. This is also true of assessments of the risks from large meteor impacts, for which computer models simulate the consequences of impacts under different conditions Furnish et al.

Although it is difficult to assess confidence in such extrapolative predictions, their results can be integrated into a larger risk analysis to prioritize threats.

In such analyses, it may be a more efficient use of resources to further scrutinize the model results only for the threats with highest priority. Computational models can also be used to seek out combinations of initial conditions, forcings, and even parameter settings that give rise to extreme, or high-consequence, events. Assessing the chances of such events comes after their discovery.

Many of the methods described in Chapters 3 and 4 are relevant to this task, but now with a focus on finding aberrant behavior rather than inferring settings that match measurements. This is the opposite of designing, or engineering, a system to ensure that interactions among the various processes are minimized.

Calculating such extreme behavior may tax a model to the point that its ability to reproduce reality is questionable. Methods for assessing and improving confidence in such model predictions are challenging and largely open problems, as they are for extrapolative predictions.

Once a high-consequence event is identified, computational models can be viable tools for assessing its probability. Such events are rare, and so standard approaches such as Monte Carlo simulation are infeasible because large numbers of model runs would be required to estimate these small probabilities. There are rich lines of current research in this area.

Picard biases a particle-based code to produce more extreme events, statistically adjusting for this bias in producing estimates. In addition to response-surface approaches, one might also use a combination of high- and low-fidelity models to seek out and estimate rare-event probabilities. Another possible multifidelity strategy would be to use a low-fidelity model to seed promising boundary conditions to a high-fidelity, localized model Sain et al.

Embedding computational models in standard statistical approaches is another promising direction. For example, Cooley combines computer model output and extreme value theory from statistics to estimate the frequency of extreme rainfall events.

Bayarri et al. A better understanding of complex dynamical systems could help in the search for precursors to extreme events or important changes in system dynamics Scheffer et al. Computational models will likely have a role in such searches—even when the models are known to have shortcomings in their representation of such complex systems. Currently, computational models are being used to help inform monitoring efforts, helping to provide early warnings of events ranging from groundwater contamination to a terrorist attack.

Recent work by Lucas et al. Also, more traditional decision-theoretic approaches e. One could imagine embedding these ideas into a computational model, using a worst-case value for a reaction coefficient, a permeability field, a boundary condition, or even how a physical process is represented in the computational model. This chapter discusses numerous tasks that contribute to validation and prediction from the perspective of mathematical foundations, pointing out areas of potential fruitful research.

As noted, details of these tasks depend substantially on the features of the application—the maturity, quality, and speed of the computational model; the available physical observations; and their relation to the QOI. Some applications involve making predictions and uncertainty estimates in settings for which physical observations are plentiful. In even mildly extrapolative settings, obtaining these estimates and assessing their reliability remains an open problem.

The NRC report on the use of models in environmental regulatory decision. The findings and recommendation below relate to making extrapolative predictions. Finding: Mathematical considerations alone cannot address the appropriateness of a model prediction in a new, untested setting. Quantifying uncertainties and assessing their reliability for a prediction require both statistical and subject-matter reasoning. Finding: The idea of a domain of applicability is helpful for communicating the conditions for which predictions with uncertainty can be trusted.

However, the mathematical foundations have not been established for defining such a domain or its boundaries. Finding: Research and development on methods for assessing uncertainties of model-based predictions in new, untested conditions i. Specific needs in assessing uncertainties in prediction include:. MPS should encourage interdisciplinary interaction between domain scientists and mathematicians on the topic of uncertainty quantification, verification and validation, risk assessment, and decision making.

NSF, The above ideas are particularly relevant to the modeling of complex systems where even a slight deviation from physically tested conditions may change features of the system in many ways, some of which are incorporated in the model and some of which are not.

The field of VVUQ is still developing, making it too soon to offer any specific recommendations regarding particular methods and approaches. However, a number of principles and accompanying best practices are listed below regarding validation and prediction from the perspective of mathematical foundations.

Subject-matter expertise should inform this assessment of relevance as discussed above and in Chapter 7.

Record the justification for each assumption and omission. Finally, it is worth pointing out that there is a fairly extensive literature in statistics focused on model assessment that may be helpful if adapted to the model validation process. Basic principles such as model diagnostics Gelman et al. Biros, A. Draganescu, O. Ghattas, J. Hill, and B. Van Bloeman Waanders. Reston, Va. Anselin, L. Badri Narayanan, V.

International Journal for Numerical Methods Engineering Bangerth, W. Basil, Switzerland: Birkhauser Verlag. Bayarri, M. P Values for Composite Null Models. Journal of the American Statistical Association 95 Berger, M. Kennedy, A. Kottas, R. Paulo, J. Sacks, J. Cafeo, C. Lin, and J. Technical Report Research Triangle Park, N. C: National Institute of Statistical Sciences. Berger, R. Cafeo, J. Cavendish, C. A Framework for Validation of Computer Models. Technometrics Berger, G.

Garcia-Donato, F. Liu, J. Palomo, R. Sacks, D. Walsh, J. Cafeo, and R. Computer Model Validation with Functional Output. Annals of Statistics Journal of the American Statistical Association Berger, E.

Calder, K. Dalbey, S. Lunagomez, A. Patra, E. Pitman, E. Spiller, and R. Berger, J. Statistical Decision Theory and Bayesian Analysis. New York: Springer. The Liklihood Principle. Accessed March 20, Besag, J. Green, D. Higdon, and K. Bayesian Computation and Stochastic Systems. Statistical Science Box, G.

Empirical Model Building and Response Surfaces. New York: Wiley. Hunter, and W. New York: Wiley Online Library. Brooks, H. Doswell III. Weather Forecasting Buser, C. Kunsch, D. Luth, M. Wild, and C. Climate Dynamics 33 6 Christen, J. Journal of Computational and Graphical Statistics 14 4 Cook, R. Applied Regression Including Computing and Graphics. Cooley, D. Climatic Change 97 1 Dubois, D. Prade, and E.

New York: Plenum Press. Easterling, R. Albuquerque, N. Efendiev, Y. Datta-Gupta, X. Ma, and B. Efficient Sampling Techniques for Uncertainty Quantification.

Washington, D. Evans, S. Inverse Problems as Statistics. Inverse Problems R Evensen, G. New York: Springer Verlag. Ferson, S. Kreinovich, L. Ginzburg, D. Myers, and K. Flath, H. Filcox, V. Hill, B. Van Bloeman Waanders, and O.

Fuentes, M. Journal of the American Statistical Association, Biometrics Furnish, M. B Boslough, and G. International Journal of Impact Engineering 17 3 Galbally, D. Fidkowski, K. Willcox, and O. International Journal for Numerical Methods in Engineering Gelfand, A. Biometrica 85 1 Gelman, A.

Meng, and H. Statistica Sinica Ghanem, R. Journal of Computational Physics 1 Gneiting, T. Weather Forecasting with Ensemble Methods. Science Goldstein, M. Hastie, T. Tibshirani, and J. Higdon, D. Kennedy, J. Cavendish, J. Gattiker, B. Williams, and M.

Hills, R. Albuequerque, N. Dowding, and L. Thermal Challenge Problem: Summary. Computer Methods in Applied Mechanics and Engineering Hoeting, J. Madilgan, A. Raftery, and C.

Bayesian Model Averaging: A Tutorial. Kaipio, J. Statistical and Computational Inverse Problems. Kolehmainen, I. Somersalo, and M. Inverse Problems Kennedy, M. Bayesian Calibration of Computer Models. Kersting, A. Efurd, D. Finnegan, D. Rokop, D. Smith, and J.

Nature Kirk, B. Peterson, R. Stogner, and G. Engineering with Computers 22 Klein, R. Doebling, F. Graziani, M. Pilch, and T. Livermore, Calif. Klir, G. Fuzzy Sets and Fuzzy Logic. Upper Saddle River, N. Knupp, P. Boca Raton, Fla.

Knutti, R. Furer, C. Tebaldi, J. Cermak, and G. Journal of Climate 23 10 Kumamoto, H. Lehmann, E. Testing Statistical Hypotheses.