Backtesting
Template:Short description Backtesting is a term used in modeling to refer to testing a predictive model on historical data. Backtesting is a type of retrodiction, and a special type of cross-validation applied to previous time period(s). In quantitative finance, backtesting is an important step before deploying algorithmic strategies in live markets.
Financial analysis
In economics and finance, backtesting seeks to estimate the performance of a strategy or model if it had been employed during a past period. This requires simulating past conditions with sufficient detail, which means that backtesting is limited by the need for detailed historical data, the inability to model strategies that would affect historic prices, and potential overfitting. That is, it is often possible to find a strategy that would have worked well in the past, but will not work well in the future.[1] Despite these limitations, backtesting provides information not available when models and strategies are tested on synthetic data.
Historically, backtesting was only performed by large institutions and professional money managers due to the expense of obtaining and using detailed datasets. However, backtesting is increasingly used on a wider basis, and independent web-based backtesting platforms have emerged.[2] Although the technique is widely used, it is prone to weaknesses.[3] Basel financial regulations require large financial institutions to backtest certain risk models.
For a Value at Risk 1-day at 99% backtested 250 days in a row, the test is considered green (0-95%), orange (95-99.99%) or red (99.99-100%) depending on the following table:[4]
| Zone | Number exceptions | Probability | Cumul |
|---|---|---|---|
| Green | 0 | 8.11% | 8.11% |
| 1 | 20.47% | 28.58% | |
| 2 | 25.74% | 54.32% | |
| 3 | 21.49% | 75.81% | |
| 4 | 13.41% | 89.22% | |
| Orange | 5 | 6.66% | 95.88% |
| 6 | 2.75% | 98.63% | |
| 7 | 0.97% | 99.60% | |
| 8 | 0.30% | 99.89% | |
| 9 | 0.08% | 99.97% | |
| Red | 10 | 0.02% | 99.99% |
| 11 | 0.00% | 100.00% | |
| ... | ... | ... |
For a Value at Risk 10-day at 99% backtested 250 days in a row, the test is considered green (0-95%), orange (95-99.99%) or red (99.99-100%) depending on the following table:
| Zone | Number exceptions | Probability | Cumul |
|---|---|---|---|
| Green | 0 | 36.02% | 36.02% |
| 1 | 15.99% | 52.01% | |
| 2 | 11.58% | 63.59% | |
| 3 | 8.90% | 72.49% | |
| 4 | 6.96% | 79.44% | |
| 5 | 5.33% | 84.78% | |
| 6 | 4.07% | 88.85% | |
| 7 | 3.05% | 91.90% | |
| 8 | 2.28% | 94.17% | |
| Orange | 9 | 1.74% | 95.91% |
| ... | ... | ... | |
| 24 | 0.01% | 99.99% | |
| Red | 25 | 0.00% | 99.99% |
| ... | ... | ... |
Hindcast
In oceanography[6] and meteorology,[7] backtesting is also known as hindcasting: a hindcast is a way of testing a mathematical model; researchers enter known or closely estimated inputs for past events into the model to see how well the output matches the known results.
Hindcasting usually refers to a numerical-model integration of a historical period where no observations have been assimilated. This distinguishes a hindcast run from a reanalysis. Oceanographic observations of salinity and temperature as well as observations of surface-wave parameters such as the significant wave height are much scarcer than meteorological observations, making hindcasting more common in oceanography than in meteorology. Also, since surface waves represent a forced system where the wind is the only generating force, wave hindcasting is often considered adequate for generating a reasonable representation of the wave climate with little need for a full reanalysis. Hydrologists use hindcasting for model stream flows.[8]
An example of hindcasting would be entering climate forcings (events that force change) into a climate model. If the hindcast showed reasonably-accurate climate response, the model would be considered successful.
The ECMWF re-analysis is an example of a combined atmospheric reanalysis coupled with a wave-model integration where no wave parameters were assimilated, making the wave part a hindcast run.
See also
- Template:Annotated link
- Template:Annotated link
- Template:Annotated link
- Template:Annotated link
- Template:Annotated link
- Template:Annotated link
- Template:Annotated link
- Template:Annotated link
- Template:Annotated link
- Template:Annotated link
- Template:Annotated link
- Template:Annotated link
- Template:Annotated link
- Template:Annotated link
- Template:Annotated link
References
Page Template:Reflist/styles.css has no content.
- ↑ Script error: No such module "citation/CS1".
- ↑ Script error: No such module "citation/CS1".
- ↑ Script error: No such module "citation/CS1".
- ↑ Script error: No such module "citation/CS1".
- ↑ Taken from p.145 of Yeates, L.B., Thought Experimentation: A Cognitive Approach, Graduate Diploma in Arts (By Research) dissertation, University of New South Wales, 2004.
- ↑ Script error: No such module "citation/CS1".
- ↑ Script error: No such module "Citation/CS1".
- ↑ Script error: No such module "citation/CS1".
Script error: No such module "Check for unknown parameters".
