Noise Test and Lying Backtests

Can you trust a backtest?

Below is a simple strategy that uses Intermarket signals from the Japanese Yen to trade US 30 Year Bond Futures.

It only has 2 rules for entry. The strategy produces fairly stable in-sample returns especially for being so simple. The in-sample period is from January 2003 to August 2012.

The Intermarket strategy actually performed quite well out of sample on our unseen data as well. The out of sample data is from September 2012 to October 2016 (highlighted in blue). The ability for a strategy to continue to perform on unseen data is paramount and mission-critical for many system traders and strategy architects.


However, can you trust this backtest? In BuildAlpha, there are many validation and robustness tests that can add confidence to your system development process. There is a special test called the “Noise Test” which creates 1000 new price series by adding and subtracting varying amounts of noise from the underlying price data. The test then “re-trades” the strategy on the new 1000 price series.

The idea is that if a strategy is overfit then it is overfit to the noise of the original data and is not trading the underlying signal we hope it is. Changing the noise will reduce performance if the strategy is overfit to the noise of the original data. Ideally, we want to see a strategy continue to perform well on the new, noise-adjusted price series.



This “Noise Test” can help prevent a backtest from lying to us and hopefully prevent us from wasting further research time or taking a weak strategy live! You can see that our original backtest in blue is significantly better than the 1000 “Noise Test” results. This is worrisome and most likely a sign of an overfit strategy that will surely fail on live data.

Viewing this simple strategy’s performance from October 2016 to June 20, 2017, confirms the value of the Noise Test. The strategy has been in a nosedive since going “live”, but a smart trader would have never taken it to live – no matter how pretty the backtest – due to the failed Noise Test.


Any kind of trading – manual or discretionary – should be subjected to these rigorous tests before getting real money allocations. If not, you may be in for a surprise from a lying backtest or unreproducible hypothetical returns!

Thanks for reading.

Dave is the founder of BuildAlpha software. Build Alpha is software that automatically creates systematic trading and investing strategies allow the validation and testing of each strategy, and generates exportable and executable code for each strategy – no programming necessary.

Originally Posted: http://www.seeitmarket.com/system-traders-the-noise-test-and-lying-backtests-16997/

Comments

Post a Comment

Popular posts from this blog

Noise Test

Image
First, let’s talk about overfitting and what it is. I will simply define it as fitting a function or model to data so well that the function or model is not generalizable and cannot/will not work on another data set. In trading terms, it is designing a strategy that trades historical data so well that it will surely fail on new data . Build Alpha :  The first and most obvious test to prevent curve fitting is out of sample testing. In other posts I’ve described this, but in short, it is simply withholding a portion of your data to “validate” the strategy. For example, you have 10 years worth of data and you designate the first or last 30% to be out of sample. You build your model on the remaining seven years of data and then validate it on the withheld 30% or out of sample data. However, and for various reasons, this might not be enough to prevent against curve fitting. There are two parts to the underlying data: the signal and the noise. It is often said that when something i...
Read more

Downloading Custom Data for Build Alpha using Python

Image
Happy Friday! For this Free Friday edition, I am going to do something new. I am going to make this slightly educational and give away some code. I get tons of questions every week, but they mainly fall into two categories. The first question is in regards to adding custom data to Build Alpha . You can add intraday data, weekly data, custom bar type data, sentiment data, or even simple single stock data. The second question is in regards to using or learning Python. In this post, I will attempt to “kill two birds with one stone” and show a simple Python code to download stock data from the Yahoo Finance API. In fact, we will use Python to pull and save data ready formatted for Build Alpha for all 30 Dow stocks in less than 30 seconds. You can view the entire script later in this code or in the video below. The first few lines are simple to import statements pulling public code that we can reuse including the popular pandas library. import pandas ...
Read more

Trading Trends with Build Alpha

Image
Trading is basically buying and selling of things to capture a change in price. There are many types of trading activities which are there in the market. Due to the development in the technology, there are many softwares and tools made for the convenience of different trading activities. Among all trading activities, trading in stock market requires the utmost help. For this purpose, Build Alpha helps the trader the most. It is one of the leading softwares in the trading market which has done wonders with its expert skills. With the changing trend of the market every now and then, the human expertise and knowledge is not efficient to analyze all the trends keeping in mind all the factors that affect the trend. This is how the human expert’s evaluation turns vague in front of the software analysis. Trading software helps in trading and analyzes the movement of the various stocks, currencies and commodities worldwide. Many brokerage houses provide their clients wi...
Read more