Randomized Out of Sample

First, what is out of sample testing? Out of sample testing is purposely withholding data from your data set for later testing. For example, let’s say we have ten years worth of data and select to designate the last 30% of the data as out of sample data. Essentially what we do is take the last 3 years (30%) of the data set and put it in our back pocket for later use. We then proceed to create trading strategies using ONLY the first 7 years of data or the “in-sample” data.

BuildAlpha: Let’s assume we find a great looking strategy that performs extremely well on the first 7 years worth of data. What we would then do is take the last 3 years worth of data, or our out of sample data, out of our back pocket and proceed to test our strategy on this “unseen” data.

The idea is… if the strategy still performs well on this unseen, out of sample data then it must be robust and we can have increased confidence it will stand the test of time or at least in how it will perform on future market data.


However, there is a major pitfall to this approach given certain data sets. What if our amazing looking strategy happened to be long only and only performed well on our unseen, out of sample data because the last 3 years (our 30%) were extremely bullish. See picture below (red marks last 30% of data):


What’s a solution? What if we use the first 30% of the data instead of the last 30%? What if we use some middle 30% of the data? Well, we actually still run the risk that any continuous 30% of the data may be extremely trendy. Here is a photo showing a strong trend in the beginning 30% of a data set.


Here is a photo showing a strong trend in the middle 30% of a data set:

The answer? Let’s take our original backtest results and run them through a randomized out of sample test. First we randomly select dates until we have randomly selected 30% of the dates from our data. Now we have a “randomized out of sample” period to put in our back pocket. Basically, we will then run through our entire data set but filter out any trades that would have occurred on our now randomly selected, out of sample dates.

Then we can view the results of the trades that only took place on our randomly selected out of sample dates as a new “quasi-out of sample” equity curve. We’ve now significantly reduced the chances that any successful out of sample performance is solely due to some underlying trend or market regime as our out of sample data is now no longer continuous. Here is an example of random 30% of data selected for out of sample.


However, one more problem remains. What if we randomly created a trend or favorable market regime through our random sampling/selection? That is, we’ve randomly selected 30% of the dates and when they’re put in chronological order for our out of sample testing they coincidentally create a massive up trend that could skew our new out of sample results?

The answer? Run this test 1,000 times where we re-select a random 30% of the dates each time to act as our out of sample period. Some may create a favorable market regime, but we can see how our sample strategy does over a variety of randomly created regimes or out of sample periods.

Build Alpha Solution:

BuildAlpha offers this complete process and test with the click of a button. The leftmost graph below is the in-sample equity curves from the 1,000 runs. These are basically trades that occurred on dates that were NOT randomly selected to be part of the out of sample period. The middle chart is our randomized out of sample test. These are the equity curves from the 1,000 runs of the trades that occurred on the dates that were randomly selected. The thick blue line on the left and middle graph are the in and out of sample tests from our original back-test. The rightmost chart is simply the original back-test.

The example below shows our original backtest’s out of sample equity curve to be at the very top of the distribution of all the random out of sample tests (middle chart below). This is a sign that the underlying data contributed quite favorably to our results and may have produced unrealistic or unrepeatable out of sample results (most likely something like the first picture of this post where the last 30% of the data is considerably bullish).



In the picture below I’ve displayed a simple demonstration that would show a “passing” random out of sample test or a situation where you can feel comfortable the original out of sample period is a “fair” period of time. The middle graph’s thick blue line (our original out of sample results) is in the middle of the random out of sample distribution. This means as we randomly selected random out of sample periods we created some out of sample equity curves that had better results than our original backtest and some that had worse results than our original backtest. This can only mean that our original out of sample period was “fair” and did not overly contribute to the favorable results, but rather the favorable results come from the strategy itself.


Some Takeaways:

We know we want to test on “unseen” data, but also want to reduce the chances that an underlying trend produced favorable out of sample results. In order to avoid being fooled by a favorable out of sample period, we randomly create thousands of out of sample periods and want to see general profitability across them all.

If the original backtest’s out of sample line is significantly greater than all the randomized out of sample equity curves then we can conclude we had originally selected an overly optimistic out of sample period that contributed too greatly to our strategy’s out of sample success (similar to the first photo in this post).

Additionally, if our original backtest’s out of sample line is one of the worst performers compared with the randomized out of sample equity curves then we can conclude the original out of sample period chosen may actually be too unfavorable (and throwing the strategy away based on out of sample results alone may be ill-advised).

All in all, Randomized Out of Sample testing is just one more test Build Alpha offers to provide the utmost confidence a strategy will perform to expectation moving forward.

 Originally Posted: https://www.buildalpha.com/randomized-out-of-sample/

Comments

Post a Comment

Popular posts from this blog

Noise Test

Image
First, let’s talk about overfitting and what it is. I will simply define it as fitting a function or model to data so well that the function or model is not generalizable and cannot/will not work on another data set. In trading terms, it is designing a strategy that trades historical data so well that it will surely fail on new data . Build Alpha :  The first and most obvious test to prevent curve fitting is out of sample testing. In other posts I’ve described this, but in short, it is simply withholding a portion of your data to “validate” the strategy. For example, you have 10 years worth of data and you designate the first or last 30% to be out of sample. You build your model on the remaining seven years of data and then validate it on the withheld 30% or out of sample data. However, and for various reasons, this might not be enough to prevent against curve fitting. There are two parts to the underlying data: the signal and the noise. It is often said that when something i...
Read more

Downloading Custom Data for Build Alpha using Python

Image
Happy Friday! For this Free Friday edition, I am going to do something new. I am going to make this slightly educational and give away some code. I get tons of questions every week, but they mainly fall into two categories. The first question is in regards to adding custom data to Build Alpha . You can add intraday data, weekly data, custom bar type data, sentiment data, or even simple single stock data. The second question is in regards to using or learning Python. In this post, I will attempt to “kill two birds with one stone” and show a simple Python code to download stock data from the Yahoo Finance API. In fact, we will use Python to pull and save data ready formatted for Build Alpha for all 30 Dow stocks in less than 30 seconds. You can view the entire script later in this code or in the video below. The first few lines are simple to import statements pulling public code that we can reuse including the popular pandas library. import pandas ...
Read more

Trading Trends with Build Alpha

Image
Trading is basically buying and selling of things to capture a change in price. There are many types of trading activities which are there in the market. Due to the development in the technology, there are many softwares and tools made for the convenience of different trading activities. Among all trading activities, trading in stock market requires the utmost help. For this purpose, Build Alpha helps the trader the most. It is one of the leading softwares in the trading market which has done wonders with its expert skills. With the changing trend of the market every now and then, the human expertise and knowledge is not efficient to analyze all the trends keeping in mind all the factors that affect the trend. This is how the human expert’s evaluation turns vague in front of the software analysis. Trading software helps in trading and analyzes the movement of the various stocks, currencies and commodities worldwide. Many brokerage houses provide their clients wi...
Read more