A Statistical Method for Evaluating Trading Systems

Much of technical analysis, particularly as applied to futures trading, is ad hoc. Most technical indicators, for example, while written in the language of mathematics, are essentially someone's view of how the markets should work. The stochastic oscillator, for example, is based on the idea that in a rising market, the closes tend to be near the top of the recent high-low range. If prices hover near the top of this range, then presumably the market is 'overbought' and likely to fall. This has no more (or less) mathematical justification than the algebra on which the stochastic is based. This is not to dismiss technical analysis, some of which can be quite useful. However, it probably pays not to be overly impressed with complicated-looking technical formulations, most of which have only anecdotal justification for their intended use.

There are exceptions, of course. I would put statistical methods into this category. Anytime you can find a statistical method that applies to trading, it's probably worth looking into. In fact, I tend to think of even simple statistical methods, like the one I'm about to present, as a level above most technical analysis.

One question that can be addressed by the use of statistics is whether a trading system is inherently profitable. We can approach this problem using confidence intervals for the average trade. If we have a sample of, say, 100 trades from a trading system, we can compute the average trade, T. Of course, we expect T to be greater than zero, indicating that the system has been profitable on average. However, if we took a different sample of 100 trades, we would, in general, find a different average trade, T. If the variation among the trades is large enough, it's possible that some of these averages could be less than zero, indicating that the system was not profitable on average for those trades.

By computing the confidence intervals for the average, T, we can determine whether it's likely that the average will be greater than zero. The confidence intervals specify upper and lower bounds for the average. The true average lies within those bounds with some specified probability or confidence level, such as 95%. The equation for the confidence intervals is as follows:

    CI =  t * SD/sqrt(N)

where t is the Student's t statistic, SD is the standard deviation of the trades, N is the number of trades, and sqrt represents "square root." The average trade is likely to lie between T - CI and T + CI. For the system to be profitable at our specified confidence level, we need T > CI.

The value of t depends on the specified confidence level and the number of trades, N. The exact value can be found in a statistics table for the t distribution or calculated in software, such as from the TINV function in Excel. However, provided we have a reasonably large number of trades, the exact value is not necessary. If N = 60, the t value for 95% confidence is t = 2.00. For larger values of N, t will get slightly smaller, dropping to 1.96 for very large N. To be conservative, then, we can take t = 2.00 as long as we have at least 60 trades. If our actual value of N is larger than 60, we will have slightly larger intervals than if we used the exact value of t.

Under this assumption, then, we have

    CI = 2 * SD/sqrt(N);  N >= 60, 95% confidence.

As an example, I calculated the confidence intervals for MiniMax II on the E-mini S&P. Going back to 1998, there are 241 trades. The average trades is $248 with an standard deviation of $990. This gives us a confidence interval of CI = 2 * 990/sqrt(241), which is equal to $128. So, we can expect the average trade to lie somewhere between 248 - 128 and 248 + 128 or between $120 and $376. The lower number, $120, is greater than zero, so the system should be profitable.

Because the square root of N is in the denominator, all other things being equal, the more trades we have, the smaller our confidence intervals will be. This is a way to quantify what most of us already know from intuition and/or experience: if you want to know whether a trading system is profitable, the more history the better. In fact, we can re-write the CI equation to tell us how large N needs to be in order to demonstrate profitability:

    N > 4 * (SD/T)^2

where the ^2 indicates "square." This assumes we have a good estimate for the standard deviation and average trade.

As an example of this equation, let's take the numbers from above. We had an average trade, T = 248, and a standard deviation, SD = 990. Plugging these into the equation for N, we get N > 63. In other words, with these average trade and standard deviation numbers, there's a 95% chance the average trade will be profitable (i.e., greater than zero) provided we have more than 63 trades in our sample.

What are the limitations of this approach? As usual with statistics, we have to pay careful attention to the underlying assumptions. On a positive note, we don't have to worry about normality provided we have at least 30 trades. The central limit theorem of statistics says that as long as we have at least 30 "observations" (i.e., trades in our case), the distribution of the averages will be normal even if the trades themselves are not normally distributed.

The one concern is that the accuracy of the confidence intervals is dependent on the distribution of trades remaining the same. In statistics, this is called "stationarity." If the true average and standard deviation change over time, the confidence intervals may not be accurate, depending on how much things change. For example, if the trades are based on a system developed by "over-fitting" the system to the market, then the system may not hold up well in the future. This will be reflected in a change in the average and standard deviation of trades. This confidence interval technique can't tell us anything about how robust a system is or if it's been over-fit to the market.

Next month, I'm going to draw an analogy between least-squares curve-fitting in mathematics and trading system optimization. This can tell us something about how many trades we need when we optimize the parameters of a trading system to avoid the over-fitting problem.