Detect over-optimized trading systems by performing statistical significance testing on your trading system or method using Market System Analyzer (MSA). Given the profit/loss history from a trading system or method, MSA will determine whether the system or method is likely to be profitable in the future. MSA can be applied to any trading system or method regardless of market or time frame. Significance testing can be a powerful tool to aid in the development and testing of trading systems and discretionary trading methods.

What is a Significance Test?

A statistical significance test is a mathematical test for determining if the apparent difference between two results is significant or due merely to chance. With regard to trading, a significance test can be used to help determine if a trading system or method is likely to be profitable in the future. Significance testing can also be used to detect curve-fit or over-optimized trading systems.

Over-fitting or over-optimization means that the trading system's parameters have been selected to work on specific markets or under limited market conditions. An over-fit or over-optimized trading system is unlikely to perform well on other markets or when market conditions change. Trading experts agree that over-optimized systems should be avoided.

The most common significance test is the Student's t test, which can be applied to the average trade for the trading system or method under consideration. The test determines if the average trade is significantly greater than zero at a specified confidence level. For example, the test will determine if the average trade is greater than zero with, say, 95% confidence.

Significance Testing is Simple in Market System Analyzer

MSA automatically performs a Student's t test when the current sequence of trades is analyzed. The settings for the significance test are made on the Analysis Setup window, as shown below, for market systems or on the Settings and Options tab of the Portfolio Setup window for portfolios.

The parameters for the significance test for a market-system file are entered on the Options tab of the Analysis Setup window in MSA.

The number of rules and restrictions for the trading system or method and the confidence interval for the t test are entered in the window. The results are displayed under the Significance Test tab of the Performance Results window, as shown below.

In MSA, the significance test results include the average trade with confidence intervals and the probability that the average trade is greater than zero.

The Performance Results window displays the average trade at the specified confidence level (average trade +/- confidence intervals), the worst-case trade at the specified confidence, and the probability that the average trade is greater than zero.

If the worst-case average trade at the specified confidence level is greater than zero, it means that the system or method is inherently profitable subject to the assumptions of the test.¹ In this case, a message will be displayed indicating that the trades pass the significance test at the specified confidence level. If the worst-case trade is less than or equal to zero at the specified confidence level, as shown above, a message will be displayed to indicate that the trades did not pass the significance test.

To learn more about Market System Analyzer (MSA), click here.

To see how to optimize system parameters and position sizing parameters simultaneously using Market System Analyzer and TradeStation/MultiCharts, click here.

¹These assumptions include that the statistical properties of the trades remain the same. Specifically, if the average trade and its standard deviation remain the same in the future, the results will continue to be valid. However, as markets change and evolve over time, the properties of the statistical distribution of trades may change as well, so caution is warranted in interpreting the results. The test also assumes the strategy in question is from a single evaluation. If the strategy was arrived at by evaluating multiple candidates, such as optimizing the strategy inputs and selecting the best result, then a different, stricter test would be required.