Find Your Edge...
If you're still looking for an edge in the markets, mechanical trading systems are the best way to get it. Learn more.
Software for Significance Testing
Detect over-optimized trading systems by performing statistical significance testing on your trading system or method using Market System Analyzer (MSA). Given the profit/loss history from a trading system or method, MSA will determine whether the system or method is likely to be profitable in the future. Market System Analyzer can be applied to any trading system or method regardless of market or time frame. Significance testing can be a powerful tool to aid in the development and testing of trading systems and methods.
What is a Significance Test?
A statistical significance test is a mathematical test for determining if the apparent difference between two results is significant or due merely to chance. With regard to trading, a significance test can be used to help determine if a trading system or method is likely to be profitable in the future. Significance testing can also be used to detect curve-fit or over-optimized trading systems.
Over-fitting or over-optimization means that the trading system's parameters have been selected to work on specific markets or under limited market conditions. An over-fit or over-optimized trading system is unlikely to perform well on other markets or when market conditions change. Trading experts agree that over-optimized systems should be avoided.
The most common significance test is the Student's t test, which can be applied to the average trade for the trading system or method under consideration. The test determines if the average trade is significantly greater than zero at a specified confidence level. For example, the test will determine if the average trade is greater than zero with, say, 95% confidence.
Significance Testing is Simple in Market System Analyzer
MSA automatically performs a Student's t test when the current sequence of trades is analyzed. The settings for the significance test are made on the Analysis Setup window, as shown below.
The parameters for the significance test are entered in the Significance Test dialog window in Market System Analyzer.
The number of rules and restrictions for the trading system or method and the confidence interval for the t test are entered in the window. The results are displayed under the Significance Test tab of the Performance Results window, as shown below.
In Market System Analyzer, the significance test results include the average trade with confidence intervals and the probability that the average trade is greater than zero.
The Performance Results window displays the average trade at the specified confidence level (average trade +/- confidence intervals), the worst-case trade at the specified confidence, and the probability that the average trade is greater than zero.
If the worst-case average trade at the specified confidence level is greater than zero, it means that the system or method is inherently profitable subject to the assumptions of the test.1 In this case, a message will be displayed, as shown above, indicating that the trades pass the significance test at the specified confidence level. If the worst-case trade is less than or equal to zero at the specified confidence level, a message will be displayed to indicate that the trades did not pass the significance test.
To learn how to trade crossovers of the equity curve using Market System Analyzer, click on the Next button at the bottom of the page or go to the online store below to purchase your own copy of MSA.
1One of these assumptions is that the statistical properties of the trades remain the same. Specifically, if the average trade and its standard deviation remain the same in the future, the results will continue to be valid. However, as markets change and evolve over time, the properties of the statistical distribution of trades may change as well, so caution is warranted in interpreting the results.
Download a fully functional trial version of Market System Analyzer. Evaluate MSA for up to 30 days. Click here to download now without obligation.
For a general article on significance testing in trading, click here. For a full list of available trading articles, select the Article Library link at left.
If you'd like to be informed of new developments, news, and special offers from Adaptrade Software, please join our email list. Thank you.
For Email Marketing you can trust
Copyright (c) 2004-2019 Adaptrade Software. All rights reserved.