Interview Question: Back-Testing Pitfall

Sample Question #160 (case-type statistics question)

A common thing people on Wall Street do when testing trading strategies is to construct portfolios consisting of some number of stocks from a well-known stock index, for example, the SP 500 or Russell 2000, and then conduct back-testing on these stocks using the strategy. What potentially serious statistical problem do you see this approach suffers?

Advertisements
This entry was posted in Sample Qs. Bookmark the permalink.

One Response to Interview Question: Back-Testing Pitfall

  1. Brett says:

    ANSWER
     
    The serious problem is survorship bias, and this occurs at two levels:
     
    1) Many people construct the portfolio using stocks in a stock index as of research date; but these stocks already "survived" difficult times. When the trading strategy is used in real time, we can never tell if a stock can survive into the future.
    2) Stock indices are already carefully constructed by their vendors, and these construction rules can change in the future which may affect the profitability of the strategy in real trading.
     
    Survivorship bias is a very serious data sampling problem and occurs all the time when you work with data.  The only way to deal with it is to always keep in mind how to properly interpret the results.
     

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s