Suppose you like to play the horse races, and your bookie tells you he has developed a surefire, computer-generated formula for picking winners. It's based on hundreds of variables such as the weather and a horse's age, and he has "backtested" it on 20 years of track history. Now he's offering to let you bet $50,000 using the formula's recommendations.
(
More:
One of Wall Street's most popular trading strategies is failing)
Would you do it? How can you be sure that the historical success isn't a fluke? Even worse, what if he deliberately designed his formula to look better than it is, so he can take your money at odds he knows are in his favor?
Backtesting is a complex statistical process that can generate misleading results. Often, formulas that do well in hindsight — for example, by picking winning horses or stocks — don't work in the real world. They can even be engineered to "fit" the past by incorporating hundreds of coincidences that have zero predictive value.
Financial companies spend a lot of time and effort creating proprietary indexes that backtest well, so they can sell various investments based on them. Industry rules generally bar brokerages from marketing backtested results to retail investors, because of skepticism that such results have any predictive value. But insurers face less stringent oversight, so their marketing materials often show how well the indexes on which they base their products would hypothetically have done in the past.
The National Association of Insurance Commissioners is considering tightening rules on how long an index must exist before its historical results can be used in investment-product illustrations, sparking a debate between industry groups and consumer advocates. The former argue that the proliferation of engineered indexes is great for consumer choice. The latter say that savers need much better protection from investments based on indexes that may not perform as well as their marketing materials suggest.
Who's right? To get a sense, let's look at how some actual backtested indexes perform after they're launched in the real world.
Consider the BNP Paribas Multi-Asset Diversified 5, the index driving an Athene insurance product that an agent marketed to me. Here's a chart from an Athene brochure, with 12 years of backtested data that appear to show how the index would have delivered a comparable return to the S&P 500 with less volatility (never mind that the chart excludes the dividends that an investor in the S&P 500 would have received):
The picture, however, looks very different if we focus on the period since January 2016, when the index was actually launched. A $100 investment would have grown to just $112 in the MAD 5 index, compared with $159 in the S&P 500 (dividends included):
https://cdn-res.keymedia.com/investmentnews/uploads/assets/graphics src="/wp-content/uploads2019/06/CI120102617.JPG"
Next let's explore the NYSE Zebra Edge Index, which insurer Nationwide employs in some of its investment products. A Nationwide brochure shows how, had it existed, the Zebra would have beaten large-cap U.S. stocks for decades:
But how has the Zebra performed against large-cap stocks since its actual inception in October 2016? It's up about 16%, compared with 37% for the S&P 500 (again, dividends included).
https://cdn-res.keymedia.com/investmentnews/uploads/assets/graphics src="/wp-content/uploads2019/06/CI120103617.JPG"
These aren't isolated examples. Time and again, indexes marketed by insurers fail to fulfill the promise suggested by their hypothetical histories. Granted, the insurers tend to include footnotes such as Nationwide's, which states that "past performance is not indicative of, nor does it guarantee future performance." But if the backtested illustrations are no guide, then why use them at all? (An Athene spokeswoman said that the hypothetical data "provides more information on the index that should help the comparison." Nationwide did not respond to questions by time of publication.)
All too often, the hypothetical illustrations serve to draw retail investors into illiquid, complex and fee-laden products, which have no inherent advantage over cheaper, mainstream strategies. Given the number of Americans already likely to fall short of their retirement goals, this is a serious problem.
(
More:
Betterment offering automated investing and advice for Optum Bank HSA)
What to do? I have three recommendations for insurance commissioners.
Follow the guidelines set out by the Financial Industry Regulatory Authority in the brokerage context: Don't allow communication or display of backtested results to any prospective retail insurance customer.
Require indexes to be fixed and in existence for 20 years — enough to prove themselves through several market cycles — before they are used in any illustration of projected returns.
Require all marketing materials and illustrations to compare the offered products' actual historical returns to those of mainstream, investable indexes — including the dividends that investors in those indexes would receive.
Retail investors can't be expected to understand all the ways in which backtesting can make an investment look much more attractive than it is. Only better regulation can give them the protection they need.
Ethan Schwartz has worked as an investment manager and financial services executive for 21 years. He was a special assistant to the deputy secretary of the Treasury in the Clinton administration.
