There’s Still No Evidence You Can Beat the Market


I recently updated Article 5.2 on Evidence Regarding Excess Returns, which I first published in 2016.  I added new data from several annually published research reports, which evaluate investor’s ability (or inability) to achieve returns beyond those obtained by investing in index funds.   I routinely update articles throughout Mindfully Investing, but I thought it was a good idea to write a post about this particular update, because the results continue to be so important for individual investors.

I’ve read literally hundreds of personal finance blog posts about investing over the last year as part of my development and maintenance of the Guide to Personal Finance Blogs.   I’ve found that investing bloggers fall into two main camps.  The first camp extols the virtues of index investing and accepts unequivocally that picking individual stocks is doomed to failure.  Sometimes these bloggers present evidence to support their position, and sometimes they just seem to accept the merits of index investing without question.  The second camp is still convinced that, through diligence and careful assessment of individual stocks, they can achieve better results than index investing.  Some of these bloggers pick their own individual stocks and some use actively managed funds that contain stocks picked by “experts”.

The original 2016 version of Article 5.2 found that there was practically zero evidence that anyone, even the very best professional investors, can routinely beat index funds.  It’s probably no real spoiler to say that updating the article with 2017 data in no way changed this conclusion.  I should note that this article mainly compares actively managed funds (of all types) to index funds.  Regardless, “active management” generally means that professional investors are picking stocks for these funds.  So, I think these results also clearly apply to the do-it-yourself stock pickers out there.

Whether you fall into the pro indexer camp or the pro stock-picker camp, I think it’s important to recognize the mountain of evidence that supports an indexing approach.  For the one camp, this validates your investing approach, whether you had a rationale for it or not.  For the second camp, this is tough-love news that’s risky for you to ignore.

Without further preamble, let’s look at the evidence (or lack thereof) for investing excess returns.

Professional investors

S&P Dow Jones Indices publishes SPIVA (S&P Indices Versus Active) scorecards for professional money managers.  The SPIVA scorecard published in mid-2017 indicated that the following percentages of stock fund managers under performed the most relevant indices over the last 5 years:

  • 82% of large-cap managers
  • 87% of mid-cap managers
  • 94% of small-cap managers

Vanguard presented a 2017 research paper containing numerous statistics on active fund under performance including the graph shown below.  The dark portion of the bars include currently operating funds, while the light portion includes funds that closed during the period of analysis, often due to poor performance.  (The closings and mergers of funds is one complicating factor in fairly evaluating fund performance, and failing to evaluate closed funds can artificially enhance the success rate for active funds.)

While the SPIVA scorecard is confined to stock funds, the Vanguard graph shows that active fund under performance is epidemic in stock and bond funds, both foreign and domestic.  The percentage of under performing funds is well above 50% in almost all cases, particularly over longer time periods.  The average excess return shown by the red numbers under each bar are negative (meaning you would have received greater return by investing in the index instead) in almost all cases as well.

Active funds have not performed well in the past, but is it possible they are improving?  Although some suggest there is recent improvement in the last year, even these observers admit that index funds are still well ahead for periods of 10 to 15 years as seen in the Vanguard graph above.  Similarly, here is a graph from one analysis  that focuses on hedge fund excess return (or alpha) from 1996 through 2013.

Perspective 10 14 fig 6

The clear trend over time is toward under performance.  And recall that hedge funds attract the top talent because of their high cost structure.

But perhaps these comparisons are too broad.  Could there be a subset of active fund managers who outperform the market consistently?  Maybe we could just select that subset of funds to create a beat-the-market portfolio.  S&P Dow Jones Indices also publishes a “persistence scorecard”, which examines whether picking funds based on past performance, presumably because some managers are more skillful than most, would provide a better outcome for individual investors.  In the 2017 persistence study, S&P Dow Jones evaluated 2,274 actively managed domestic stock mutual funds.  They found that of the 568 funds that were in the top quartile for performance in March 2015, only 1.9% managed to stay in that top quartile by March 2017.  That is, almost every fund that was out performing its peers in 2015 was no longer doing so just two years later.  They examined performance persistence in a number of other ways as well.  By every measure, almost none of the funds were able to out perform their peers for long periods.

The very best investors

What if we took it one step further?  Surely there are a few exceptional professional investors who usually beat the market, allowing for a bad year now and then.  Good information on exceptional investors is a little hard to find, because so few investors seem to beat the market.  So, let’s look at the often-cited best investor of recent times, Warren Buffett.  He is famous for beating the market by purchasing stocks of a few well-managed companies, particularly at times when these companies were undervalued by the market.

Here is a graph of the out performance (excess returns) of Warren Buffett’s company, Berkshire Hathaway, as compared to the S&P 500.

Buffett performance graph

Clearly, Berkshire Hathaway has an outstanding record as compared to most of the professional investors we’ve examined.  The company has 22% compound annual returns (including dividends) since 1965, while the S&P 500 had only a 10% annual return in that same period.  But even here we notice that the heydays of stellar excess returns are over.  From 1996 through 2014, Berkshire Hathaway excess return had an annual average of about 2.3%.  Random chance predicts that ,out of thousands of brilliant active investors, at least a few should beat the market over the long-term just based on luck.  I am not saying that Warren Buffett is all luck.  It’s likely that his investing methods contributed to this success.  But I think it’s reasonable to say that his success is at least a combination of well applied methods, some luck, and later in his career, the advantages of being a big player.

Regardless, mindful humility advises that it’s unwise for individual investors to try to copy the best known professional investor, who spends almost all his time, energy, and obvious intelligence on this single endeavor.  If we humbly submit the question to the guru himself, Warren Buffett has provided crystal clear advice for us individual investors.  For example, he advises his wife to, after his demise, buy a stock index fund with 90% of their assets and simply hold it.  He has given essentially the same advice for all individual investors and noted that hiring someone to manage your money is essentially useless.  It’s hard to argue with him.

Individual investors

If professional investors have a hard time beating the market, you’ve probably guessed by now that the news is worse for small individual investors like us.  Here is a graph of returns by asset class from JP Morgan with the average investor returns added.  If individual investors can’t seem to routinely perform even as well as holding a 100% bonds, then in aggregate, there are very little excess returns happening for individual investors, although some outliers assuredly occur.

Investor behavior – The reason for this under performance is emotions.  Individual investors jump in and out of investments due to emotional decisions (like panic selling) and their innate biases.  This behavior results in many investors missing multiple days of positive stock performance.  Many of the biggest up days in the stock market happen after a long-term bottom is achieved and emotional investors have just sold.  In his book “Winning the Loser’s Game”,  Charles Ellis notes the following decrease in average annual compound returns for the S&P 500 during a 28 year period when a few big days are missed:

  • S&P 500 – 11.1% return
  • S&P 500, but missing the 10 biggest days – 8.6%
  • Missing the 20 biggest days – 6.9%
  • Missing the 30 biggest days – 5.5%

Approximately half the average annual return of the S&P 500 in this 28 years can be attributed to just 30 days.  So, seemingly trivial decisions, like being out of the market for a few days, can end up having a huge impact on your investment success.

Individual investor gaps – Other research verifies the poor performance of individual investors.  Looking at investments in specific funds, several studies have shown that individual investors achieve returns that are 1 to 3 percent lower than the funds themselves.  This percent “gap” between the investor and fund performance over time is summarized in this graph from the Maymin and Fisher study as a “behavioral trading cost”.

Behaivoral cost gap

And Morningstar reported that these behavioral trading gaps occur in almost all groups of funds examined, as shown in this graph.

Chasing performance – The opposite of the panic selling is called “chasing performance”.  As I noted above, there is little evidence that fund managers who perform well for several years will continue that good performance into the future.  So, selecting funds based on past performance is unlikely to be fruitful.  Individual investors seek the “hot hand” only to find the luck has gone cold once they get in the game.  This behavior leads to lower individual investor returns as compared to simply staying in the same fund over the same period as shown in this graph from a Vanguard study.

Bar graph buy hold vs performance chasing

More importantly, because the vast majority of actively managed funds fail to beat index funds, when individual investors put their money in active funds they often get the double whammy of poor performance from both the fund and their own emotional investor behavior.  This dual poor performance mostly explains why the aggregate returns for individual investors (as shown in the JP Morgan graph above) is so far below the returns for most investment asset classes.


Another year of data provides no new evidence that you (or anyone) can routinely beat an index investing approach.  The deck is heavily stacked against you, whether you try to pick individual stocks yourself or you attempt to pick the “very best” fund managers and pay their fees.  And all evidence suggests that the longer you strive for excess returns, the worse your returns will be as compared to simply buying and holding an index fund.  And personally, I find index investing so much easier and less frustrating that trying to pick stocks.  Back when I used to pick stocks, more than a few times I bought a great looking stock after hours of research just to see it plummet because of some new surprising event or disclosure.  Even though I had some stock picking successes too, I really don’t miss that game at all.


  1. Great article! I’d caveat it with a few points:

    1) Professional managers do beat the indices, they just don’t do it net of fees. So basically, they’re not generating sufficient alpha to justify their fees.

    2) Alpha is coming down over time because it’s gotten so competitive. With algorithmic trading and other quant strategies, it’s much harder to find mis-pricing vs. the past. And the fact that there are too many HFs following the same strategies.

    • Karl Steiner says:

      I agree on both points. First, some professional managers do indeed beat the indices before fees are included. But let’s not forget that many don’t. And if I’m an individual investor that can’t devote every waking hour to looking for mis-pricing, what are the chances that I’m going to do better than most of these professionals? And it appears just as hard to pick a manager who might beat the market consistently based on these persistence statistics. Second, I totally agree that, if anything, attempts to beat the market are getting harder, not easier. Thanks for your thoughtful comments.

      • Just because something is hard doesn’t mean you shouldn’t try:)

        Index funds definitely have a large place in a portfolio, but when you invest on your own, you learn about, a) companies and how they make money, and b) how the stock market works.

        Maybe you lose a couple of points of growth, but those can be pretty valuable lessons.

Leave a Reply

Your email address will not be published.