The S&P 500 Index: Is It All Investors Need?

Dan Wiener:

Most active stock and bond portfolio managers lag their index benchmarks due to high fees and lousy picks. Even the best stock-pickers have periods when they do poorly. But responding with the notion that you could just buy an S&P 500 index fund and go home? Well, that’s a myth.

Dan Wiener:

Hello, and welcome to another The Adviser You Can Talk To Podcast. I’m Dan Wiener, chairman and co-founder of Adviser Investments. And today I’m joined once again by my partner and Adviser Investments’ director of research, Jeff DeMaso.

Jeff DeMaso:

Hey, Dan, thanks for having me.

Dan Wiener:

Anytime, Jeff. It’s always fun when you and I are podcasting together.

Jeff DeMaso:

Sure is.

Dan Wiener:

So, let’s dive in here. Stop with all the chitchat. Warren Buffett, probably one of the best-known stock-pickers in modern market history, shocked his followers years ago when he suggested that upon his death, his trustees should—rather than picking stocks or finding someone who could—simply buy an S&P 500 index fund.

Dan Wiener:

I see you grimacing back there, Jeff. Today, with the stock market having risen further and faster than just about any other asset class (other than, say, bitcoin), the advice seems to be: “Why don’t you just go out and buy an index fund?” That seems to have come back in vogue. I know that you’ve got some opinions about that, Jeff. Talk to me.

Jeff DeMaso:

Yeah. Look, I get it. I understand. Let me just put some numbers behind that idea that the stock market has risen further and faster. Over the past 10 years, the Vanguard 500 index fund is up some 340% or so, small caps up 270%, foreign stocks up 100%, bonds up just 30%, real estate up 175%. So clearly, you just wanted to buy the S&P 500.

Dan Wiener:

Sounds like it to me.

Jeff DeMaso:

Well, okay. So here’s the secret, Dan, and one you might not expect from me: The S&P is potentially all you need, but there’s some really big caveats. So let me get there, right?

Dan Wiener:

Yeah. I heard you say “potentially.”

Jeff DeMaso:

Yeah. If someone comes to me and says, “Hey, I’m just starting out my investment career. All I’m going to do is buy the S&P, and I’m going to hold it the entire time. In fact, I’m going to add to it periodically through regular 401(k) contributions.” Okay. I expect that they’re going to do very well, even better than most over their full investment lifetime, which I hope we’re measuring in decades.

Dan Wiener:

Yeah, I hope so.

Jeff DeMaso:

Right. But I hear people say that all the time, or I hear people say, “Oh, I’ll just go buy the S&P.” I have never once seen anybody actually own just the S&P.

Dan Wiener:

I was just going to ask you, how many portfolios have you ever seen that are invested in one index fund, the S&P index fund?

Jeff DeMaso:

No, I’ve pretty much never seen it. And we see a lot of portfolios, even a lot of 401(k) portfolios, which have really bad choices. And you might think someone there would just own one index fund. We don’t see it, and there’s some very, very good reasons for it. I think this debate is actually a little bit bigger than just talking about putting your money in an index fund. It encompasses active versus passive, the role of bonds in a portfolio, foreign stocks. It really gets at that question of diversification.

Dan Wiener:

Right.

Jeff DeMaso:

So, where do you want to start?

Dan Wiener:

When people talk about the S&P 500 as the sole investment in their portfolio, I always start with the math, right? And then we can get into the diversification stuff. So, the math of losses: There’s this old thing about “If I’m down 10%, I only need an 11% gain to get even. If I’m down 20%, I need a 25% gain to get even. And if I’m down 50%, guess what? I need a double. I need a 100% gain to get even.” And this lesson was learned during the great financial crisis.

Dan Wiener:

When we saw some of the active managers—well, one active manager in particular that we invest with—down 35% at the worst during the great financial crisis and the bear market, versus the market being down 51%. On the recovery after the market bottomed in March 2009, this manager who was down 35% recovered all of his losses two years before the S&P 500.

Dan Wiener:

So, I always say it sounds great in theory, when stock markets are going up, to just have all your money in the S&P 500. And the minute the risk shows its ugly face, all of a sudden you wonder about whether diversification might have been a little easier for you to live with.

Jeff DeMaso:

Yeah. Let me pull on that, Dan. Because I think that’s the right thread to go into diversification and managing the drawdowns. It’s great when the markets are going up, but we know not everyone can stomach a 100% swing of the equity market.

Jeff DeMaso:

And this is where bonds enter the picture. And look, I know right now everyone’s saying, “Why bother owning bonds when the 10-year Treasury is yielding one and a half or 1.6% or so?” That’s not a great return proposition. You’re only going to earn 1.5% over the next 10 years, so why own bonds?

Dan Wiener:

But it’s a heck of a lot better than the 0.5% yield we saw not too long ago, right?

Jeff DeMaso:

It sure is. And then that’s a point—that interest rates do go up and you can start earning more income. Well, look, even Warren Buffett, in his instructions to the trustees, wasn’t 100% in the S&P. It was actually 90% S&P and 10% in Treasury. So that was still providing a bit of ballast in the portfolio coming from Warren Buffett. And I think that’s really important.

Dan Wiener:

Yeah. And there’s also the business about risk, right? I mean, there’s the risk question about bonds versus the risk question of stocks, right?

Jeff DeMaso:

And they’re totally different. Look, a bad day for stocks might be a bad year for bonds and vice versa.

Dan Wiener:

Right. A bad day in stocks, you can see stocks down 2%, 3%, 4% in a day. That’s a bad day. The worst, what is it? The worst return we’ve seen for the bond market as a whole over the last few decades was I think a 5% decline?

Jeff DeMaso:

Right. And that’s where, again, you’ve got to think about bonds and what they’re doing in your portfolio. They’re not going to offer a lot of return where we are today, but they’re still very valuable as a means to control the amount of risk in your portfolio. Again, not everyone can handle 100% swings of the stock market, and that’s fine. We’re not all born to do that and see our accounts go up and down, particularly to go down that much. And that’s where bonds still serve that role in the portfolio.

Dan Wiener:

And do I get to jump in here with a public service announcement? All of this talk about using dividend-paying stocks as a substitute for bonds? I mean, really, it scares the heck out of me because of the risk side of that equation. Sure, those dividend yields look really nice compared to, say, a 10-year Treasury bond’s yield, but oh boy, just wait till the stock market takes a digger—that yield isn’t going to be worth very much to you.

Jeff DeMaso:

Well, it’s in the name: They’re dividend-paying stocks. It’s that stock part that’s going to matter in the bear market more than the dividend part.

Dan Wiener:

All right. More about diversification.

Jeff DeMaso:

Yeah. Let’s talk about foreign stocks because this is a big one right now. I mean, you have Warren Buffett saying, “Buy the S&P.” Jack Bogle, founder of Vanguard, famously said to just by U.S. stocks as well. So, a lot of people are wondering, “Why should I buy foreign stocks at all? And why do I need them?”

Jeff DeMaso:

Well, this is really reflective of the past decade. And we’re in the longest period of U.S. outperformance relative to foreign stocks over the past 50 years. But just rewind a little bit. If you’re back to the end of 2010—the decade of the aughts, I guess we could call them—the S&P was up 1.3% a year. It gained 14% in a decade.

Jeff DeMaso:

It’s essentially a lost decade. Foreign stocks did much better, compounding at 5% a year, up 65%. It’s not a gangbuster, but it’s certainly a good sight better than U.S. stocks. Small-cap stocks doubled in that decade. They grew at 7% a year, right? So again, the S&P is just 500 large companies in the U.S. There’s a lot more out there in the investment world, and there’s going to be periods where those 500 large U.S. companies simply trail and lag and don’t have a great return.

Dan Wiener:

Yeah. We weren’t getting a whole lot of requests for 500 index funds back then, were we?

Jeff DeMaso:

No. If anything, you saw people asking for more international. You saw companies like Vanguard and Fidelity ratchet up the amount that they had in international and their default one-size-fits-all investment portfolios.

Dan Wiener:

Right.

Jeff DeMaso:

It’s kind of chasing some of that return, I think.

Dan Wiener:

So this question: “Why don’t I just own the S&P 500?” First of all, we’ve talked about bonds being a buffer in your portfolio because stocks don’t always go up—although it’s been this incredible period for the stock market, where we’ve had just one very short bear market. I mean, it lasted like two months in early 2020. And as you have pointed out again and again and again, we haven’t even seen a 5% drawdown, or we just barely saw one since the COVID lockdown. And then foreign stocks have also lagged dramatically over the last decade.

Dan Wiener:

Again, getting people interested in the idea of “Well, why don’t I just own an S&P 500 index fund and be done with it?” But there’s a third piece, and I think it’s the active versus passive argument, right? We know the data is there. We know that most active managers lag the benchmarks. Why? Usually because of cost, and often because they’re not really great stock-pickers.

Dan Wiener:

Not everybody is above average, as Garrison Keillor used to say. But there are great active managers. It’s hard, though. And even the best active managers have some periods of underperformance. So, where do you go with active versus passive, Jeff?

Jeff DeMaso:

Yeah, I think you nailed it. Most active managers do lag the benchmark and active investing is hard. And if you go back, there’s some high-profile investors—Warren Buffett, we keep throwing his name out there. He talks about the average investor just buying index funds. The late David Swensen, who led the Yale endowment and really pioneered the idea of adopting and using private investments in a really big way. Both of them have said, “Average individuals should just buy an index fund.” And I can’t say I disagree with them. But if you look at how both Warren Buffett and David Swensen invested, they’re very active. They didn’t just buy the index fund themselves. So it’s a little bit of what do they say versus what do they do?

Dan Wiener:

Right. I think their point was: We’re real experts. We’re pros. We know how to outperform and you don’t, right?

Jeff DeMaso:

Well, but more to it, they’re saying: If you have the time, energy, discipline, access, all these pieces, then active investing can be rewarding. It’s not going to be all the time, but over time it should be. But if you don’t have all those structures in place, you don’t have the time, you don’t have the energy, the discipline, whatever it is—if you don’t have all those pieces, then yeah.

Jeff DeMaso:

If you just keep it simple, keep costs low, you’re going to come out ahead over time. And you’re just going to participate in the market and compound your wealth that way. And it’s a very effective shortcut to being in the market.

Dan Wiener:

Right. We’ve talked about bonds, we’ve talked about foreign stocks, we’ve talked about active versus passive. As two guys who spend a lot of time following what Vanguard does—Vanguard, home of the S&P 500 index fund, right? They don’t even recommend using the S&P 500, or even their broader Total Stock Market index fund, as your sole portfolio.

Dan Wiener:

What’s amazing is they’ve got these Target Retirement funds that retirement plans use—401(k)s, what have you. They own Total Stock Market funds in both U.S. and foreign markets. And they own Total Bond Market funds in both U.S. and foreign markets.

Dan Wiener:

In one sense, this is their best thinking. Or you could look at their Managed Allocation fund, which is currently what I like to think of as their “best thinking on portfolio management” because they’re buying index funds, they’re buying commodities. I mean, this is really the all-encompassing portfolio. How’s this fund done? It’s up about half as much as the market this year. This is a broadly diversified portfolio versus owning just the S&P 500 index.

Jeff DeMaso:

Yeah, I agree. I’ve got to argue in the Managed Allocation that they’ve maybe gone too far in the diversification realm without concentrating in what’s maybe the best ideas.

Dan Wiener:

I mean, they’re doing this even though they’ve bought value, they’ve taken a position in value. They’ve taken a position in commodities. Their timing wasn’t so good. But that’s the point—they are trying to diversify and do better. And in fact, in this particular year, it ain’t working.

Jeff DeMaso:

Well, yeah. Look, they’ve also taken it into a bunch of the alternatives—the Alternative Strategies, the Market Neutral fund. It’s where I think they’ve maybe gone a little too far in trying to be a little fancy with some of these alternatives. I say it all the time, but investing doesn’t have to be complicated. It’s certainly not easy, but making it more complicated doesn’t lead to a better result. And I think the Managed Allocation fund is a prime example of that.

Dan Wiener:

At Adviser Investments, we’ve been managing diversified portfolios for almost 30 years. And I began writing about Vanguard even further back than that. I don’t even want to tell people how old I am. But it was 1991.

Dan Wiener:

I’ve been in this game long enough to have heard most of what I’ll call the “advice” that comes from people who don’t believe or tend to lose their interest in diversification when one asset or one asset class is really outperforming.

Dan Wiener:

So I’ve heard that you ought to favor small stocks over big ones. And jump in here if any of these ring a bell for you, Jeff. Lately, or last year and early this year, we were hearing: “You’ve got to buy value stocks, not growth stocks.” I’ve also heard commodities are the answer. Boy, they’re the answer like one year out of 15. There was a time when these 130-30 funds that used leverage were going to beat the market. They cratered horribly, right? “Don’t buy an index fund, buy a 130-30 fund.”

Dan Wiener:

“Smart beta, which has now been renamed as factors, will give you the edge.” Even ESG, which is a whole other topic. There are periods when ESG outperforms, and there are periods when ESG underperforms. I’ve also recently heard people saying, “We ought to go all in on emerging market stocks.” Gold also gets a lot of fans out there. Energy companies… And please, how many times have you heard: “Oh, we don’t need the stock market. We just need to invest in real estate.” Right?

Dan Wiener:

The only time real estate really works for you is when you use a lot of leverage. The best thing you can do in real estate is buy a home to live in. Don’t think of it as an investment. Think of it as an investment in your lifestyle, not an investment for your portfolio. And my favorite, which I’ll leave you with, is that when markets are down, you get more and more people suggesting we have should just gotten out and gone to cash.

Jeff DeMaso:

Yeah. If only it was so easy. But that’s who we are today, right? I think that was a great rundown of all the different styles that kind of come in and out of favor. But people look at the market today and say, “Investing is easy. You just buy the S&P.” And as you mentioned, there haven’t been really any big drawdowns. It’s not that painful. Heck, right now people actually say it’s easy to get rich, just go buy bitcoin or meme stock. Not that I necessarily want to touch either of those rails right now.

Dan Wiener:

No, I don’t either.

Jeff DeMaso:

I already said I don’t think investing is easy, but making it complicated doesn’t necessarily improve things. But I think investors have to be prepared for a more difficult road ahead.

Jeff DeMaso:

We haven’t had an extended bear market, but we know that that happens. You can have drawdowns that last longer than 23 days, which happened during the COVID crisis, right? They can last months and months, and they’ll be more painful. And those times, just owning the S&P is going to look like not such a bright idea.

Dan Wiener:

Yep. Totally agree. Very well said. So, you got anything else you want to add in here, Jeff, before we take this one out?

Jeff DeMaso:

No. Again, just to try and reiterate it: The idea is you have a plan, right? Know why you own—whether it’s the S&P or another equity fund—why you own bonds, know why you have your portfolio built for the long term. And that long term should include the expectation of both bull and bear markets.

Dan Wiener:

And you should stick with it, right? Don’t try to change that horse mid-race.

Jeff DeMaso:

Yeah. The best investment strategy is the one you can stick with through time.

Dan Wiener:

Yep. All right. Well, thanks. This has been Dan Wiener and Jeff DeMaso from Adviser Investments. And I want to thank you for listening to The Adviser You Can Talk To Podcast. If you enjoyed this conversation, please subscribe and review our show.

Dan Wiener:

You can check us out at adviserinvestments.com/podcasts, and that’s “adviser” with an E. Your feedback really is always welcome. If you have any questions or topics that you’d like us to explore, please email us at info@adviserinvestments.com. And remember, that’s “adviser” with an E. Once again, I’d like to thank our editors, Kailey Steele and Ashlyn Melvin, for making this podcast a reality. And thank you for listening.