An Investment Forecasting Exercise, and How It Morphed Into a Endless Quest For Data
About 4 years ago, our investment office went down a forecasting / decision analysis rabbit hole. The exercise started as an effort to quantify and measure our forecasts. It quickly morphed into a never ending search for data sets on the typical outcomes investors realize.
At its heart, investing is an exercise in predictions
Investing is predicting possible future outcomes (i.e. base case, worst case, upside case). You compare those predictions to current values and attempt to make good investments: those with good base cases, or good upside and limited downside. You constantly make predictions and are frequently wrong.
We implemented a new required section in our investment memos: a prediction of the full range of future outcomes, from 1% worst (1 in 100 chance), to 1% best (1 in 100 chance). Explicit and quantiative predictions.
What happened is it quickly turned into an exercise in base rates.
What the heck is a base rate?
It’s an idea with multiple names: “base rate” or “outside view”
Many have written or talked about this idea. Michael Mauboussin described the concept nicely on this recent podcast
There’s this idea that Daniel Kahneman popularized called the inside versus the outside view.
So the inside view says, when I pose a problem to you, the natural way we all solve it, and by the way, we all do this, this is our natural way of doing: we gather information. We combine it with our own input and experience. So we’re melding this information we’ve gathered. And by the way, we may not be gathering all the information we need, but we gather information, we combine it with our own experience, and then we project into the future. The outside view says, I’m going to think about my problem as an instance of a larger reference class. I’m going to ask a broader question, which is what happened when other people were in this situation before?
That’s a very unnatural way to think for two reasons. One is you have to leave aside what you’ve done, your effort, and your own experience. You almost have to discount your own experience and your own input. Second is you have to find and appeal to the base rate, which is the outside view, which may not be at your fingertips.’
Applying base rates to investment predictions
An example of applying this idea to an investment prediction:
Inside view: I believe this stock has a good management team, a strong market position, and I predict it will grow revenues by 20% per year for the next 10 years.
Outside view: How many similar stocks have existed, both in the past and the present? How many of them grew by 20% over their subsequent 10 years? What was the range of outcomes? Would 20% be typical or unusual relative to this history of similar companies?
This idea could also apply to investment funds:
An example outside view of a venture capital fund.
What’s been the range of outcomes associated with venture funds? How many fail to return capital. How many have positive returns? How many triple your money?
Our prediction template and how it works
The template is drawn as a chart. And the prediction is also stored in database form. Below is a simple example. Say an analyst predicts a base case of 10% return, a low case of 5%, and a high case of 15%. Say the analyst predicts a 33% chance of any of those cases. This is the visual form of that prediction: From 1% to 33% of the time, 5% return. From 33%-67% of the time, a 10% return. And from 67% to 99% of the time, 15% return.
And it’s all tied to a date: 5 years in the future in this case.
Actual predictions tend to be more granular. This prediction was made about 4 years ago. By 12/31/20, returns of 14–26% were predicted (25–75% of the time). About a 25% chance of being flat (0% return). By time stamping prediction dates, we can look back to actual outcomes. In this example, we got smashed on foreign currency depreciation.
With a good sized dataset of predicted and actual outcomes we can examine different questions:
If outcomes are lower than predictions, we err by being too optimistic
If outcomes are higher, we are too conservative
If outcomes come in way higher or way lower than our best or worst case scenarios, our forecasts are too narrow.
That was the idea, but once we started doing this, the exercise quickly morphed.
What started to happen
Imaginations expanded
Once we had to write down a 1% and 99% outcome, we started searching for data on those extreme outcomes. No one wants to write down. “This has a 1% chance of losing X” and later face a future where that investment lost more than X. Imaginations expanded in a generally healthy way.
“Failure comes from a failure to imagine failure” — Attributed to Josh Wolfe
Forcing a full range of future possible outcomes forced more contemplation of failure, and also of outrageous success.
It kicked off an endless search for data
For every investment, we were forced to think about the broad class of similar investments. How frequently do buyout funds lose money? How often to single coinvestments lose money?
For reference, about 7% of our buyout funds have lost money, 33% of our coinvestments, and 38% of our venture funds. In aggregate all have outperformed public markets, but sometimes the skew is extreme.
If we had enough internal data, we started studying it in depth. If we didn’t have enough internal data on a topic, we started to look to outside.
It kicked off a search for more timely feedback
Long dated investments have long dated feedback cycles. Where do we have shorter cycles? We make shorter term decisions all the time actually. An example: Most election decisions we have to make are short term predictions.
One simple election decision. We had a sidepocket linked to the value of Greensky stock. We could elect an early exit or hold to a liquidity event at the end of 2020. It forced an explicit prediction of the future $GSKY price. A discussion that could have been squishy and qualitative became more structured. What do we think this stock will trade at in roughly 2 years, and what has to happen for each scenario?
Another example: A legal claim. We had an asset worth 100 cents on the dollar, but accruing legal and admin fees while litigation was pending. Theoretically we could lose the litigation and the asset would be worth 0, or close to it. A buyer made an lowball offer for our claim. It forced an explicit prediction of the future value. That prediction: 30% chance of a 3 year time to realization. 69 cent recovery after costs. 70% chance it goes 5 years, and 45 cents ultimate realization. In this case, the analyst assigned 0% of not winning the claim, but a possibility of high attritional losses.
Ongoing search for better data
Our quest for base rates and outside views continues. For example, I’m skeptical of private equity continuation vehicles. This is where a fund is near end of life, but a company remains where the manager wants to continue in hopes of more future value. These are companies that have been worked on by professionals for years and haven’t gone according to plan. At the end, these deals go through competitive bid processes. I hypothesize that the assets have less potential, and the valuations are high, and that investors are better off selling… but… data is wanting. We’ve made a few decisions on these offers in recent years. But our number of data points is modest, and hard to use for hard conclusions. I wish someone like Pitchbook, Prequin, Cambridge Associates, or an academic like Josh Lerner would take up a study.