Trust your numbers, but never trust your gut.

Always trust your numbers. Never trust your gut.

Those are the words of Daniel Kahneman, the economics Nobel laureate. In an video interview series with Inc. Magazine’s Editor in Chief, Eric Schurenberg,  Kahneman tells us that experts tend to be overconfident, and make predictions without enough information. We are also inconsistent decision-makers. Here’s the story.

Only trust your numbers if you know they are credible. If you have high quality, credible data and interpret it correctly and with rigor, you will avoid the traps of overconfidence and inconsistency, and make rational decisions. But beware, credibility has two parts: the quality of your data and which data points you decide to use in your assumptions. Depending on these inputs, you can derive very different outputs. Remember that “Garbage in, Garbage out” (GIGO) still applies.

As luck would have it, I couldn’t reach Professor Kahneman today, but I did speak with my colleague Cal Braunstein, CEO Executive Director of Research at The Robert Frances Group (RFG). Cal and I are working on a Return on Investment (ROI) calculator project. It is the embodiment of Kahneman’s thesis of numbers over gut. I wanted Cal’s take on ROI, Total Cost of Ownership (TCO) and related approaches to making decisions by the numbers. Should you trust your numbers? It all depends…Here is what Cal had to say.

Which project “go no-go” approaches are RFG clients using?

To determine whether it makes sense to “do” something, our clients have traditionally considered Total Cost of Ownership (TCO), ROI, Internal Rate of Return (IRR) and Payback Period. Today, they are also looking at agility (time to market) and how fast they might make mid-course development changes. Our product vendors want to know how they can better define their value proposition.

To read more about lean start-ups and minimally viable products, look here.

Do clients start with one measure and evolve to use others?

Yes, you may find people starting with ROI because it is a relatively simple calculation. If the ROI you calculate  doesn’t meet your hurdle rate than there is no value in further exploration. If the ROI does meet the hurdle rate, then clients will drill in with a TCO to get a true feel for the costs.

Does the role of the evaluator influence their analysis?

Yes, another factor in choosing the analysis technique is who is doing the analysis. For example, finance people will do an ROI, but they have no real idea of  the nitty-gritty details that go into a useful TCO.

It turns out that operations people are often more comfortable with TCO because they have real insight into the costs involved in building products, developing and offering services, or both. Cal cited a case of “go no-go” for adopting a new software suite.

The finance team performed an ROI to get a high-level view of the impact of acquiring the proposed software suite. It passed their testing, and the analysis task was passed to IT and Operations. These groups had us develop a TCO that considered re-training costs, staff back-fill, and the software vendor’s consulting fees, among other factors we knew would influence the cost of the project. In the end, the TCO revealed the new software acquisition would be a wash when compared to sticking with their existing solution. The acquisition was nixed, at least for the time being.

Why did your end-user clients ask RFG to do the TCO analysis?

Now we all know that TCO, ROI, and the other typical analysis techniques have been around a long time. I thought it likely that both product vendors and enterprise clients could perform their own analyses. So, I asked Cal why clients would reach out to RFG. Do they trust Cal’s numbers more than their own?  It turns out the answer is “Yes.”

Often. analyses are used in an enterprise to make a point. As a third-party, RFG doesn’t have a point to prove. We simply want to get the analysis right. When we do the analysis, our clients know that what we deliver is unbiased and therefore more credible.

How important are the assumptions you make?

Well, they are extremely important. SWAG assumptions tailored to meet outcomes are no good to anyone. Making unrealistic or poorly informed assumptions will probably lead to incorrect conclusions. The idea is to help your client make a rational decision, not to game the analysis outcome to produce a particular result.

Do you have an example?

OK, think about the assumption of electrical costs for a data center.If we think electricity costs 8 cents/kwh but in some other part of the country it is really 12 or 16 cents, the decision one would make on investing in energy-saving devices would likely change. If the ROI or TCO analyst only considers what he or she pays per month, but does not look at rates in other parts of the country,  or in other countries, the analysis will have a built-in weakness.

 Another one, perhaps?

Sure. Another example concerns the operating characteristics of a server. If your assumption is that your servers run 4 hours per day and are otherwise off, but they run 4 hours per day and are older servers without multiple levels of powered down states and stay, you will make an incorrect assumption, and probably come to the wrong conclusion.

What about product and service vendors ?

The value of RFG for vendors is that nobody trusts the vendor’s numbers. When the vendor tells you “use my system and save millions,” who would believe that? A vendor’s homegrown analysis is designed to show their product or service in the best possible light. An unrealistic assumption set may sell a solution but then fail to deliver for a user. However, a report from an analyst firm like RFG informs readers about the analyst’s belief. There is no question that our work and our assumptions are transparent, and our results are credible.

 The Bottom Line

With any analysis that influences your decision-making, it is crucial to dissect  the analysis to know if it is based on reasonable assumptions, and that the recommendations offered are credible and suitable to support your decision. In Dr. Kanheman’s video, he states

…if the algorithm is based on optimistic assumptions, the algorithm is not going to work.

Decision-making is a bias-prone and complex set of activities that engage your analytic and organizational abilities. TCOs, ROIs, and similar analysis vehicles can offer valuable insights, but the smart decision-maker considers the source and the method. Make sure you have confidence in the quality of the data used to prepare the analysis. Make sure you understand the assumptions made in the analysis, and make sure they make sense for your situation. Look before you leap.

Cal Braunstein and the Robert Frances Group team have decades of experience in preparing analyses and critiquing those prepared by others. Isn’t It time to get Cal and the RFG team working for you? Contact them now, and get your ball rolling.