Project Success Measures: Going Beyond OTOBOS

It has been common for years to use project success measures based on the triple constraint. Reporting on whether the project is on time, on budget and on scope (OTOBOS) has strong appeal and most organizations still use this approach. But these project success measures are out of date and fail to report on what is really important—whether or not the project is delivering the desired business outcomes.

The Triple Constraint Doesn’t Tell the Whole Story

I first learned of the triple constraint back in my first PM class back in the early 1990s. I learned that the three constraints of time, scope and budget formed the basis of every project. And I leaned that if any single constraint changed, the other two had to be evaluated for impact.

Though it was simple and perhaps convenient to consider only those three constraints, most people today recognize that there are more than three constraints in a project (resources, quality and risks are also frequently mentioned). Unfortunately, most people still use just cost, time and scope as a project success measure or when reporting project status. People have limited their definition of success or failure to whether the triple constraint was managed. And this misses the entire point – OTOBOS is not the goal of the project.

OTOBOS is Not the Project Goal

With all the focus on OTOBOS, many people think that it is the goal of the project. Many project managers certainly believe that is their goal to deliver the project on time, on budget and on scope. They have lost sight of the real project goal—and that is delivering some sort of business value. That should be our first goal, and not an afterthought.

PMI seems to be slowly addressing this. A Guide to the Project Management Body of Knowledge (PMBOK^® Guide) has included guidance on success and benefits management measures. In my experience, these measures are rarely put in place. Very few people are actually keeping an eye on the achievement of benefits—it is a second-class citizen compared to OTOBOS.

The most recent PMI Pulse of the Profession acknowledged that only now is benefit realization being included in success measures. For the first time in 2017, PMI included benefits realization maturity along with the traditional project success measures in Pulse of the Profession[i].

Perhaps measuring benefits as a component of success isn’t more common because the realization of project benefits often lags the completion of the project. Unless the PMO or other organization is measuring it, it may not happen since teams are quickly disbanded and project managers are typically reassigned.

That is a big problem—the separation in time of the execution of the project and the realization of the benefits. The longer the timeframe, the higher likelihood that people will lose interest. The original assumptions may be forgotten. Or, so many other things changed in the environment that the original benefits can’t be accurately measured.

Rarely if ever does anyone go back after the end of the project to see if the promised business benefits were achieved. In fact, I’ve seen large cost-reduction projects that were supposed to result in headcount reductions that resulted in more people afterward!

Note: It is also a problem that the people tracking the realization of benefits (the PMO) are separate from those responsible for delivering it (the PM and project team).

OTOBOS and Business Value Measures May Conflict

There are situations where our project delivery measures like OTOBOS are exactly opposite the business values measures. Consider the following two examples from Jim Highsmith’s book, Agile Project Management[ii]:

1. The production of the Titanic movie in 1997 was significantly behind schedule and delivered at a cost nearly double the original budget. By any project management standard, this project was a complete failure. Yet the movie earned over $2B in revenue and was the highest grossing movie in history (until surpassed by the movie Avatar). It also earned 11 Academy Awards, which brought additional attention and prestige to the studio and actors (as well as numerous other intangible benefits).
2. The Motorola Iridium Project had the reverse problem. Iridium was a phone system based on the launch of 66 satellites. The project was considered a success from a project delivery perspective (on time, on budget and on scope). But the business value was never achieved from the original project because of faulty assumptions. Motorola spent nearly $5B on the project before declaring the project bankrupt and selling it off for pennies on the dollar.

So, it is possible to have projects that are successfully delivered and fail to deliver benefits—and projects that fail to be successfully delivered and yet yield business benefits. What is success then?

Similarly, some organizations measure “scope creep” and view that as a negative. But if a change in scope serves to improve the benefits of the project, shouldn’t that scope change be valued? Wouldn’t it be more important to change our plan to achieve our benefits, rather than simply adhere to the plan?

We Shouldn’t Trust Our Assumptions about Business Value Attainment

The Iridium problem of faulty assumptions is one that should have been uncovered early in the project. That particular project was planned in detail over the course of several years! There was plenty of time to validate the assumptions about the business value.

The problem was that the team believed that if you built it, they would come. This Field of Dreams thinking is problematic. We cannot assume that just because we deliver a project that it will result in business benefits. We should not trust our assumptions about business values—they are frequently wrong. And focusing on OTOBOS is a distraction from what is really important.

The First Moonshot Project

In 1961, President Kennedy proposed to the Congress that the United States should put a man on the moon before the end of the decade. The unstated reason was dominance in science and technology and a response to a threat from others. The Soviet Union had successfully launched a man into space just weeks prior.

This was a massive undertaking (what some might call a “Big Hairy Audacious Goal”). It took nearly a decade to accomplish it. There were lots of assumptions about the feasibility of this project. The term moonshot was coined from this mission.

No Moonshots Today

Most organizations today try to avoid moonshot projects. While big goals are important, few want to bet the company on a make-or-break project. Betting the company on a moonshot project like Iridium can result in the death of the company.

Instead, organizations take calculated risks. They make assumptions about the benefits that they can get by implementing a project. And they continually test those assumptions.

The Lean Startup Method

The 2005 book The Lean Startup describes this process of challenging and validating assumptions. In the Lean Startup Method, there is little or no focus on traditional project measures like OTOBOS. Instead, there is a continuous focus on learning and on validating the assumptions about business benefits—and continuous monitoring of those business benefits. If the business goal is to increase customer revenue, the organization continuously measures customer revenue and tests different actions to determine what it takes to get that revenue.

Agile Projects Use Different Project Success Measures

Similarly, agile projects focus on delivery of business benefits. They leverage incremental and iterative development based on a “product backlog” of business-related features. Every product backlog item should have business value. Agile projects follow a “build, measure and learn” cycle to constantly inspect and adapt. For those projects where the outcome is fuzzy or the path to get there is risky or uncertain, it is important to have many of those build-measure-learn cycles.

Using traditional OTOBOS measures with agile projects is ineffective and wasteful. Measuring variance from the plan is a distraction, and trying to minimize variance may be futile. Rather, the focus should be on measuring the achievement of business value. This is often referred to as value-driven development.

Recommendations for Project Managers

Given the above, what should project managers do—especially when organizations seem to be rigid about reporting projects using OTOBOS measures? I’ve outlined five recommendations that I believe may be helpful:

1. Eliminate big moonshot projects. Those moonshot projects with big goals, long time frames and all the benefits delivered at the end are risky and irresponsible. Break all big projects down into smaller ones. Utilize short build-measure-learn cycles. Smaller projects are significantly more likely to succeed than big ones based on Standish Group Chaos Studies[iii].
2. Shorten and improve feedback loops. This idea goes with smaller projects. The greater the separation between project execution and the realization of benefits, the less opportunity to make adjustments. Scope changes and scope creep may be exactly what is needed mid-project in order to maximize business benefits.
3. Deliver value continuously. When breaking up big projects into small ones, make sure each small project delivers some type of business value, rather than deferring all the benefits to the end. This may require using iterative and incremental delivery rather than a phased approach.
4. Recognize that delivering OTOBOS is not the goal. I credit agile expert Craig Larman for teaching me this valuable idea. The plan is not the goal! The project manager/team’s goal is not to deliver the project on time, on budget and on scope. The goal is to deliver the business value of the project—the desired benefits. We shouldn’t take our eye off the delivery of those benefits.
5. Constantly test and validate assumptions. Project managers should design the project to test and validate assumptions about the realization of the business value. Most assumptions sound like “if we do this, then we achieve that.” That is assumed and not a guaranteed outcome of the project. Don’t blindly trust the assumptions. Instead, find ways to validate the assumptions throughout. Focus on the project activities that deliver the designed business outcomes.