Demystifying the Fine Art of Cloud Spend Forecasting

As IT organizations embark on aggressive cloud migration and development programs, they inevitably must set expectations regarding the financial benefits of those programs. Cost savings are usually the most measurable benefit and very often the financial driver of the business case. Each cloud migration business case must forecast 3 things:

1. The cost of the migration project
2. The expected run cost of the current deployment mode
3. The expected run cost in the cloud

Cloud program credibility hinges on the accuracy of cloud spending forecasts, made during the business case exercise and repeated throughout each year after deployment. Cloud forecasting should address multiple timeframes:

• Monthly (to identify spending anomalies)
• Annually (to support budgeting)
• Multi-year periods (to support strategic planning)

The annual cloud forecast will be a key driver of the cloud budget for the next year. Performance against this budget (“budget variance”) will set management expectations regarding whether cloud programs are on track.

Most companies struggle with cloud spend forecasting because the drivers of future cloud spending are different than on-premises spending in 3 fundamental ways:

1. On-premises spending is driven by a capacity plan, determined in advance based on long-term demand projections. Spending is upfront (usually as capital investment) and will constrain the scalability of each application. In the cloud though, capacity is scalable in real-time and spending is directly tied to demand.
2. On-premises, spending is very predictably represented by depreciation and amortization. In the cloud, companies do not purchase equipment, they subscribe to services (sometimes on a “per minute” basis), making spending much harder to predict.
3. The pace of change in cloud environments – from new migrations to new development to changes in existing footprint – is much more rapid than on-premises. Forecasting and budgeting must therefore stay ahead of all these changes to have any hope of accuracy.

We’re going to quickly walk through what we see as the main challenges that IT organizations face in maintaining accurate cloud spend forecasts. We will separate these challenges into three categories:

• Application-specific forecasting
• Portfolio-level forecasting
• Forecasting governance

IT organizations need to systematically address challenges in each category to achieve the level of cloud maturity necessary to maintain executive support for their cloud program.

Challenges with forecasting at the application-specific level require new thinking along three dimensions:

• First, forecasts must consider architectural changes necessary for an application to run with equivalent performance as on premise. Very often, with a “lift and shift” migration, teams simply assume that infrastructure configuration should replicate the on-premises equivalent. However, cloud application performance may need to be tuned by increasing resources (e.g., CPU and memory) beyond what was needed on premise, causing significant variances between projected and actual cloud run costs.
• Second, application forecasts must consider potential benefits of cloud commercial options, including use of savings plans, reserved instances and software license plans such as Azure Hybrid Benefit.
• Third, application forecasts must consider projected change in demand once the application is available in the cloud. IT analysts must assess potential drivers of demand and decide what forecast model the application requires – e.g. constant usage, linear growth, or demand correlated to exogenous factors. Forecast models will need to be adjusted as experience allows better insight into demand patterns.

Accurate measurement of application-specific spending will be necessary to enable fine-tuning of application forecasts. This will require careful tagging of environments and resources, as well as the ability to attribute reserved instance and savings plan usage and allocate shared resource usage to individual applications. We’ll talk a little more about this last challenge in the next section.

Executive monitoring of cloud spend does not occur at the application level; companies aggregate cloud spending by business unit or function and manage spending across a portfolio of applications. Unfortunately, simply aggregating application-specific forecasts is not enough to ensure accurate cloud portfolio forecasts. Cloud portfolio forecasts must consider:

• Allocation of shared resource spending (mentioned earlier)
• Layering of new spending into the forecast
• Ongoing management of forecast and budget variances

Corporate IT has always struggled with the allocation of shared costs across business organizations. While cloud options have enabled the transition from cost allocation to consumption-based charging, inevitably, some applications and resources will be shared across organizations and costs must be allocated. In these circumstances, IT should first minimize the percentage of spend that must be allocated (tag as much as possible), and second, choose allocation variables most aligned with actual resource consumption (we’d suggest variables such as users or minutes of use).

Within an organizational cloud portfolio, we like the concept of “layers of spend.” The foundational layer is the aggregate forecast of all operational cloud applications. The second layer of spending should include all new applications already approved for migration or development, using run-rate forecasts from their business cases. The final layer of spend represents applications not yet approved. Incorporation of this layer into the budget enables new approval decisions to be made throughout the fiscal year without additional budget.

The third key component of mature portfolio-level forecasting involves variance analysis. Every month, the difference (“variance”) between actual spend and budget or forecast must be understood, and the appropriate corrective actions should be taken. While budget variance should be tracked at the application level, it should be managed at the portfolio level.

As actual demand for applications is understood, budget can be moved within the portfolio from low demand or low priority applications to higher priority applications. Budget can also be moved between existing applications and future applications. Controls can be put in place to throttle demand for applications that are not business critical. And finally, budgets can be increased (or decreased) if the increase is in the best interest of the company (with positive Net Present Value).

We refer to the final category of challenges as “forecasting governance,” included in the overall governance of the cloud program (we are not suggesting a “forecasting steering committee”).

Management of cloud spend requires close collaboration between business users, application teams, and the infrastructure organization. In many cases, the infrastructure organization is responsible for cloud spend in their budget, but business and application teams make the decisions that drive that spend.

A proper governance model ensures that all parties work together in optimizing cloud spending while still meeting performance requirements of each individual application. Governance should clearly define roles and accountabilities across organizations, specify communication protocols between groups, and define appropriate management reporting and dashboards. The governance model should raise visibility and awareness into how business decisions are translated into cloud spend.

Across all these areas, companies need to recognize that cloud spend forecasting must be done differently from previous financial forecasting. Skills, systems, processes, and governance must change to accommodate these differences. However, these challenges are worth addressing. The promise of the cloud is real and the benefits are within reach. We simply cannot underestimate the change required to realize both the savings and performance improvements potential.