The performance of the Eurosystem/ECB staff macroeconomic projections since the financial crisis

Prepared by Kyriacos Lambrias and Adrian Page

Published as part of the ECB Economic Bulletin, Issue 8/2019.

This article evaluates the performance of the Eurosystem/ECB staff macroeconomic projections for the euro area in the context of the elevated macroeconomic volatility and uncertainty that has prevailed since the financial crisis. It finds that there has been considerable variability in projection errors over time. With regard to real GDP growth projections, errors that were substantial during the sovereign debt crisis have become more limited in recent years. As for headline inflation, unexpected fluctuations in oil prices – which in the staff macroeconomic projections are assumed to follow the path of oil price futures – played a dominant role in explaining the errors, as was the case during the pre-crisis years. On the other hand, HICP inflation excluding energy and food has been persistently overprojected since 2013. While these projection errors can also partly be attributed to errors in the conditioning technical assumptions, other factors (such as modelling errors, changes in economic relationships or judgement) have also played a key role at different points in time. The forecast performance of the Eurosystem/ECB staff macroeconomic projections has been broadly similar to that of other international institutions and of private sector forecasters, suggesting that projection errors have been mainly driven by common elements. These may include economic shocks unforeseeable to any forecaster and developments that have become more prominent since the financial crisis, including, among other things, structural reforms, changes in the relationship between slack and prices, globalisation and digitalisation. The article is structured as follows: Section 1 explains how the staff macroeconomic projections are constructed/compiled, Section 2 provides an overview of the errors made in projecting real GDP and HICP inflation since 2010,^[1] Section 3 reviews some of the sources of the errors, and Section 4 provides a comparison with the forecasting performance of other institutions and private sector forecasters.

1 Introduction

Every quarter, the ECB publishes projections of real GDP growth, inflation and a range of other macroeconomic variables which serve as a key input into the monetary policy decision-making process.^[2] In March and September these staff macroeconomic projections are produced by ECB staff, while in June and December they are produced jointly by staff from the ECB and all of the Eurosystem national central banks. The projections are published for the current and the two subsequent calendar years^[3] and serve as a coherent reconciliation of all available information based on a wide range of econometric models and on conditioning technical assumptions (such as for oil prices, exchange rates and interest rates), which can also be supplemented by staff judgement. It should be noted that the projections are neither endorsed by the Governing Council nor do they necessarily reflect the views of the Governing Council on the outlook for the euro area. Instead, they reflect the views of Eurosystem/ECB staff. Nevertheless, they serve as a key input into the monetary policy decision-making process and it is important that they provide reliable indications of the most likely course of future economic developments.

Regular evaluations^[4] of the performance of the projections serve two main purposes. First, identifying patterns in projection errors and understanding the reasons why such errors were made can help to improve future projections. For example, persistent errors with the same sign may imply that the econometric models used to produce the projections are not capturing some of the structural changes in the economy and that the models or the tools used for the projections may require some adjustments. Second, past projection errors can provide an indication of the degree of uncertainty surrounding the projections at various horizons ahead.^[5]

2 Overview of errors in projecting real GDP and HICP inflation

GDP growth was strongly overpredicted during the sovereign debt crisis, but the accuracy of projections has improved in recent years. Chart 1 shows the projections of annual average real GDP growth in each of the quarterly staff macroeconomic projection exercises since 2010, together with their outcomes. For each year, twelve projections are shown, starting with the projection produced in the first quarter two years prior to the reference year and ending with the projection produced in the last quarter of the reference year. The projections produced in the last quarter of the year preceding the reference year are indicated with a dot (e.g. the projections produced in the fourth quarter of 2014 for the year 2015). This is the last projection exercise for which no hard or soft indicators are available for the year being projected.^[6] Since economic data, in particular GDP estimates, tend to be revised, Chart 1 shows two outturns: the vintage of the outturns available one year after their initial publication and the latest vintage. As would be expected, in most cases, the projections became successively more accurate as the projection horizon was reduced and as more information became available. An exception is 2010, for which the initial projections produced in 2008 were quite accurate, but were revised downwards heavily with the onset of the financial crisis in 2009 and then revised upwards again as the recovery took hold. The largest errors after 2010 came with the intensification of the sovereign debt crisis between 2012 and 2013, which led to much weaker growth than expected. Over the subsequent years, although the accuracy of the growth projections has been much better, there has been a tendency to underpredict growth, particularly for 2017. This tendency was exacerbated by persistent upward revisions to official data (as illustrated in Chart 1 by the gap between the outturn one year after initial publication and the latest vintage).

Eurosystem/ECB staff projections underpredicted HICP inflation in the early part of the sample and overpredicted it between 2013 and 2016, but HICP inflation projections were broadly accurate in more recent years. Chart 2 shows the projections and outturns for HICP inflation since 2010. The projections made following the financial crisis significantly underestimated inflation, which was expected to fall more significantly in response to the additional economic slack during this period, a feature common across advanced economies at the time.^[7] In the ensuing years, particularly between 2014 and 2016, inflation surprised persistently on the downside. In each year, initial projections were around 1.5%, but they were persistently revised downwards, in particular in the light of sharp declines in oil prices, which led to outturns of around 0.2% to 0.5% between 2014 and 2016. More recently, in 2017 and 2018, inflation outturns have been more in line with the projections.

3 Sources of projection errors

This section reviews some of the key factors behind the projection errors for growth and inflation. The Eurosystem/ECB staff macroeconomic projections are conditioned on a number of technical assumptions which can play an important role in explaining the errors that have occurred. The main assumptions are those related to commodity prices (such as oil prices) and financial assumptions (such as interest rates and equity prices), which are based on market expectations, together with assumptions on exchange rates, which are held constant over the projection horizon.^[8] Euro area projections are also dependent on the staff assessment of the outlook for the international environment, which can be an additional source of errors. For the purposes of this article, errors are defined as the outcome minus the projection, so a positive error indicates a higher than expected outcome (i.e. an underprediction), while a negative error indicates a lower than expected outcome (i.e. an overprediction).

Errors in the projections for international developments and the technical assumptions explain only a part of the projection errors made for real GDP growth. Chart 3 decomposes the projection errors in the March projections for the following calendar year. Since official GDP data are normally available up to the fourth quarter of the previous year, this corresponds to a two-year projection horizon. Panel a uses elasticities from Eurosystem macroeconomic models to decompose the errors into what can be explained by (i) errors in the conditioning technical assumptions; (ii) errors in the outlook for the international environment; and (iii) all other factors. The decomposition reveals that errors in technical assumptions played a limited role in explaining errors in the GDP growth projections, with the exception of 2015, when a sharp drop in oil prices gave an unexpected boost to growth, and 2018, when the appreciation of the euro exchange rate in the course of 2017 and early 2018 weighed on exports. Errors in international projections (i.e. euro area foreign demand) explain part of the underpredictions of growth for 2010 and 2017, which were years in which euro area exports grew strongly. Global trade was persistently weaker than expected between 2012 and 2016, but the impact on the growth projections was partly offset by other errors with an opposite sign arising from the technical assumptions. The overpredictions of global trade, particularly for emerging market economies, was partly related to a structural shift in the trade intensity of global economic activity. Such structural shifts are difficult to anticipate in real time, but projections since 2016 have been anchored around the view that – over the medium term – global imports will grow at broadly similar rates to global activity.^[9] Since that year the projections for global trade have been more accurate. “Other factors” in Chart 3 refers to all other sources of errors, including model misspecification, domestic factors, such as fiscal, structural and monetary policies (beyond the effects captured in the technical assumptions), and judgements introduced by staff. In most cases, the errors in the international projections move in the same direction as the other errors, which may reflect indirect effects beyond direct trade channels. For example, shocks to the global outlook could also have had a negative impact on domestic confidence, leading to lower domestic demand.

Turning to the errors made in the projections for the expenditure components of GDP, there were broad-based overpredictions during the sovereign debt crisis, while recent errors were concentrated in net exports. Panel b of Chart 3 shows a decomposition based on the contributions of the projection errors in the expenditure components to the overall projection error for real GDP growth. In 2012 and 2013 all domestic demand components turned out to be weaker than expected, while the negative surprise on imports was larger than the negative surprise on exports, leading to a more positive than expected contribution from net exports. In the subsequent years, domestic demand components were somewhat stronger than expected, supported by the oil price declines mentioned above and also by successive monetary policy packages which eased financing costs for households and firms. Most recently, the projection errors were dominated by the contribution from net exports, which surprised on the downside in 2016 and on the upside in 2017 and (to a lesser extent) 2018.^[10]

As regards inflation, errors in the conditioning technical assumptions, particularly for oil and exchange rates, play a significant role. Chart 4 shows similar decompositions for the projection errors in inflation. As seen in panel a, in general, errors in assumptions explain a larger proportion of the errors in inflation projections than was the case for the growth projections. In particular, large unexpected swings in oil prices played a dominant role in the underpredictions of HICP inflation in 2010-11 and 2017-18 and in the overpredictions in 2015 and 2016. Fluctuations in the exchange rate also contributed to the projection errors. In particular, a large depreciation of the euro in 2014-15 offset part of the downward impact from the lower than expected oil prices in 2015. Apart from the contribution from the technical assumptions, there remains a contribution from other factors which, since 2013, has been persistently negative, albeit rather small over the past two years.

Outcomes of HICP inflation excluding food and energy have surprised persistently on the downside since 2013. Panel b of Chart 4 shows the decomposition of errors in the inflation projections according to the main HICP sub-components and shows that the volatile food and energy components explain the majority of these projection errors, largely driven by the errors in technical assumptions. The projection errors in HICP inflation excluding energy and food, although of a much lower magnitude than those for the volatile components, have shown a persistent negative bias (i.e. overprediction) over the past six years. Errors in technical assumptions may have played some part in generating these errors, such as indirect effects of unexpected declines in oil prices over recent years on non-energy consumer goods and services as well as the appreciation of the euro in 2017 and early 2018. Nevertheless, even after accounting for the errors in technical assumptions, there still remains a persistent overestimation.

A number of explanations have been put forward for the persistently and unexpectedly low underlying inflation over recent years.^[11] There may have been an underestimation of the amount of slack in labour and product markets, which may have contributed to lower than expected wage growth and thus reduced cost pressures on prices. Another explanation is related to “pent-up restraints”, meaning that, while declines in prices and wages during the sovereign debt crisis were limited by downward nominal rigidities (contributing to the higher than expected inflation outturns during that period), in the subsequent years, price and wage inflation may have been correspondingly subdued.^[12] It may also have been the result of a strengthening of the relationship between slack and prices, i.e. a given level of slack may have resulted in a stronger downward impact on price developments. This may, for example, be a reflection of the role and impact of structural reforms in some countries which were designed to lower nominal rigidities. A further explanation relates to the role of global influences on domestic inflation, which could extend beyond what is captured by the import and commodity prices typically considered in the staff macroeconomic projections. Low inflation might partly reflect increased trade flows, the integration of emerging economies into the world economy and the rise of global value chains shifting parts of production abroad.^[13] Within the Eurosystem, a dedicated working group is tasked with improving the models used to produce the projections by looking at the latest advances in forecasting techniques, performing regular analyses of forecast errors and identifying and modelling structural breaks and non-linearities.^[14] Eurosystem staff have also tried to address the large contribution of errors in technical assumptions by examining alternative ways of projecting oil prices, rather than using oil futures prices. An approach using a combination of oil price models was found to perform relatively well over some sample periods. However, so far no method has been found which can consistently outperform oil price futures over longer periods.^[15]

4 Comparisons with other forecasters

The Eurosystem/ECB staff macroeconomic projections for growth and inflation have performed similarly to those of private sector forecasters and other international institutions. Chart 5 shows the root mean squared forecast error (RMSFE) and the bias (average error) of the projections for the next calendar year produced by Eurosystem/ECB staff compared with the ECB’s Survey of Professional Forecasters (SPF), the European Commission, the International Monetary Fund (IMF) and the Organisation for Economic Co-operation and Development (OECD) – in each case taking the forecasts produced in the second quarter of the year. Different institutions publish at different times during the second quarter, with the Eurosystem staff macroeconomic projections being the last to be published (in June), implying that the latter may have some comparative advantage due to the availability of more recent economic data. In order to adjust for this, the figures shown for the Eurosystem/ECB staff macroeconomic projections are the average RMSFE and bias for the projections published in March and June of each year. Chart 5 shows that the forecast accuracy since 2010 has been rather similar across forecasters and that they share a similar (low) degree of bias. For growth, the RMSFE in the Eurosystem/ECB staff macroeconomic projections was above that of the SPF (indicating a worse performance), but below that of the European Commission and the IMF and similar to that of the OECD. Over this period, the average error for growth was low for all forecasters, albeit slightly negative (indicating overprediction), ranging from 0.0 percentage points for the IMF to -0.16 percentage points for the OECD, with the Eurosystem/ECB staff macroeconomic projections at -0.08 percentage points.

For inflation, the RMSFE of the Eurosystem/ECB staff macroeconomic projections was equal to that of the SPF and below those of the other forecasters, implying a better performance for the Eurosystem/ECB staff macroeconomic projections. The average error of the Eurosystem/ECB staff macroeconomic projections was the closest to zero (-0.04 percentage points), indicating the lowest bias, while the largest bias was for the European Commission and the SPF (-0.12 percentage points).^[16] Nevertheless, as can be seen in Chart 6, all institutions and the private sector made inflation forecast errors consistently in the same direction over the period between 2010 and 2018, albeit at different magnitudes. That is to say, all forecasters underpredicted HICP inflation between 2010 and 2012 and in 2017 and 2018 (positive forecast errors), but inflation surprised everyone on the downside between 2013 and 2016 (negative errors). This could suggest that the source of the error was common to all forecasters, which could include fundamental economic shocks that were unforeseeable and/or changes in economic relationships, but is unlikely to be related to errors in judgement. Panel c of Chart 6 also shows that the persistent overprediction of HICP inflation excluding food and energy in Eurosystem/ECB staff projections over recent years was a feature they had in common with the forecasts produced by other institutions. The performance of the Eurosystem/ECB staff macroeconomic projections for GDP and inflation over a much longer sample was studied in a recent ECB working paper and the results are discussed in Box 1.

5 Conclusions

Overall, the article suggests that the Eurosystem/ECB staff macroeconomic projections have performed well over recent years and in line with projections of other institutions. At the same time, continuous efforts are needed to improve the projections by (i) exploring alternative methods that could increase the accuracy of the technical assumptions; (ii) ensuring that the economic models used for the projections follow the latest advances in forecasting techniques; and, importantly, (c) better evaluating or anticipating fundamental changes in economic relationships resulting, for example, from the impact of structural reforms, globalisation and digitalisation. In order to address these issues, a dedicated group of experts within the Eurosystem regularly reviews the performance of the staff projections and refines the tools used to produce the projections.