Silly rankings don't help

The World Bank should change the methodology of its Doing Business ranking

In August 2020 the World Bank announced a “pause” in the publication of its annual Doing Business rankings because of cases of data manipulation. The currently ongoing “comprehensive review” ordered by the chief economist should be used to address substantial methodological shortcomings identified by ESI that make the ranking easy to trick.

ESI report: Rankings that fail - Bosnia, Macedonia and Doing Business 2015 (23 May 2015)

ESI has long warned that the World Bank’s popular Doing Business ranking is easy to manipulate. In a 2014 newsletter we examined how Georgia, according to the World Bank, made it into the top 10 countries with the best business climate globally. In 2012, Georgia’s top export was used cars, sold to its oil-rich neighbour Azerbaijan. And yet, according to the World Bank’s 2013 ranking, Georgia’s “business climate” was top. Also North Macedonia had started a dramatic catch-up race, winning place 23 of 185, ahead of Switzerland (28). 

In 2012, World Bank experts also found that there was no member state of the European Union where it was as easy to start a business or to register property as it was in Belarus. Or that an investor was better protected in Belarus (82nd) than in Germany (100th).

Unexpected success stories reinforce the World Bank’s message that if any country follows concrete prescriptions it will achieve amazing results. It is the promise of fairy tales: the ugly duckling turns into a beautiful white swan, giving hope to all ugly ducklings in this world.

The problem is that the ranking is based on ten simple indicators that can be measured in all countries on this planet, from very simple economies based on subsistence agriculture and small businesses to advanced high-tech and service economies. They include issues such as starting a business, getting electricity and obtaining construction permits. ESI noted

“To suggest that the number of days it takes to register a shop or café is one of only ten decisive indicators for a country’s business environment is misleading. 

The analysis does not take into account many factors that are obviously crucial in an investor’s decision to invest, among them: security; market size; macroeconomic stability; state of the financial system; and level of training and skills of the labour force. And within the indicators that are measured, certain critical aspects are left out, such as: the reliability of utilities; the cost and availability of credit for firms; and subsidies.”

Nevertheless the ranking was popular and taken seriously. In Bosnia, improving its Doing Business ranking became part of the reforms the EU expected Bosnia to carry out most urgently. It even turned into an EU condition for accepting a membership application.

In March 2015, ESI wrote to the Bosnian Presidency, suggesting 14 simple steps to turn Bosnia into a wunderkind in Doing Business rankings. We argued that while a rise in these rankings would not change the business climate – for this other reforms are urgently needed – it would take away an excuse to further delay Bosnia’s EU integration process.

Bosnia improved spectacularly in the rankings for 2016 (published in late 2015), “jumping” nearly 30 places from rank 107 to rank 79 within one year.

A closer look, however, revealed that Bosnia had not changed much. The improvement was due to what the World Bank calls “data revision” of previous rankings, a process it conducts every year. This meant that the World Bank retroactively changed Bosnia’s 2015 rank from 107 to 82. In 2016 Bosnia thus improved only by 3 places, from 82 to 79. But nobody knew what to make of this ranking, as another “data revision” was to be expected twelve months later ... 

The European Commission stopped making references to the Doing Business rankings in its reporting on the Balkans. But other wanna-be swans were inspired. In the last available ranking (2020) not only has Georgia (7th) consolidated its rank among the top 10 and North Macedonia (17th) beaten all but three EU member states. Now also Kazakhstan (25th), the Russian Federation (28th) and Azerbaijan (34th) have overtaken Switzerland (36th). Serbia is close behind (44th).

In August 2020 the World Bank announced a “pause” in the publication of its annual Doing Business reports, because “a number of irregularities have been reported regarding changes to the data”. An initial assessment by management identified “irregular” data changes “made outside of the appropriate review process and not justified by the Doing Business methodology” concerning Azerbaijan, Saudi Arabia, the United Arab Emirates and China. The World Bank Group’s audit unit found that:

“Pressure from stakeholders as part of the DB report production and publication process has not been effectively managed. DB team members reported undue pressure, both directly and indirectly by Bank management to manipulate data in 2017 ... and in 2019 ... The lack of a safe speak-up environment within the DB team led to a fear of retaliation for those who would escalate and report pressures to manipulate data.”

A “verification of the management review” by the Group’s audit unit noted that “the management review was limited to data changes that occurred after the circulation of data as part of the institutional review processs.” This means that the possibility of manipulation before data was circulated in the institutional review process was not addressed. 

In December 2020 the World Bank’s chief economist, noting that “the Doing Business methodology has been the subject of widespread discussions and debate inside and outside of the World Bank Group,” ordered an external review that “will serve to crystalize key elements of these debates and draw implications from them for consideration by World Bank management as they seek to refine the Doing Business methodology and enhance its policy impact.”

ESI calls on the external review panel to address the severe methodological problems of basing the ranking on ten simple indicators of which several can be easily manipulated.