ICGFM Promotes Knowledge Transfer Among Public Financial Management Experts

Working globally with governments, organizations, and individuals, the International Consortium on Governmental Financial Management is dedicated to improving financial management by providing opportunities for professional development and information exchange.

Tuesday, May 26, 2009

OBI Response to Blog Posting

Editor's Note:

There was some controversy at the 23rd Annual ICGFM Conference surrounding the assessment of the Government of Honduras by the Open budget Initiative. Mr. Vivek Ramkumar had summary data on hand when he received questions at the conference. He has kindly provided ICGFM with a comprehensive answer.

As the manger of the International Budget Partnership’s (IBP) Open Budget Initiative, I would like to respond to the ICGFM’s headline (“Is the Open Budget Initiative Rating Accurate?”) for its May 20th blog posting of my responses to questions on the Open Budget Index 2008 (OBI) findings raised at its recent meeting in Miami, Florida.

The Open Budget Index 2008 is a comparative measure of the overall commitment of the governments in 85 countries to budget transparency. The OBI 2008 is calculated using a subset of data collected through the Open Budget Survey 2008—a comprehensive analysis and survey that evaluates whether central governments give the public access to budget information and opportunities to participate in the budget process. The Survey also examines the ability of legislatures and auditors to hold their governments accountable.

The Survey is implemented through a rigorous data collection and review process that spans nearly two years. Although the Survey is based on a questionnaire, it is not an opinion poll. All researchers’ responses to the 123 questions in the survey must be based on verifiable evidence, such as reference to specific budget documents, the country’s laws, or documented communications with government officials.

Within each country, a researcher or team of researchers, who are drawn from civil society or academia and are independent from the government and political parties, complete the survey and provide the required evidence for their answers. Their work is analyzed and reviewed by IBP staff to cross check it against available information. This include those budget documents that countries made available on the Internet, data collected by the Bank Information Center (a Washington, DC-based nonprofit that monitors the activities of international financial institutions); the International Monetary Fund (IMF) Reports on the Observance of Standards and Codes (ROSCs), covering fiscal transparency; IMF Article IV reports; World Bank documents and publications, including Public Expenditure Reviews; and the Organization for Economic Co-operation and Development-World Bank budgeting practices database.

Next, the completed questionnaire for each country is subject to a thorough peer review by two in-country budget experts, who were independent of both government and the researcher’s institutions.

Finally, governments were invited to comment on the completed questionnaires prior to publication. The decision to extend an invitation to a government to comment was based on a request by the research organization that completed the questionnaire for that country. In total, 61 countries were invited to participate. (For more details on the Survey’s methodology and findings, visit www.openbudgetindex.org.)

In addition to the thorough review process, IBP also conducted two further tests to check the reliability and robustness of the data. First, the Survey results were compared with the results of other indices of governance and transparency to see how the Survey fares as an overall indicator of the governance situation in a country. The Survey results showed a strong, positive correlation with the World Bank’s World Governance Indicator on Voice & Accountability (0.737), the Global Integrity Index produced by Global Integrity (0.681), and the Democracy Index produced by Freedom House (0.691). These positive results suggest that OBI is a relatively good proxy for broader measures of governance and the quality of institutions in the countries covered.

IBP also constructed a “unanimity score,” a measure to capture the degree of debate between the researcher and the two peer reviewers responsible for completing the questionnaire in each country. The scores for this Measure of Unanimity are available in the Survey report at http://openbudgetindex.org/files/FinalFullReportEnglish_lores.pdf.

The Open Budget Survey is intended to provoke public debate about budget transparency, public participation in budget debates, and accountability of budget institutions. As such, the research process frequently results in debate among the experts in the country responsible for completing and reviewing the questionnaires about important public financial management topics. IBP attempts to capture this debate through the “unanimity score” and by publishing within each questionnaire the exchanges between researchers and reviewers that led to the selection of a final answer to each Survey question (available at www.openbudgetindex.org).

Since the February 1, 2009, release of the Survey, challenges to OBI 2008 scores have come almost exclusively from governments that earned low scores. I am happy to report that in some cases these governments have made positive changes that should result in more budget information being available to the public. However, since these changes were made after the cutoff date (September 28, 2007) for data collection for the 2008 Survey, these efforts were not reflected in their OBI 2008 scores. If these efforts are sustained, though, they will be reflected in higher scores on the next round of the Open Budget Survey in 2010.

As I explained in my video response to the questions raised by the representative from Honduras about his country’s low OBI 2008 score, there are specific actions that the government could take that would improve their performance. Honduras’ OBI 2008 score of 11 out of a possible 100 is primarily due to the fact that as of September 27, 2007 the government did not produce a Pre-Budget Statement that outlines the macroeconomic framework of the budget, a Citizens Budget (a presentation of the budget that is delivered in a way that is easily understood and widely accessible), or a Mid-Year Report that compares the actual expenditures and revenues to those budgeted. The government did produce an Executive’s Budget Proposal but did not release it to the public. By producing and making these documents available to the public, Honduras could raise its OBI score significantly by 2010. A relatively quick and low-cost first step would be to post the Executive’s Budget Proposal, which is already being produced for internal purposes, on the government’s website. A summary of Honduras’s results on the Open Budget Survey 2008 is available at http://openbudgetindex.org/files/cs_honduras.pdf and the completed questionnaire for the country can be found at http://openbudgetindex.org/files/IBPQuestionnaire2008Honduras.pdf.

Finally, I would like to reiterate the point I make in the video that the IBP believes that governments must be strong and effective to meet the needs of their people, particularly the poor. This belief drives the Open Budget Initiative, which seeks not only to measure levels of budget transparency and accountability at particular points in time but also to provide guidance on how to improve.


Jonathan Eyler-Werve said...

There's three general categories of response when someone brings bad news:

1) fix the problem
2) ignore it
3) kill the messenger

Option three is popular with some governments. Ironically, the very transparency and granular detail that makes tools like the Open Budget Index so valuable also makes it easier to attack. While results based aggregations of surveys may be completely nuts, no one has access to the source data, so we just have to accept it. For these new second-wave assessments that have open, original source data, there's always something to pick away at, even if it represents a tiny fraction of the assessment and an overall vast improvement on accuracy and usefulness of the better-known governance indices.

Governments can be expected to push back on negative assessments, but one rather hopes the wider community won't reward this.

ICGFM said...

Vivek has made a case for the effectiveness of the Open Budget Index. As we heard in the discussions about PEFA in May and MCC in December at ICGFM conferences, there is something to be said about objective evaluations. The numbering algorithm might not be entirely effective, but the results act as an excellent benchmark.