SPB Newsfeed: Interview with Dr. Kelly – Policy Evaluations Promote Learning and Improvement
Countries in South East Europe (SEE) have been incorporating performance information into their budget processes to strengthen the relationship between the allocation of public resources and the results achieved from use of these resources. Assessing the effectiveness and efficiency of programs requires robust and meaningful performance information, which focuses far more on the accountability of line ministries than has been the case in the region. To learn more about how to use program evaluation to strengthen both budget preparation and execution processes, we talked to Professor Dr. Joanne Kelly, Director of the Australia and New Zealand School of Government (NSW) and an Associate Professor in Government at the University of Sydney.
Dr. Kelly joined us as a lecturer at the “Program Budgeting, Evaluation and Spending Review”* workshop to share her experience with program evaluation and spending reviews. She specializes in the techniques, politics, and processes of government from a comparative perspective. She is the author of books, journal articles, and book chapters on different aspects of public financial management. She has worked in both government and academia, including four years as academic advisor to the Secretary of the Treasury Board Secretariat of Canada. She has written reports for and provided advice to governments in Australia, United Kingdom, Canada, Italy, New Zealand, and South Africa.
Dr. Urška Zrinski, who leads research and learning program in budget preparation and execution at the CEF, talked to Dr. Kelly during her visit to the CEF.
Despite making valiant efforts, most of the countries in the region have experienced ongoing difficulties in moving towards a performance based or performance informed approach to budgeting. In many ways the region is not unique in this. In your experience, working as an academic advisor to various governments, what would you say have been the main lessons learned in designing and introducing a performance based approach to the budget process?
Introducing performance budgeting or performance informed approach to budgeting, as it is sometimes called, is a complex process and the requirements of reform are unique to each jurisdiction. As you know, performance budgeting is typically traced to the US Defense force in the early 1960s and the fact that not one country can hold up a victory flag and say it constantly uses performance information in the budget process is a sign of how difficult this reform actually is.
My observations or lessons are that the challenges in introducing performance budgeting fall into two main categories. At a technical level, the introduction of performance information into the budget system is relatively straightforward. If you have people with the skills, resources and time available to undertake the reform project required to redefine the entire accounting, financial management and appropriations system on a performance basis, it can be done. It is not particularly simple - and there will be furious debates over whether or not we are using the right type and number of indicators to measure the right level of performance. All of this will take time but it can be done and we know how to do it. So, at the technical level, the introduction of performance budgeting is not overly challenging: yes, it can be complex but debates over what performance information can be resolved if the right skills and expertise are brought to bear.
The problems with performance budgeting really come into play when we start thinking about how to use performance information in budget decision making. How can we use this information to shape decisions on allocation, reallocation, program termination or redesign? This is the really difficult part of the performance reform – and of course it is also the most important. If you are not using performance information then what is the point is developing a system to generate all of that data.
On the whole, experience has shown that people are resistant to performance information in two ways. First, people resist having their performance measured or evaluated for various reasons, including that it is often viewed as another control mechanism rather than one to improve performance – and unfortunately this is often pretty accurate. People are also resistant because they contest the meaningfulness of many of the indicators developed by the so-called ‘technicians’. As a result, poor quality data is often being fed into the performance system and when this happens the integrity of the entire system is undermined.
Secondly, budget decision makers in both cabinet and the legislature seem quite reluctant to adopt performance information as the basis of resource allocation in the annual budget process. It is very apparent that budget allocations are decided using a whole range of different criteria, and that initial assumptions hoping to link performance outcomes with resource allocations have proved very naïve.
In this highly contested arena of the annual budget process, performance information can only ever be one input and we should never fool ourselves that it can be anything different. When you have people trying to make decisions about complex things for society, for example making decisions between education and defense force, these are choices of values and people need to be involved in seeing themselves in them. Yes, you should bring evidence into the mix and that is the role of performance information. But we have to recognize the complexity of budgeting, and how the process works and differs hugely between countries.
In contrast, most really good senior public sector managers will demand quality performance information as the basis for deciding the allocation and reallocation of resources within their department or ministry. At this level of decision-making much of the heat and fury of the annual budget processes is dampened, and the charged questions of whether the new money goes to education or health are largely resolved. But it is no less important if we are to ensure the public money is being used for the right things in the right way. Remember the annual budget process only really deals with about 5 per cent of the total expenditure each year. The remaining 90-odd percent of public resources are largely uncontested and allocated in just the same way as the previous year.
So to that extent, we are probably more likely to have success if we shape performance information so it can be used by ministers and departmental heads when they are involved in their internal budget and expenditure management processes. To that end, one important step in the reform process is to show when and how performance information has been used successfully. When it has improved outcomes and make better allocation decisions; that we have been able to alleviate some uncertainties, tensions, and conflicts within the department.
And, really, the best way to do this is to start small. One of the biggest problems with the introduction of performance budgeting is that we try to redefine the entire budget process for the entire country, when we should actually be starting small. Let’s start with small agencies, do pilot projects, and see how that works. And we also need to look at the cultural norms stopping managers from adopting performance information. If it does not work, you need to think of why it is not working, and not be scared of admitting to it. It is ok to recognize you did it wrong this time and just think of how you can do it differently the next time. There is not one perfect answer and that has been demonstrated any number of times. And that is ok. We can change things, adopt them, drop them, and then pick them up again. It is thinking about what we do that it is important.
Only then should we start to expand, using lessons learned and successes as the basis for building a more tailored and comprehensive system if it is needed. That tends to be the approach of countries that use performance information routinely as part of budget decision making: it has worked because people genuinely believed it is the right thing to do, they are not just doing it because they have been told to.
Program evaluation is an important tool for enhancing the accountability of line ministries in the budget formulation process. However, only a few countries in the region perform evaluations of program performance. What in your opinion are the main advantages of performing such evaluations?
If you do not evaluate, you do not know how you are doing. You don’t know if you are doing things well or badly, and you do not know how you could do things differently. It is as simple as that. It is a way of improving. It is a way of making sure that the precious resources we have available are being used to achieve much-needed societal goals. That is why we need to evaluate things. It is not and it should not be a punitive thing. It should not be about external rewards and sanctions. If performance evaluation is going to be used anywhere we need to recognize and explain the intrinsic value of evaluation as an end in itself. Not as a mode to ensure that ministers are punished if things go wrong.
If we talk about accountability as a way for ministers to take stewardship or ownership of how well public resources are used, we need to know if resources are being used as well as they can be. Yes, we need to know if what we are doing is legally authorized, but it is much more important to know if we are effective and efficient at what we are doing. As the financial crisis has shown, financial resources are precious. And they are delivering programs and services that are deeply meaningful for people’s lives. And if we do not take the time to evaluate, if we do not build evaluation capacity into our programs and processes, then quite frankly, we are just being arrogant.
If we are operating in a way that says it does not matter what I do with societal resources or donors’ resources - which are also coming from the pockets of other taxpayers - because I am in such a position that I can decide on my own, then we need to be checked and held to account. That attitude is a throwback to a period that is hopefully long gone in most of our countries, an attitude that says government knows it all, and it cannot be questioned. Thinking that there is nothing we can do better, is just wrong. Evaluation should not be threatening. As some wonderful spark in the group said today during our training session here at the CEF, “maybe we can use program evaluations to improve what we do or even prove that we are actually doing quite a good job”. It is as simple as that: that is why we need evaluations.
The idea of introducing greater accountability for line ministries through better use of performance information, which we spoke about earlier, appears to be a logical approach, and seems to make sense. However, in practice it seems to be extremely difficult to achieve. What, in your opinion are the main challenges in increasing accountability of line ministries, and are there any lessons to be learned from country experiences? Also is the introduction of a performance based budgeting approach possible in the absence of full accountability by line ministries and agencies?
The question of accountability is incredibly important but it is highly misunderstood. What is accountability and what does it actually mean? The notion of accountability comes from the Westminster system of government that puts a chain of accountability between the voters, the legislature, and the executive. But once you dig in more into the operational meaning of the term, you see it is used to mean three rather different things. First, it can mean a constitutional accountability in which we simply want to ensure that power is fragmented between different institutions in society. In that understanding we want different levels and different arms of government questioning each other so that they are all holding one another to account. You would include the role played by a strong and independent media and the wide range of active groups in civil society in this framework as well. Under this model of constitutional accountability, it is by questioning that governments are held to account and those questions can be framed on any basis, - and it certainly does not require performance information. Secondly, is the model of democratic accountability in which power is shared between the democratic institutions – typically refers to relationships between the legislature and the executive, with the judiciary playing the role of arbiter. Most important here is that the legislature has the power, resources and information necessary to hold out the executive to account for its actions. For many years that has occurred in many countries without any performance information. More recently, this has been the focus of reforms that incorporate performance information into appropriations documents with the intention that these will be linked to annual reports, and then form the basis of questioning in parliamentary forums such as public accounts committees etc. The basic assumption is that performance information will be used as the basis upon which to reward or sanction either individual ministers or the government more generally. This model has proved problematic as it is typically adversarial in nature and tends to result in suspicion on both sides. Executive don’t trust that the legislature will use the information properly; the legislature don’t trust that the executive has provided full and frank information – and both suspicions have some justification!
The third model of accountability is learning based accountability and this is typically associated with improving performance, about improving program delivery. Clearly, we need performance information to be built into this model. We need detailed, complex, informed discussion on how we can actually improve performance, and how we can learn to do things better. In this case performance information is important. But if we are operating in a system of accountability that is adversarial and is not based on learning then this information is actually irrelevant and can be used in the wrong way. In fact, all of the problems that we see around the introduction of performance, the suspicion between people, the lack of trust, disagreements on how to actually identify what the indicators should be, could be, what to do with the information, is often linked to the fact that is being used in adversarial, punitive fashion, which is typically associated with that democratic or constitutional accountability as opposed to using it in learning mode of accountability. That does not mean that there are no consequences if performance information shows that we are not achieving acceptable results.
Finally, when we talk about accountability and performance, too easily we shift it into a technical discussion. But discussions about accountability get at the very heart of our democratic and constitutional systems. If we want to make performance and performance measurement the basis of accountability then we much engage the key actors in parliament and society more broadly in the definition of the performance indicators and the outcomes that you want to achieve. If you merely defer the definition of performance indicators to technicians then you are never going to link the two. We forget that too often and shift to judging programs on the basis of economic or financial performance – forgetting to ask the more difficult questions of was it effective, was it equitable, was it right? Forgetting to think about how to design performance systems that allow us to evaluate according to multiple – sometimes conflicting and difficult - criteria. Elegance of system is not what we need, we need sophistication, we need depth and we need to really understand what is going on if we really want to improve government programs.
Having lectured at CEF learning events both this and the previous year, what have been your impressions of the progress in introducing budget reforms in the region? Do you get the sense that progress is being made and where do you see the main challenges over the medium-term as many countries in the region gear up for EU accession or attempt to meet the requirements of the EU fiscal compact and other EU legislation surrounding management of the public finances.
It has been an absolute pleasure to be involved in this project. The engagement and the enthusiasm of the participants and seeing this new, young generation of public servants coming through and being interested and engaged and learning new ideas and trying to work it out, is really encouraging. Yes, there is sometimes discouragement because the changes we are talking about often seem to be hard, but the overriding feeling is genuine curiosity and thirst for improvement. And the work being done here at the CEF is just invaluable for that. Because it allows people to hear from both experts and peers about new ideas and concepts, how they are being used and experienced across the region, and on the basis of that, you get a feeling that participants believe they can do things. This is really the birthplace of learning: when a person reflects on what they can and cannot achieve within their own system, they see what needs to change, and when a person sees the positive and the value of learning something new, then they really start to explore and learn new things.
I think it is incredibly important that we do not treat these reform programs and processes as something that can be done in one or two years. When I look back on the work that was done to reform the budget processes and systems in Australia, I see that they started somewhere in the mid-1960s, and only got any real traction in the mid-1980s. After that there was a period of real dynamic reforms in the mid-1980s and 1990s. So you see we are talking about 20, 25 years here. There was an entire department dedicated entirely to really trying to nut out and think about the problems of domestic level.
One of the greatest challenges is that we only see public expenditure management reform as a response to specific crisis rather than a way of really building up capacity across the region. And I can see that some of the pressure to reform quickly may have its downside so people with the power to push reform need to be incredibly sensitive to the realities on the ground. We need to design systems for the reality of each country and decide what is needed on the ground in a very practical and realistic way. Not develop processes based on what we believe is needed. If we do the latter, we are going to see this huge chasm between the formal system and the informal system. And that is where the failure lies, in that chasm. If, however, it is seen to be useful for the country itself, then people will engage in the decision making, the creation, as well as in the creativity. And the only way to be creative is if you got the skills. And the institutions such as the CEF and this project you are running provide those skills. I think the EU is doing such a great job in funding this program. But do not be impatient, and do not think that you can achieve things overnight. You simply cannot – the job it too big and important to be done in a rush. What you can do, however, is put yourself on a track that says reform and innovation is not just ok, it is the norm and it is desirable. And in doing so we are actually building a better society. We are building better capacities for ourselves, not for anyone else.
Here is where the ownership over reforms comes in. The ministers have to believe in them, the heads of departments have to believe in them. We need to innovate and we have to take chances. But at the same time international institutions have to be careful that when countries do innovate and do take a chance, that they do not punish them if they do not work the first time. It has to be a creative and dynamic process. That is how reform processes work. And you have to resource it. You cannot just get people to do it out of their side of their desk while they are doing their day-to-day things.
And one final comment, in reforms one of the most difficult things to recognize is that if you are creating new systems, then you have to stop doing some things. The forgotten part of reform is to identify the things we have to stop doing and whether or not we are still achieving objectives of the reform process. It actually happens often that when we go to talk to people in the budget department for example about doing spending reviews, one of their main fears is that this is going to be on top of their daily job. They are then, of course, resistant. But when you explain that no, in doing this you can actually stop doing all of these other things, some people celebrate and some people again get nervous because they do not know how to. And that is where the training comes in. This is where it is so important for institutions like CEF to continue providing training, and in training you empower people to innovate a whole new raft of reform that will in turn improve the lives of people across this region.
* The “Program Budgeting, Evaluation and Spending Review” workshop was delivered as part of the Strategic Planning and Budgeting (SPB) project, funded by the European Union. The overall objective of the project is to contribute to strengthening of beneficiary countries’ capacity to design and implement medium-term macro-fiscal policy.
