Members of the Task Force of the Strategic Planning and Budgeting (SPB), European Union funded and CEF managed project, successfully finished preparing the project LogFrame.

LogFrame Approach to Measure Results

Assessing the results of capacity development actions is generally challenging, yet producing evidence of development outcomes is crucial to ensure that resources are used effectively. Have you been working with project LogFrames – 4 x 4 tables showing what you do and get out of a project? You may have experienced how it empowers you to structure your thoughts about the theory of change that shall be achieved, and the causal logic linking the project interventions and the monitoring of the impact they have. (See here for a recent blog post of our colleague Jana Repanšek, discussing the LogFrame approach as an analytical process and a set of tools that respond to the results-based management approach and reductionist understanding of capacity development.)

Preparing a Project LogFrame

Your development partners may have requested you to prepare a LogFrame to better communicate what you are up to with the project and how you measure results from input to impact. You may have even mastered the challenges of consolidating the expectations of your beneficiaries and donors with what is feasible within the limited resources – making a LogFrame realistic yet ambitious is not easy. Much advice is shared on how to best do your LogFrame and coming to a final version may remain a moving target, but in the end it is your call to conclude revising the LogFrame at some point. The beneficiaries, donors and you yourself may be happy to have come to a shared picture about the project, and you may be full of hope that the chosen indicators will indeed be SMART (Specific, Measurable, Attainable, Relevant, and Time-bound) when it comes to monitoring and evaluating the project.

Monitoring the Project

Next, you start to implement the project and get back to your LogFrame at last when drafting the first narrative report. But now you face another obstacle: how to actually tell a narrative with your LogFrame? Matching the figures on all these well-chosen indicators with your activities, outputs, purposes (or outcomes) and goals (or impact) gives you a good snapshot on what you have done and achieved. How should you use the indicators to convey a story? Have you been thinking in advance of what stories you might like to tell with the project and the specific change(s) that you try to promote? Can the indicators be useful in this regard? You might come to the conclusion that there is no way out other than interviewing some of the project stakeholders about how they experience the project. You would quickly realize how much additional information they share, which you did not envisage when drafting your LogFrame. Of course, there are opportunities to further strengthen your skills in drafting and applying LogFrames. How should you match them with stories that the project stakeholders convey?

Value Creation Stories

We came to experience this learning journey some time ago, and reached the conclusion that we need to better frame how we engage with our stakeholders in story-telling. The Value Creation Story (VCS) framework of Wenger and Trayner quickly turned out to be the best fit for this purpose, as it got us thinking also of the next steps on our learning journey. (See here for a recent blog post of our colleague Polona Sirnik, describing the VCS framework.) According to this framework, we have started first to frame our interviews with stakeholders at the moment of reporting, looking for stories that would help us better tell the narrative behind the LogFrame of our capacity development project on Strategic Planning and Budgeting (SPB), funded by the European Union.

Interviews with Beneficiaries

Interviews with SPB beneficiaries help collect the applied value that defines changes in practice. Applied value needs qualitative assessment, unlike immediate and potential values that can be collected through evaluation surveys. Collecting applied values often seems challenging, as a certain time is needed to pass between the end of a learning activity and the interview. We have experienced that re-interviewing beneficiaries repeatedly helps collect values that are measurable after some time. Also, repeated interviewing helps stakeholders more systematically reflect on their own learning experience, which supports comprehensive story-telling. Value creation stories can be used as an ongoing process. Eventually, beneficiaries even reveal the so-called secondary stories that show what others have learned from them, after they attended a specific learning event.

Matching LogFrame and Value Creation Stories

The SPB project’s LogFrame indicators are also augmented for the VCS framework in a way that specific activities are linked to desired outcomes. For example, we are interested in how improvements of medium-term macrofiscal frameworks are applied, and how effective policies are coordinated with respect to EU requirements. To measure these applied and realized values, we use concrete sources of verification, such as assessments of beneficiary countries’ macrofiscal planning documents (i.e. their Economic Reform Programs) and other sources of verification (e.g. project reports and feedback provided by beneficiaries in the value creation interviews and in the final event survey). It is this integration of qualitative and quantitative data through combining the stories with rigorous data collection what makes the process so powerful.

The CEF, together with its partners, plays a role of enabler of learning and networking activities. For example, for the typical assumption stated in the LogFrame of beneficiaries being in full support of the envisaged action through its implementation, such enabling role is crucial to ensure that beneficiaries get maximum value out of the action.

Sharing our Experience

It has also been helpful for us to share practical experiences and good practices in implementing the VCS framework with others using this framework. We know now that the stories empower us to do more than mere reporting on our projects to relevant stakeholders – they give us rich insights to better our work at different stages. Having started to feed back what we have learned from the stories of our stakeholders into our work, we have come to see the unused potential behind the many indicators that we monitor in our work. For example, we have realized that conducting value creation interviews with beneficiaries has a value as such, helping them reflect on their own learning experiences (see here for an example). There is more to do to fully internalize looping such feedback into our daily work, but it is a very exciting experience and learning from the narrative stories is a very rewarding reality check. In pursuing to get better in telling and learning from the stories of our value creation, we are now strengthening another important step – thinking more comprehensively at the design of a project about the stories we aspire to tell during and after implementation.

Note: The views expressed are those of the authors and do not necessarily represent those of the CEF.

References:

Better Evaluation. Sharing information to improve evaluation. Accessed March 27, 2015 at: http://betterevaluation.org/evaluation-options/logframe.

Repanšek, J., 2015. Theory and concepts behind CEF’s approach to capacity development part III: Reductionist and systemic perspectives to capacity development. Accessed May 5, 2015 at: http://knowledgehub.cef-see.org/?p=562.

Rockefeller Foundation, 2014. Digital Storytelling for Social Impact. Hattaway Communications.

Sirnik, P., 2015. Articulating the Value of Our Learning Activities. Accessed March 27, 2015 at: http://knowledgehub.cef-see.org/?p=535.

Wenger, E., Wenger-Trayner, B., and de Laat, M., 2011. Promoting and assessing value creation in communities and networks: a conceptual framework. Accessed March 27, 2015 at: http://wenger-trayner.com/resources/publications/evaluation-framework/.

Wenger-Trayner, B., 2014. World Bank report and recommendations. Learning partnerships in the program for capacity building to strengthen good financial governance in Southern and Eastern Africa.

Wenger, E., and Wenger-Trayner, B. E-mail list on value creation framework. Accessed March 27, 2015 at: http://be.wenger-trayner.com/listinfo.cgi/be-vcf-wenger-trayner.com.

Wenger-Trayner, B., 2015. Quoted in M. Čarman and K. Jurtela, Knowledge Hub as an Enabling Environment for Social Learning? Center of Excellence in Finance. Accessed March 27, 2015 at: http://www.cef-see.org/news/knowledge-hub-as-an-enabling-environment-for-social-learning-2015-03-25.