#4.5 Evaluation

Beth Simone Noveck in Analyse & Kritik (2018)

Read #4.4 Implementation

Sadly, policymaking and legislation often end with enactment. There is no systemic effort to understand the impact a law had, for whom, how and why. Most evaluation takes place long after the fact with limited data. Yet evaluation serves as an important piece in the feedback loop to improve existing service delivery and inform future policy formulation. But some of the oft-cited challenges to evaluation include scarcity of resources and access to relevant data that limit feedback processes. This presents an opportunity for engagement, including asking the public how to measure impact, what data to use for that purpose and engaging them in the process of evidence gathering (Dinesh 2018). Enabled by new technology, a watchful community can improve the outcomes of government activities and render legislation, as a solution to public problems, more accountable because it can be evaluated. Sometimes known as ‘social audits’, these participatory processes have the potential to enable more iterative regulation and legislation. Alas, the social auditing projects usually have nothing to do with formal lawmaking or, as we shall see in the case of Chile, where they do, they do not use technology to give them broader scope and scale. This stage of the lawmaking process is the one most in need of crowdlaw projects.

4.5.1 Social Auditing in Ghana: TransGov

TransGov is a platform created in 2014 to help Ghanian citizens monitor the progress of local development projects by empowering citizens to hold government accountable for faulty or incomplete infrastructure projects and service delivery in their localities.

They curate a list of projects in local communities and give people the ability to comment. Today, TransGov has 600,000 registered users who provide feedback through the TransGov website, mobile app, by SMS or using Interactive Voice Response (IVRS). Six employees run Transgov and handle the technology, management and communications of the project. Although not strictly legislative in nature, it is an instructive example of using the distributed power and collective intelligence of citizens to monitor policy outcomes and create an evaluative feedback loop.

4.5.2 Evaluation in Chile: Evaluación de la Ley, Chile

In the legislative context, there’s also a unique crowdlaw project in the Chilean Senate, where they have hired a full-time facilitator who runs evaluative focus groups post-implementation of a new law to understand how it is working (Evaluacion 2018). “The ex-post evaluation has as a goal to determine if the regulatory framework has fulfilled the desired objectives, if the law or regulation was sufficiently efficient and effective in its implementation and to what extent expected and unexpected impacts of the intervention regulations were adequately addressed when conceiving the regulatory instrument.” This is a formal part of the lawmaking process in Chile but nowhere else. The facilitator convenes lawmakers, staff, stakeholders and citizens to discuss evaluative metrics and the success of the implementation to data.

The Department of Evaluation of the Law has developed a three-stage project to evaluate the effectiveness of a law. It is determining compliance with the metrics set out when the saw was enacted. It gauges the perception of citizens about the law and its implementation. It also suggests corrective measures for the law and its implementation to bring it into line with the stated metrics. The process is not applied to every law. It is used for laws that do not regulate highly ideological or partisan issues and are relevant to large numbers of people. Evaluation is also subject to a feasibility assessment to determine if it is, in fact, possible to measure efficiency or effectiveness in the implementation.

4.5.3 Citizen Monitoring in Brazil: Projeto Controladoria na Escola, Brazil

In Brazil, a decree passed in 2007 ensures that public schools have the autonomy needed to spend funds assigned to them by the federal district for maintaining and operating the school. The intention behind the decree (called Programa de Descentralização Administrativa e Financeira (PDAF) was to help public school management to respond in an agile manner to local needs. Yet, audits conducted in random municipalities by the national comptroller have shown that there are deficiencies in school infrastructure quality across the country. Studies have attributed several reasons for these deficiencies including lack of resources, corruption and student behavior but there is less information available at granular levels to pinpoint where the issues are (Ferraz et al. 2012).

Therefore, in 2016, the comptroller launched an experimental project called the Projeto Controladoria na Escola to engage students in 10 public schools in Brasilia in the process of auditing school infrastructure, mapping commonly raised issues and fostering civic education in schools The project involved asking students to collect data about their local school environments, reporting the major issues they faced, identifying the root causes of those issues and proposing ideas to fix them. In one school alone, the students identified 115 issues and within just 3 months 45% of the issues were fixed either by the department of education or, where possible, by the students and school management themselves. (Moll 2016) The success of the project was two-fold. It not only enhanced the CGDF to conduct detailed audits of every public school but also generated great buy-in from the schools to identify, report and fix issues in their surroundings. The civic audit model employed in Brazil is a great example of organizing citizen-led campaigns to foster civic education and to help government oversight agencies understand local issues with granular details. It also helps build a sense of community and, done right, motivated citizens to take action to fix those issues. It is still unclear, however, if the campaign improved educational outcomes and if the medium and long-term solutions were implemented. But social auditing and monitoring of this kind that takes advantage of the distributed power of citizens to monitor effectiveness could be a promising improvement to legislative practices, if systematically implemented as part of the lawmaking process. No smart company implements a policy without measuring whether it works and the cost for the bottom line neither should government spend taxpayer dollars without more real-time evaluation and assessment. If parliaments distribute the work of monitoring implementation, for example, to citizens with camera-phones, this could dramatically increase the ability to evaluate the downstream impact, including both cost and benefits, of legislation on people’s lives (Wilkerson 2015; Center for Civic Media 2017; Björkman 2015).

But to be done well, the metrics for evaluation and the data collection task must be clear to all parties lest the social auditing process result in a partisan or ideologically-motivated attempt to undermine the legitimacy of the legislation.

Read the next part — #5 Conclusion