What are monitoring , review and evaluation

  • Derek Gill and Susy Frankel “Learning the Way Forward? The Role of Monitoring, Evaluation and Review” (Cross-cutting theme paper prepared for the NZ Law Foundation Regulatory Reform Project, 2013). The distinction in types of ex post evaluations is an important one. In formative evaluations the focus is on ‘are we doing things right’ while in summative evaluations the focus is on ‘are we doing the right things’. Summative evaluations can be used to assist in answering three questions: is the regulation still necessary, is the regulation efficient (does it achieve the objectives at a lesser cost than other alternatives) and is the regulation effective in achieving the objectives.[9] The focus of the questions that the evaluation is to answer will frame the approach taken, what data is privileged over other information and what the evaluation might conclude. Unlike evaluation, there are no textbooks, journals of monitoring or a professional body for monitors. ‘Monitoring’ is a practice but not a profession. The plain English definition of monitoring is “to be aware of the state of a system and to observe for any changes that occur” which implies “surveillance that is arms-length from that responsible body.” [10] Monitoring collects data on what is happening that can be used to inform any review or evaluation that is undertaken. Monitoring can indicate what has happened and sometimes insights into ‘how’ it happened while evaluation provides information on ‘why’. In this paper we use the term ‘review’ to describe a deliberative examination with a view to taking action. Reviews can occur at two levels. They can occur at the level of an individual case or transaction as a means of providing an assessment of procedure and fairness of process. Reviews can also be focussed on the overall regime and its effectiveness, drawing upon evaluations where these are available.

Learning from accountability processes

  • Derek Gill and Susy Frankel “Learning the Way Forward? The Role of Monitoring, Evaluation and Review” (Cross-cutting theme paper prepared for the NZ Law Foundation Regulatory Reform Project, 2013). New Zealand, like other OECD countries, has been active in developing a regulatory management system since the mid-1980s but the prime focus has been posing ‘big policy’ questions to review the flow of new regulations rather that reviewing how the stock of existing regulations is implemented, enforced, monitored, evaluated and reviewed. This paper is concerned with the review of existing regulations – through monitoring, evaluation or ad hoc or programmed review. This paper discusses the existing formal requirements for review of central government regulations and how these requirements compare with practice. The research finds that the formal requirements for review of existing regulations are limited relative to other comparable jurisdictions and underdeveloped relative to the provisions which apply to new regulations. The paucity of monitoring, evaluation or programmed review is paradoxical, particularly in the face of uncertainty about the effectiveness of regulation. The paper concludes with a discussion of the policy implications and the lessons for the overall regulatory reform project. A consistent finding from this project is that regulation can be characterised as a process of learning and experimentation. The very uncertainty about the impact and effectiveness of regulation creates an imperative to actively manage the stock of regulation. It suggests the need for more regulation of meta-regulation by introducing compulsory new requirements for all proposals for new regulations proportionate to their significance. These requirements should include a monitoring and measurement plan, a discussion of review options as well as an evaluation plan.

  • Dean Knight and Rayner Thwaites “Administrative Law through a Regulatory Lens: Situating Judicial Adjudication within a Wider Accountability Framework” in Susy Frankel and Deborah Ryder (eds) Recalibrating Behaviour: Smarter Regulation in a Global World (LexisNexis 2013). Finally, it is encouraging to see some allusion to the learning perspective and the extent to which judicial supervision makes a measurable contribution to regulatory betterment. Hammond J made passing reference to instrumentalism and the importance of understanding the impact of judicial review when shaping judicial review doctrine. He exhorted that:[70] … we should not overlook the problem that if the goal of administrative law is to be defined partly in terms of somewhat broader objectives – such as, for instance, the promotion of good governance – one would normally pay close regard to the empirical evidence that administrative law can actually achieve that end. Hammond J lamented the lack of local empirical evidence assessing "whether administrative law as a behaviour modification mechanism in government actually works", while noting the overseas experience is that administrative law only makes a "modest contribution" to this goal.[71] The corollary of this is, he said, that if "substantive doctrines as are developed for merit review should go only to what might be termed 'true excesses'."[72] Hammond J properly questions the assumption underlying the promotion of more intensive judicial supervision, that curial supervision is an effective mechanism for achieving good governance and improving administrative decision making. While Hammond J's remarks can be read as an implicit reference to the learning perspective, he did not dissect this instrumental dimension. Indeed, his remarks bundle together impact which is intended to sanction and deter non-compliance (in the sense of policing conformity with administrative law standards) and impact which facilitates learning (in the sense of encouraging reflexive and systemic improvement in the effectiveness and efficiency of administrative action). Nor is the extent to which the former may inhibit the latter drawn out in any detail. In any event, Hammond J's broader proposition is sound. It is unsatisfactory that the assessment of the value of judicial supervision as an instrument of regulatory betterment is undertaken in the absence of empirical evidence drawn from local conditions. It is incumbent on the protagonists for more extensive and more intensive judicial intervention in the heart of administrative decision making to provide greater evidential foundation of its curative role. Only then can the move to greater intervention be justified on the grounds of its success as a behavioural modification tool in administrative and regulatory decision making.

  • Rayner Thwaites and Dean R Knight “Administrative Law through a Regulatory Lens: Situating Judicial Adjudication within a Wider Accountability Framework” in Susy Frankel and Deborah Ryder (eds) Recalibrating Behaviour: Smarter Regulation in a Global World (LexisNexis 2013). Accountability, as defined above, is a social relationship between an actor and a forum. In what follows, the Regulatory Standards Bill is characterised in terms of the three components of our integrated evaluative framework set out earlier: "information provision", "debate", and "consequences".[84] Under the Regulatory Standards Bill, the Minister and the chief executive are explicitly directed to provide the relevant information to Parliament.[85] As for any judicial declaration of incompatibility; the forum in which the court's declaration constitutes information is not specified under the Bill. In so far as the judiciary is conceptualised as "monitoring" the certification process, then the forum for a judicial declaration, understood in terms of Bovens' information provision is Parliament. Nonetheless, the true target of the Bill lies elsewhere. While falling outside the formal accountability framework provided under the Bill, the Taskforce clearly intended the Bill to influence the persons behind the legislation, the policy makers and drafters responsible for generating and drafting the relevant regulatory intervention.[86] The Bill is intended to indirectly influence these individuals by putting to justification those who have to certify a regulatory measure's compatibility with the principles, or those who have to defend a claim of compatibility in court.[87] The nature of the debate about those principles in Parliament takes its cue from the constitutional perspective, and concentrates on conformity of the legislation with the relevant principles, where that is given content by judicial rulings on the principles, or legal advice directed at anticipating those rulings.[88] As to consequences, there are no formal consequences as long as one confines oneself to the courts. The issuance of a declaration of incompatibility does not affect the legal validity of a measure.[89] The expectation is that a court ruling constitutes information to be fed into parliamentary and wider public debate, potentially generating a response in the legislature. The consequences are political. The intent is that this political response will in turn modify the behaviour of those developing and drafting legislation, bringing them into conformity with the nominated regulatory principles. There is an incomplete forum, the courts, nestled within the larger forum of Parliament, to complete the accountability relationship.