Derek Gill[*]

The current system for making [regulatory] choices is broken. It is largely based on faith, rather than evidence. The efficacy of many regulations is never assessed. Many others are only evaluated before they are implemented – the point when we know the least about them.[1]

15.1 Introduction

The Stage One chapter[2] in this project discussed how New Zealand, like other OECD countries, has been active in developing a regulatory management system since the early 1990s.

The focus of this chapter is applying the logic of the regulatory management to itself. The details of the Treasury's Option 5 will not be explored because they are still being developed as this chapter is being written (mid-2012).[6] Instead, the perspective is that of a researcher "'one floor above' the policy world they describe".[7] This chapter explores the following in turn:

(1) What are special features of regulation that require the special measures of a regulatory management regime?

(2) How does the unique context facing New Zealand affect these special measures?

(3) How are these special measures meant to work?

(4) What are the likely costs and benefits of regulatory management?

(5) What are the potential indirect and unforeseen consequences?

(6) What remains to be done?

The chapter concludes with a discussion of the policy implications for the regulatory[8] reform.

15.2 What are the special features of regulation that require special measures?

To address the question concerning the special features of regulation that require special measures, this chapter starts with the special measures in the regulatory management regime and then address the special features of regulation.

15.2.1 Special measures used in regulatory management

The Stage One chapter[9] discussed the key features of regulatory management applying to central government.[10] It defined the regulatory management regime as the set of special measures used to augment the formal government system that apply specifically to primary, secondary and tertiary regulation.

In the New Zealand central government context there are a number of potentially important institutions. Parliament has the Regulations Review Committee (RRC) while within the executive there is the Treasury Regulatory Quality Team, the Parliamentary Counsel Office (PCO), the Legislation Advisory Committee (LAC), and the now largely defunct Legislation Development Committee (LDC). In 2009, the government introduced a policy statement "Better Regulation, Less Regulation" which, as discussed below, is linked to disclosure requirements. The requirements for Regulatory Impact Assessment (RIA) are becoming increasing institutionalised both in domestic legislation[11] and as is discussed below (at [15.6.2]) as part of international trade and investment agreements.

In addition to these institutional arrangements, there are a number of tools or procedures that apply to primary and secondary legislation before they are considered by Cabinet, including a Preliminary Impact and Risk Assessment (PIRA) to determine whether a Regulatory Impact Analysis (RIA) is required, the application of the RIA framework itself, and the preparation of a RIS. The latter includes a requirement that the Minister certify whether the proposed legislation or regulation is consistent with the policy statement, and a named official is required to certify not only that the proposals are consistent with policy but also that other requirements have been met. In addition, while the focus is on ex-ante screening, there are very few ex-post procedures that apply to regulations once they are introduced.[12]

Annex 2 in the Stage One chapter, showed special measures for legislation occurring across the different phases of the central government regulatory life cycle:

(1) "Big Policy" Design – five special measures: PIRA, RIS disclosure statement, RIS quality assurance, Ministerial Certification, the LDC.

(2) "Little Policy" Design – four special measures: LAC, BORA (including section 7 reports), PCO drafting (mandatory use of specialist drafting), and the Cabinet Legislation Committee.

(3) Deliberation – four special measures: free access to primary and secondary legislation, RIS published, RRC and LAC (potential submitter on Bills).

(4) Implementation – no special measures.

(5) Administration and Enforcement – three special measures: post-implementation review[13] (selected regulations only), regulatory scanning and administrative review.

(6) Review – two special measures: ad hoc reviews of regulatory programmes and the RRC.

Two things stand out from the analysis. The first is that the main focus of regulatory management is on "big" policy design of new primary and selected secondary regulation, rather than tertiary regulations, "little" policy,[14] or the implementation and administration of existing regulations. The second is that the main locus of attention is executive procedures rather than those of parliamentary or judicial branches of central government. As discussed in the introduction, the Regulatory Standards Bill currently before the House would strengthen the role of judicial review against good regulatory principles while Treasury Option 5 involves strengthening Parliament's role in reviewing new regulation.

15.2.2 Specific features of regulation

Turning to the question of what is special about regulation that requires special measures, regulation is often contentious, the effect of regulations is often uncertain and regulatory analysis is difficult.

(a) Regulation is often contentious

Figure 15.1: Contention about means and/or ends from regulation[15]

Source: Author

Contention occurs, in part, because there is scope for disagreement about both the means used (the methods or policy interventions) and the ends or desired objectives. For simplicity, Figure 15.1 shows the various possible combinations of high and low contention as binary choices. In reality, policy issues range across a continuum, rather than being a binary choice between high and low. What Figure 15.1 shows is that even where there is analytical clarity about means and ends, political will to act may still be lacking as illustrated by the case of New Zealand Superannuation and population ageing.

(b) The effect of regulation is uncertain

The policy process tends to emphasise the potential contribution and minimise the uncertainty about the effects of regulation. As such, regulation is a form of random rather than designed experimentation.[16] The Stage One chapter discussed how there are four distinct possibilities concerning causation. Figure 15.2 illustrates Kurtz and Snowden's argument concerning the different extents to which cause and effect are knowable in advance.

Figure 15.2: Models of causation

Source: Adapted from Kurtz and Snowden (2003) ^[17]

Where the problem is known but the solution is not, then a regulatory approach is required that involves learning the way forward. This "learn to build" approach requires quick response times and measurement of key characteristics. Learning and experimentation requires allowing for "fast failure" by promptly reversing bad change and reinforcing good change. In domains characterised by uncertainty and complexity, policy tools to screen and scan new regulations such as cost-benefit and cost-effectiveness analysis are less useful, as it is not possible to divine in advance the one best way to address a policy problem. It suggests that in a world where the impact of regulations will evolve in ways that are impossible to predict in advance, more is to be gained by more effectively learning the way forward and by monitoring and managing the stock of existing regulations, rather than screening the flow of new regulations. The implications for management of the regulatory stock are discussed in the final part of this chapter.

Learning and experimentation also require reviewing traditional approaches to judicial review which is discussed in the companion chapter on the place of adjudication in regulatory reform.[18] A consequence of drawing in the courts is that material not intended to be law (such as guidelines and standards) becomes "soft law", as the courts use it as reference to guide their decision making in areas with significant policy content.

(c) Regulatory analysis is difficult

Regulatory analysis is difficult in part because views are contested. Reasonable analysts have different views on the efficacy of different regulation, based on different judgments regarding the extent of the failure of government, market and the courts. Market failure suggests scope for public interest regulation, while governments can fail due to problems of information asymmetry, rent-seeking[19] and regulatory capture. Even in the absence of government and market failure, Shleifer[20] argues that we have to add the possibility of "efficient regulation" due to courts failure. "Efficient regulation" results in regulations replacing contracts where courts fail.[21]

The roles and responses of different actors will not be binary "either/ors". Ayers and Braithwaite's[22] work on regulatory compliance demonstrated how there are a range of possible responses (the willing compliers, the muddled non-compliers, the deliberate evaders) for whom different enforcement strategies are effective. This requires the segmentation of clients and the adoption of different strategies based on client characteristics. This suggests that the effects of regulation are often uncertain and involve complex judgments about how uncertain dynamics will evolve. The emphasis for regulatory management needs to focus less on screening regulations before they are implemented in order to optimise net benefits, and focus more on building in resilience and adaptability over time.

This part of the chapter has discussed how regulation is often contentious, the effect is often uncertain and regulatory analysis is difficult. Much the same comments could have been made about other public policy interventions including those that use taxes or provide similar programmes. This leaves open the question: what is unique and different about regulation that requires the special measures of a regulatory management regime?

15.2.3 Special features of regulation that require special measures

The Stage One chapter explored a number of public policy rationales for regulatory management regimes due to the special features of regulation that set it apart from other government interventions. The argument goes that with specific regulations the state acts as a coercer and reduces the set of choices open to businesses and citizens by precluding certain activities or requiring others. In contrast with fiscal interventions, the state generally[23] acts more as an enabler (although still underpinned by the coercive power to tax) by expanding the set of choices open to citizens and business. Taxes, subsidies and transfers act by changing the income constraint or the relative prices people face in making choices. Thus there is an argument that the special character of regulation poses risks in particular to human rights and to liberty.

The argument that regulation is special because it constrains choices can be buttressed by other concerns. First, there is a risk that excessive new regulations will be introduced, because an explicit budget constraint is absent and the lack of transparency about the cost of regulation is likely to bias decision makers. Faced with a choice between regulatory and spending interventions to achieve a policy objective, regulation will be preferred even if this is less efficient. Second, while the costs of expenditures (other than tax expenditures) are appropriated annually and subject to review by the executive and the legislature, the on-going administrative and compliance costs that regulation imposes on citizens and businesses are not scrutinised in the same way. While, for example, the Controller and Auditor-General (C&AG) can undertake effectiveness or value for money audits on how regulations are managed, the bulk of the value for money reviews and all of the financial audits focus on public spending and public programmes. Unlike other countries supreme national audit institutions, the New Zealand C&AG has not undertaken studies reviewing the quality of RIAs. This lack of regular scrutiny is puzzling in the light of the uncertainty about the impact of regulations and the impossibility of predicting how regulations will evolve.

In summary there are three arguments for why regulation deserves special measures:

(1) the main argument is that there are risks to human rights[24] and to liberty because regulations generally impose constraints on citizens and businesses;

(2) regulation provides a path of least political resistance as the lack of an explicit budget constraint and associated transparency biases the choices of political decision makers to select regulation over other interventions; and

(3) once in place, regulations persist over time because the compliance costs are hidden and are not subject to regular scrutiny.[25]

These special features of regulation provide a strong "normative" argument for a regulatory management regime. The Stage One chapter pointed out, however, that this prima face case provided a poor "positive" explanation for many of the observed common features of regulatory management regimes. The normative case leaves open questions about the lack of focus on most secondary and tertiary regulations and the lack of emphasis on management of the stock of regulation and the capability to implement regulations. It does not seem consistent with the imposition of cost-benefit analysis to evaluate proposed new regulations particularly associated with regulatory management in the United States, the point at which as Greenstone observed "we know the least about them".[26]

15.3 How does the unique context facing New Zealand affect these special measures?

15.3.1 New Zealand's unique geography

New Zealand is a "large" small country. Compared to 223 other countries: the exclusive economic zone (EEZ) is the fourth largest; the GDP per capita is the 27th highest; the land mass is significant as it is 74th largest; but, with a rank of 123rd, we have a relatively small population.

New Zealand's relatively small population has a number of consequences for regulatory quality, some positive and some negative. New Zealand's smallness, along with a history of openness, has meant that the public has ready access to select committees and there is a well-established credible submission process. Unlike comparable jurisdictions we make extensive use of primary legislation (rather than relying on secondary or tertiary regulations) and under Standing Order 280 all legislation (other than Bills under urgency) goes to a select committee.

By contrast the small population size, extremely high number of ministerial portfolios and large number of small public organisations means that policy advisers are spread across a large number of public organisations, and the capability to design and implement regulation is dispersed. This raises concerns about the lack of critical mass of expertise in more advanced and complex areas of regulation.[28]

Moreover, the system is strongly vertically aligned – while Cabinet government is strong in New Zealand, there is no strong tradition of horizontal functional leadership at the level of the public service. A recent report by the State Services Commission[29] proposed changes to increase "functional" leadership focused on achieving more economic input costs (procurement, ICT, property and finance), but with no proposals for "functional" leadership of outputs including policy or regulatory design and enforcement.

15.3.2 New Zealand's unique constitutional arrangements

There are a number of unique constitutional features that constrain the quality of law making. These include a unitary and extremely centralised state with one House, a three-year parliamentary term, limited functions undertaken by local government and the relative paucity of checks and balances. This all leads to an imperative for legislative haste over quality – what Palmer described as "the fastest law in the west."[30] Mai Chen attributes this speed to a range of features including the limited scope for filibusters, the use of omnibus Bills, the ability to truncate house procedures, and Part-by-Part (rather than clause-by-clause) review during the Committee of the Whole House phase.[31] These constitutional factors together with the lack of deeply entrenched industry lobby groups make it easier to enact legislation in New Zealand than in other comparable jurisdictions.

Offsetting this, New Zealand has evolved other systems. Consultation with indigenous Māori people, while sometimes marked in the breech, has become more common.[32]

Regulatory management regimes have largely been "welded" onto the existing machinery of government and constitutional arrangements. Many of the special features of the regulatory management regime are common to all developed countries. However, the "devil is in the detail" of the fine-grained rules that determine how these broad regimes apply. For example, the United States and United Kingdom regulatory management regime superficially looks similar to New Zealand. In practice, the United States regime only applies to rule making by independent regulators and does not apply to law making by the United States Congress, while the United Kingdom regime does not apply to European Union regulations. In New Zealand, by contrast, the RIS regime is meant to apply to all legislation developed by the executive so essentially only private members Bills would not be subject to the regulatory management regime.

In summary, the effect of the small size of New Zealand's population is accentuated by the way policy regulatory design and implementation capability is distributed across a number of small public organisations. Offsetting this there are a number of institutional arrangements that give the system robustness and resilience including the active role of Cabinet Committees, Cabinet and select committee review.

The previous two parts of this chapter explored what was special about regulation that requires special measures and what issues were particular to regulatory management in New Zealand. The next part explores how the special measures are intended to operate in the New Zealand context.

15.4 How are these special measures meant to work?

This part, first, explores the objectives; and, second, the theory of change underpinning the regulatory management system in place in New Zealand.[35]

15.4.1 Competing objectives for regulatory management

15.4.2 The theory of change

This discussion of the theory of change starts from the rebuttable assumption that the regulatory management system is trying to improve regulatory quality.[41] To illustrate the theory of change we turn to the work of Gary Coglianese. Coglianese uses a simple schema to bring out the complexity of evaluating individual regulations as well as regulatory management systems. What Figure 15.3 brings into sharp focus is the key links that need to be articulated in a coherent theory of change. In particular, the figure points out the need to be clear about whose behavior the regulatory intervention is trying to change and in what way it is to be changed.

His schema, shown in Figure 15.3, starts with the formal institutions (A), the meta-system of regulatory management (B), the individual regulation of interest, which is nested in a bigger set of regulations (C), through implementation (D) to desired behavior change (E), transmitted through intermediate outcomes (F) and final outcomes (G). The diagram reminds us that regulation is part of a wider system that is influenced by factors outside of the simple linear flow.

Figure 15.3: Coglianese casual map of regulation

Source: Figure 1.1 Evaluating the Performance of Regulation and Regulatory Policy (2011)^[42]

Annex 1 uses the schema to organise the evidence from previous studies on the impact of regulatory management in OECD countries. It shows that evaluations can occur at each step along the way. Between the regulatory management system (B in Figure 15.3) and the regulation of interest (C) there are two measurement points. Content evaluation focuses on the extent to which the analysis complies with the RIA requirements, while output evaluations measure the quality of the analysis undertaken. Between the regulation of interest (C) and behavioral change (D) impact evaluation studies assess the influence of the RIA on decision making. Outcome evaluations (G) examine whether there was a change in the quality of regulation as a consequence of the RIA process. Figure 15.3 brings out the high information requirements to adequately monitor the stock of regulation and to evaluate its effectiveness.

15.4.3 Competing theories of change

Quite different theories of change or logics[43] are in operation in different jurisdictions operating superficially similar regulatory management systems including the logics of delegation, rational policy making and democratic governance. The next part of this chapter will use the Regulatory Impact Analysis (RIA) to explore the different logics.[44]

(a) Delegation

The first logic – delegation – explains the rise of RIAs in the United States. President Reagan needed to achieve greater control over rule making by executive agencies in order to pursue an agenda of deregulation.

Elements of this logic were also evident in New Zealand's early experimentation with regulatory management through business compliance cost reduction. The logic of the argument was that the vast majority of regulations are driven by bureaucrats' agendas rather than ministers' priorities. Discussions with Ministry of Economic Development staff suggest that the early regulatory management regime targeted behaviour change in bureaucrats by giving the New Zealand Cabinet collectively greater information with which to control bureaucrat-driven rule making. When first introduced the RIA placed great emphasis on providing ministers with a short (two–three page) problem definition and assessment of options and consequences so that ministers could make more informed decisions about regulatory quality.

However, this logic and theory of change does not explain how the New Zealand regime has changed over time with the growing length of RISs and the requirement to have separate agency certification and ministerial statements.

(b) Democratic governance

The second logic – democratic governance – argues that by increasing the transparency of the making and enforcing of administrative rules, regulation making will become more open to diffuse interests and more accountable to citizens and Parliament. In this context it is the behaviour of ministers and, to a lesser extent, senior bureaucrats that are targeted.

Greater transparency and increased use of consultation contributes to building the public understanding of, and hence legitimacy of, regulations. As Scott Jacobs put it:[45]

Essentially, RIA has become one of the methods through which societies speed up learning. Because it is an open and consultative technique, it stimulates social learning, in which various stakeholders involved in the issue gain a clearer sense of the options, and trade-offs, and the consequences of solutions, than in the past.

(c) Rational policy making

The third logic – based on rational policy making – has two variants. The first variant is the "formal assurance" model reflected in the United States and Australian systems. This logic suggests that the more systemic use of economic analysis (cost-benefit analysis (CBA) in particular) will lead to a more informed policy process, which in turn will lead to an improvement in the quality of decision making. The second variant is the "process transparency" model used more in the New Zealand system. In this approach emphasis is placed on including the RIA early in the policy process. Rather than RIA as a tool centred on CBA, the RIA is centred on questions and alternative options. The desired effect is that RIA is used to impose a more disciplined, rational structure on the development and communication of policy proposals which will lead to an improvement in the quality of decisions.[46] The latter approach accepts that policy making itself is not a rational process, but tries to introduce greater discipline into the policy analysis process.

In contrast with a delegation logic, where the problem was an agency problem of alignment of bureaucrats or politicians, here the objective is to address weaknesses in regulatory analysis capability. The objective of the "process transparency" approach is to change the behaviour of bureaucrats by improving regulatory analysis and the policy advice process. This logic is strongly reflected in later versions of the regulatory management policy which results in the growing length and complexity of RISs and the relocation of lead responsibility for regulatory management to the Treasury.

The theory of change is that with better analysis, bureaucrats will provide better advice which will enable ministers to make better decisions, which will then result in better quality regulations.

However, this is not the only logic underpinning the New Zealand regime. RISs of regulatory proposals considered by Cabinet are subsequently published when the resulting legislation is introduced into the House. The RIS is not updated for the Parliament debate to take into account any changes made as a result of Cabinets' deliberations. This greater transparency is consistent with the democratic governance logic involving opening the regulatory process to diffuse interests and making it more accountable to citizens and Parliament. The implication of this publication requirement for public service bargains is explored further in this chapter below.

In summary, the rational policy making logic is the dominant logic used in OECD documents and in official thinking in New Zealand. However, traces of all three logics can be found in OECD and New Zealand government documents. The logic and desired impacts on behaviour from regulatory management are different in New Zealand from comparable jurisdictions. The different logics are targeted at different actors and aim to change different behaviours in different ways. These different logics are not mutually exclusive or completely incompatible, but they do imply distinctly different theories of change. The next part of this chapter explores how the costs of regulatory management compare with the benefits and the available evidence relevant to the different theories of change.

15.5 What are the likely costs and benefits of regulatory management?

CBA is the process of comparing the likely costs and benefits of a policy proposal over time. CBAs were central to RIS processes in the United States, but less central in Europe, and New Zealand. As discussed in the previous part (at [15.4.3]) New Zealand guidance tends to emphasise RIA as a process centred on questions and alternative options, rather than RIA as a tool centred on CBA. Radaelli and Meuwse observe that, "Cost benefit tests do not make much sense in a situation in which a minister is expected by a Parliament to exercise discretion in accordance with the political preferences of the Parliamentary majority."[47] In the United States, however, the separation of powers alters the position. The Executive Orders that established the regulatory function of the Office of Information and Regulatory Affairs (OIRA) within the Office of Management and Budget also required that any new regulation proposed by an Executive Agency must pass a cost-benefit test.

However, the requirement for a CBA does not appear to have been generally applied to regulatory management before meta-regulation regimes were established. Annex 1 provides a summary of studies that evaluated the effectiveness of RIAs in a range of OECD countries. An initial literature search identified one study which attempted to quantify the cost and benefits of regulatory management, and one study of the cost effectiveness of regulatory impact assessment. The lack of citations of these papers or comparable studies suggests the field is sparse.

15.5.1 Cost-benefit analysis of the United States regulatory management regime

The study using CBA was undertaken by Robert Hahn (2009) who prepared a stylised review of the cost and benefits of regulatory management in the United States. The main direct incremental cost Hahn identifies is the cost of doing and reviewing the regulatory analysis. The main direct benefits are the present value of any improvements in regulatory policy and the greater transparency in regulatory policy making.[48] Hahn finds that the total costs in the United States for reviewing 100 major regulations each year are around USD 74 million:[49]

… I think, but cannot show definitively, there are many regulatory proposals for which net benefits increased by at least a billion annually as a result of analysis and regulatory evaluation …. Thus, the current system is likely to have benefits in excess of costs, if one makes two key assumptions: all proposed policies would have been implemented without regulatory evaluation; and the costs of policy delay[50] from regulatory evaluation are small.

Hahn's analysis highlights that while the costs are reasonably easy to identify, estimating the benefits is more complicated. One complication is indirect effects – Hahn does not attempt to quantify the benefits from improvements in transparency which is discussed in the next part of this chapter. The major complication is the difficulty of attributing improvements in regulation making directly to regulatory management. In addition to detailed information on the net costs and benefits of proposal, a CBA also requires information on the net costs or benefits of the counterfactual that would have been adopted without regulatory evaluation. The counterfactual – what would have happened otherwise – is required in order to attribute improvements to regulatory policy making as a result of the regulatory management system. In a sense Hahn's analysis evades the key CBA challenge.

15.5.2 Cost-effectiveness analysis of regulatory management in Victoria

The cost-effectiveness analysis was undertaken by staff at Victoria Competition and Efficiency Commission using the gross savings from RIA processes over the period 2005–2006 to 2009–2010.[51] The incremental costs of undertaking RIA were estimated at AUD 3.2 million per annum.[52] The average annual savings are estimated at AUD 180 million (including two large one-off savings) or AUD 90 million per annum (excluding outliers). The most important savings came from proposals not being implemented (including the two one-off savings), followed by narrowing the scope of regulatory proposals, less onerous interventions being selected, and finally thresholds and frequency of requirements being reduced. Overall, the study findings are positive for each AUD 1 spent on a RIA, with AUD 56 gross savings identified or AUD 28, excluding the one-offs. Cost-effectiveness analysis does not attempt to quantify the counterfactual or benefits such as the increased transparency.

15.5.3 Evidence from other studies

Interestingly, there is a marked contrast between the findings from the one CBA and one cost-effectiveness study available and the evidence from a number of evaluations in a range of jurisdictions on the limited effectiveness of regulatory management. NZIER's 2009 paper summarised the position as follows:[53]

Over the last decade there have been a number of papers reviewing the leading OECD nations' experience with RIA. The broad conclusions have been that RIA has a limited impact on quantifiable measures and that effective implementation is difficult to achieve.

Annex 1 contains a brief summary from a range of studies of RIA in different jurisdictions. In brief the main conclusions are that:

• the majority of studies find modest, rather than significant, impacts on the quality of regulation;
• qualitative case studies tend to be more positive about the impact of RIAs than quantitative studies that use numerical measures;
• the studies are focused on the limited efficacy and effectiveness of RIA and generally did not consider the compliance costs; and
• only one study (Jacobs)[54] finds that the quality of regulation declined following the roll out of RIAs, although this was compared to a trial period for RIAs rather than the pre-RIA era.

In short, almost all of the studies find very limited positive impacts of RIAs on the quality of regulation. However, the one CBA and one cost-effectiveness study both find significant positive impacts from RISs.

The next part of this chapter develops a stylised view of the costs and benefits of New Zealand's regulatory management regime. It explores whether New Zealand's experience is consistent with the general pattern in the literature of modest positive gains or more consistent with Hahn's assessment of significant returns from regulatory reviews in the United States and the high cost-effectiveness of the RIA process in the state of Victoria.

15.5.4 Stylised CBA of New Zealand's RIA requirements

This part discusses a stylised CBA of New Zealand's regulatory management regime. The CBA reviewed the Regulatory Impact Analysis of new regulations through the Regulatory Impact Statements (RISs) as this is the main focus of the regulatory management system.[55] Reviews undertaken by NZIER and others for the Treasury have found that the quality of RISs varies significantly. For example, the latest review by Castilia estimates that in 2011, 36 per cent fully met, 50 per cent partially met and 14 per cent did not meet, Cabinet RIS requirements (detailed in Cabinet Office Circular CO (09) 8).[56] Anecdotal evidence from New Zealand suggests that the majority of RISs are treated as a compliance exercise and completed at the end of the policy development process, if at all.[57] However, the minority are used as an integral part of the fabric of the quality assurance process for regulatory policy as it is being developed.

(a) Costs

Looking first at the costs, regardless of whether the RIA is built into the process or treated as a compliance exercise at the end, the extra costs associated with the RIA process are unlikely to be significant. The Treasury regulatory team has four full-time equivalents (FTEs) working on RIS assessments at a cost (including overheads) of around NZD 200,000 per FTE. So, based on the 2010–2011 volume of 50 RISs, that converts to NZD 16,000 per RIS. This is in the same ballpark figure of Hahn's estimate (in footnote 155) of OIRA's costs of USD 20,000 per review.

No information is available in New Zealand on the extra costs incurred by regulatory agencies. In the United States cases Hahn found that the total cost of an individual regulatory analysis in agencies ranged from USD 18,000 to USD 8 million, with an average cost of USD 700,000 and a median cost of USD 270,000, but estimates the additional costs involved in the RIS process at between USD 20,000 to USD 40,000.[58] The state of Victoria in Australia estimates the average incremental cost was AUD 77,201 per RIA.[59] New Zealand's RIS regime is less onerous than the United States and Victoria's, so Victoria estimates seem plausible conservative estimates for New Zealand.

In addition to the direct costs potentially there are indirect costs as well. It is arguable that RIAs are not just a means to improve regulation – RIAs have distinct political consequences and need to be understood as a political tool. RIAs can be used to protect the rule maker not attack it.

(b) Benefits

The benefits will depend in part upon whether the RIS process is treated as a compliance exercise or used as an integral part of the fabric of the quality assurance process. Where the RIS is treated as a compliance exercise, benefits are negligible and are likely to be outweighed by costs.

(c) Costs and benefits compared

In Figure 15.4, below, we show the Net Cost (NC) of the individual CBA as a small negative number. In the case where the RIS is used as part of the fabric of the quality assurance (QA) process, the experience from a number of the case studies was that the RIA process does help in refinement of options and selection of lower cost interventions or lower thresholds and requirements. As a result, more effective options are chosen to achieve a given policy objective. In these cases there was a Net Benefit (NB). The overall sign and magnitude of the CBA will depend upon the balance between those treating RIA as an exercise in compliance and those who use RIA as part of the fabric of quality assurance. The key point is that it takes one large net benefit (LNB) or a series of smaller net beneficial changes (NBs) to show that the RIA process overall adds value.

The stylised examples of the costs and benefits of RIAs are shown in Figure 15.4. The payoff looks like that from oil exploration or from an insurance policy. With oil exploration, a wild-cat well has a 19 out of 20 chance of being dry. In one out of 20 cases economic oil and gas reserves are identified, although often the find, while worth developing, has a small net benefit. However, occasionally in oil exploration there is a large find where the payoff is huge – hundreds of times the cost of drilling the original well. Similarly, the payoff from the RIA process is like an insurance policy with a low deductible; one pays a regular premium, in order to receive a sporadic series of small claims, but with the added potential for a very large payoff thereby averting some significant damage.[60]

Figure 15.4: Stylised CBA for RIAs

Source: Author

The Regulatory Standards Bill, discussed in the introduction to this chapter, provides an example illustration of the potential for a large payoff. The Bill, as developed by the Regulatory Taskforce, proposed an active role for the courts in reviewing regulations based on the principles of good law making. While some prominent lawyers supported the Bill, the balance of opinion in the legal and policy community was opposed.[61] One of the features of the Bill that received particular comment was the potential for introducing greater legal uncertainty, because of the more expansive role the courts required in reviewing the merits of regulation in terms of consistency with principles of responsible regulation. This would accelerate the move away from the traditional focus of administrative law reviews, compliance with the procedure and process followed, toward a greater emphasis on review of the merits of the regulation.

The background to the Bill was discussed in chapter 8[62] of the first book of this Regulatory Reform Project, and the arguments for and against need not concern us here. What is important is the role of the Treasury RIS in providing publicly available advice on alternative options, one of which (Option 5) was subsequently built into the coalition agreement. This shows the potential payback from the RIA process in "heading off at the pass" potentially damaging regulatory proposals. The question that a CBA has to answer is the counterfactual – what would have happened without the RIS? To what extent was the Regulatory Standards Bill already "dead in the water" and the RIS simply administering the last rites before death? To what extent was the RIS influential in heading off the Bill? The CBA requires the discipline of ensuring that these questions be addressed.

15.6 What are the potential indirect and unforeseen consequences?

All good CBAs also include a consideration of indirect effects, risks and uncertainties.

15.6.1 Improvement of the quality of regulatory proposals

One indirect effect of regulatory management is the improvement of the quality of regulatory proposals by deterring agencies from bringing forward high-cost proposals with low regulatory effectiveness. In the New Zealand case, some low-quality regulatory proposals do not proceed as a result of the risk screening through the Preliminary Risk Assessment.

15.6.2 The internationalisation of regulatory management

A second indirect effect arises from the internationalisation of regulatory management regimes. While regulatory management regimes were developed to focus on domestic regulatory objectives, there are two emerging international developments whose importance is likely to grow over time. The discussion under 15.3 above identified the internationalisation of policy as a major force that New Zealand faces for the foreseeable future. Standard setting is increasingly undertaken by international organisations, such as the International Federation of Accountants setting International Public Sector Accounting Standards. The introduction on RIA-type regimes applying to international organisations can introduce greater discipline and transparency into their rule making processes and provide information that member countries can use tactically to influence the development of standards.

The other aspect of internationalisation is the proposal to include "behind the border" requirements for regulatory coherence for acceding to international trade and services agreements such as the one under development in the Trans-Pacific Partnership. Any requirement to introduce RIA-type regimes would impose few direct costs on New Zealand, as an RIA regime is already in place, but does have the potential to improve New Zealand's trade access. The potential for improved access arises from RIA requirements increasing the transparency and disciplining the policy processes required in other jurisdictions before new "behind the border" trade barriers were erected.[63] However, the greater discipline and transparency introduced by the internationalisation of regulatory management can also be used tactically by other vested interests for their own advantage. The companion chapter in this volume titled "Defining the Ambit of Regulatory Takings"[64] and "Trade Agreements and Regulatory Autonomy: The Effect on National Interests"[65] discuss the potential for trade and services regimes to be used in trade disputes as a twin-edged sword. Moves to plain packaging for cigarette packs, on public health grounds, have been attacked as a breach of international intellectual property agreements. Whether regulatory management regimes can also be used in a similar way to attack regulatory sovereignty is untested.

15.6.3 Greater transparency improving the quality of regulation

Another potential indirect effect identified by Hahn was the improvement of the quality of regulation arising from greater transparency in regulatory policy making.[66] The New Zealand Treasury Regulatory Impact Analysis Handbook emphasises the need for consultation as part of the policy development process and sets out principles for effective consultation.[67] The first chapter by Mark Bennett and Joel Colón-Ríos discussed in more detail the role of consultation.[68] It identified consultation as an "input" and an "output". Consultation as an "input" into the regulatory policy development and decision making process can improve regulatory quality. Consultation can identify design flaws and potential "trip wires", and advance the design of more sophisticated "little" policy regimes.

Consultation on an "output" contributes to building the public understanding of, and hence legitimacy of, regulations. Consultation on the policy design creates "buy in" and is essential to effectiveness in the implementation phase.[69]

15.6.4 Greater transparency changing the public service bargain

The previous part of this chapter discussed the positive role of greater transparency in the policy development process increasing the legitimacy and effectiveness of the regulations. An indirect effect of this greater transparency is the impact on the conventions and arrangements that apply under Westminster-derived systems. Box 15.1 below discusses the perplexing issue of the role of RIS in the light of the roles and responsibilities of ministers and public servants.

In brief, the public servant must tender free and frank advice, but the ministers decide policy. The public service duty is to implement the policies of the government of the day. In theory under this doctrine public servants explain policy but they do not defend government policy nor make policy. In return for the loyalty to the government of the day, public servants have a right to anonymity.

Box 15.1: Morphing Public Service Bargains – the Role of RISs

The RIA process contains some provisions that seem at odds with the underpinnings of the Cabinet Manual and goes beyond the current practice under the OIA. The RIA process requires the minister to certify that the proposed regulation in the Cabinet paper is consistent with the Government Statement on Regulation: Better Regulation, Less Regulation which requires that it is "required, reasonable, and robust".[73] This requirement is unexceptional. What is more interesting is the requirement for a public servant to provide an independent attest as to the validity of the policy analysis and conclusions in the RIS. A named official is expected to provide a judgment, independent of the government of the day, on the merits of the various options which might meet the stated policy objective, including the minister's proposal. The Treasury manual also suggests that the official make a judgment on one objective compared to another. In practice, however, the focus of the RIS is on analysis of alternative means or options rather than evaluating competing goals or objectives.

The public official responsible for the RIS is required to:[74]

• [disclose] information to highlight any key gaps, assumptions, dependencies and significant constraints, caveats or uncertainties in the analysis [; and]
• [indicate] whether any of the policy options are likely to have effects which may not align with the commitments in the Government Statement on Regulation …

The RIS is made publicly available when the Bill is tabled in the house. The RIS is seen as a departmental, not ministerial, document. This is reinforced by the Independent QA[75] of the RIS (but not the Cabinet paper), either by someone from within the agency not directly involved in the proposal (for smaller policy proposals) or by the Treasury Regulatory Team for significant proposals.

What is to be made of the potentially conflicting declarations of the minister in a Cabinet paper and a public official in the RIS? Suppose the public official has already provided free and frank advice as part of the regulatory proposal development along the lines that proposed regulatory intervention is unlikely to achieve its intended goal. This advice could also conclude that the approach is not consistent with the government's overall regulatory policy. Having considered that advice, the government of the day may decide to pursue a regulatory option regardless.

Under long-standing Westminster doctrines the public servant has a duty to implement the policy to the best of their ability and to explain (but not necessarily defend) that policy publicly. Continuing to tender that advice will undermine the government's belief in the willingness of the public officials to implement the policy.

How then does a loyal public servant respond to the RIS requirements? In the Westminster tradition it is the duty to the minister that is paramount, so is it their duty to explain government policy and put the best possible face on the regulation? The RIS requires a named official to provide a judgment, independent of the government of the day, on the merits of a range of options including the minister's proposal. What if the minister does not accept that the duty to be independent trumps their duty of loyalty to the government of the day? Would the minister see this as the department undermining (sometimes termed "white anting") the minister or the Cabinet? What if the minister refuses to work with the official who was responsible for the RIS? If the minister attacks the public servant's opinion publicly does the department have the right of reply?

The answer to these questions will depend upon the context, the issue and the minister. The Treasury has long held a view independent of the government of the day, and successive Ministers of Finance and Prime Ministers have accepted that role. However, other departments have not been so fortunate and the overwhelming impression is that under successive administrations over the last 15 years, free and frank advice has been under attack. The legislation currently being considered as a response to the "Better Public Services Advisory Group Report", proposes to amend the State Sector Act 1988 to make the provision of "free and frank" advice an explicit responsibility of departmental chief executives.[76]

These are important questions for which there are no open-and-shut answers. The independent certification function in the RIA is reshaping institutions, in particular, by remaking the bargain between ministers and bureaucrats. There may be good reasons to review that relationship, and to recognise that increasingly public servants have duties which require that they act independently of the government of the day on statutory independent and mandated functions. The RIS allows senior public servants to meet their duty to advise Cabinet collectively, not just loyally serve the individual minister. Good arguments can be developed that the New Zealand public service not only has a personality, but that this personality is an important part of the constitution. My point is that the issue is bigger than the RIS, but that the long-standing bargain between the public service is moving further and further away both from the original Diceyian version, and that amended by the public sector reforms of the late 1980s and codified in the Cabinet Manual.

15.7 What remains to be done?

The previous section on the cost-benefit analysis of regulatory management provided evidence on RIA processes that scan new regulatory proposals. Relatively little evidence was available on the effectiveness of the management of the stock of existing regulations. This was because systematic management of stocks (as opposed to flows) is still in its infancy.[77] As George Tanner, former Chief Parliamentary Counsel, observed (in email correspondence), "the lack of any systematic process for post-enactment scrutiny means that routine maintenance of some very major pieces of legislation rarely happens. We paint our houses and service our cars, but we don't look after our laws in the same way."

Departments are required to undertake regulatory scans (which involve scanning the existing stock with a view to identifying which regulations are in need of review) and meeting any performance reporting required for the use of funding appropriated for the cost of administering regulation. Parliament also has a potential role through select committee reviews of departments. However, the primary focus of the regulatory management system is on the ex ante review of new legislation. In short, New Zealand is applying standard managerial approaches to the stock of regulation rather than many special measures.

15.7.1 Smarter management of the stock of regulation

So what would smart management of the stock of regulation look like? Regulation, as Michael Greenstone observed in the introductory quote, "is largely based on faith rather than evidence." The uncertain effects of regulation means that the design of regulations needs to allow for changes in tastes, technologies and methods of production and would tend to avoid options with high sunk costs and irreversible choices. Smart scanning of regulation would involve scanning based on risk assessment tools.

One of the defining characteristics of much regulation is that it is a process of experimentation as the behaviour of regulatees will adapt over time.[78] This adaption can take the form of changes in tastes and preferences (recent examples include social attitudes to smacking and smoking) and changes in technology and methods of production. Regulations based on particular technologies and tastes are less likely to be resilient than those that enable adaptation.[79]

These changes can go both ways in how net benefits from regulation change over time. Some enabling regulation such as the incorporation legislation (Companies Act 1993) and rules on weights and measures, by enabling voluntary behaviours that might not have occurred otherwise, expand the possibility sets. Other regulations that mandate minimum standards, certain technologies and techniques, constrain possibility sets. Smart regulatory scanning would focus on identifying those existing regulations where distortions increase and possibilities decrease over time.

In a sense, the state of regulatory management now is analogous to financial management prior to the passage of the Public Finance Act 1989 (PFA). Before the reform of 1989 the focus of budgeting was managing the flow of cash spending over a one-year time horizon and there was no balance sheet to measure how the stocks were being managed and limited attention to "out years". This requires addressing a number of issues: managing the flow, understanding the stock, clarifying responsibilities, addressing capabilities, resources, incentives and information.

(a) Managing the flow

The first stage in managing any stock, including the stock of regulations, is to try to put some discipline around the flow of new regulations. In the case of regulation, the RIS process and other requirements (such as regulatory planning) do that and this chapter argues that there are valid public policy reasons for applying special measures to regulations. However, the arguments for special measures apply with the same force to the management of the existing stock of regulations as they do to scanning the flow of new regulations.

(b) Know what the stocks are

The next stage in managing a stock is to know what the stocks are. The regulatory scanning process revealed that a number of departments lack basic information on the stock of regulations they had administrative responsibility for. Authoritative registers of legislation and regulation were lacking or incomplete and often did not include all tertiary regulations. The requirement for more systematic regulatory scanning and planning has addressed this deficiency.

(c) Clarifying who is responsible for what

The third stage is clarifying who is responsible for the care of those stocks, and clarifying what those responsibilities are. Every statute specifies one or more administering departments. There is little guidance or specification of the duties and responsibilities of an administering department. Similarly, the Cabinet Manual is virtually silent on the minister's role, beyond a short reference at [7.8.2]. A department's implied role is limited to the responsibility to provide policy advice to the minister on any amendments to the legislation, on whether the legislation needs amendment and answering queries from the public, Parliament and ministers about that piece of legislation. Historically, while a few departments have more actively monitored the operation of the legislation they are responsible for, the regulatory stock-take revealed that a number of departments were not aware of the stock of regulation they had responsibility for. Departments are now required to undertake regulatory scanning to inform the annual regulatory programme covering all primary, secondary and tertiary regulation.

(d) Establish a clear statement of the expectations

Beyond the limited performance reporting requirements, formal responsibility for the stock of regulation is currently lacking. The obvious next step is thus to establish a much clearer statement of the expectations for the roles and responsibilities of a department administering legislation and to whom those responsibilities are owed. This would be analogous to the department's role in the maintenance and care of physical assets on the agency's balance sheet or the assets that they are administering on behalf of the Crown. The legislative amendments to the State Sector Act 1988, as part of the changes associated with the "Better Public Services Advisory Group Report", proposes to provide in legislation for departments' roles and responsibilities.[80]

While providing formal responsibility in legislation is an important step, great clarity is required about what the expectations are of that role. The role would be likely to include a forward-looking maintenance schedule detailing work planned to maintain, modify, retire or replace regulations. There are two sorts of maintenance – regular or routine maintenance (akin to painting the house) and deeper refurbishment (akin to repiling the house). The Law Commission provides a vehicle for undertaking major legislative refurbishment in a number of areas. Addressing this responsibility raises issues of capabilities, resources, incentives and information.

(e) Augmented capabilities required

The development of a legislative maintenance schedule will require departments to augment existing capacities and skills, but need not involve a step change. Building capability requires getting the "hard" and "soft" components of organisational systems working together. The hard stuff (which is generally the easy components) includes staff skills and capability and the formal systems for collecting, storing and tracking regulations. The soft components (which are the hard components to introduce) include the behavioral elements such as a culture of inquiry, professional leadership, and engagement with key players on how the regulation is operating.

(f) Incentives

The issue of incentives presents a more difficult challenge. The financial management reforms, with the introduction of accrual accounting, balance sheet and capital change, proved very effective at improving departmental performance in managing financial assets. The reforms have proved much less powerful at improved management of non-financial assets. For example, chief executives have shown an almost pathological desire to restructure departments despite the evidence that capability and performance is thereby reduced.[81] It remains to be seen how to create the incentives to improve management of the stock of regulations and ensure that maintenance happens in the face of the multiple and conflicting objectives facing departments. Resourcing will continue to be a pressure point in an environment requiring doing better for less.

(g) Information and reporting on the state of the stock of regulations

After the roles and responsibilities, capabilities and incentives are established, the next stage is to articulate the information that the administering department would be expected to collect, and what external reporting would be required about the state of the stock of regulations that are their responsibility. This will require the development of screening tools,[82] reporting standards and the role, if any, of external audit. There is also scope to improve the links between stock management and the RIS regime, which is currently focused on managing the flows of new regulation. For example, in cases where an inadequate RIS has triggered the requirement for a post-implementation review, there is no requirement for public reporting of the results of that review, or for this review to be included in the regulatory scanning process.

The development of reporting needs to be shaped by the intended user and purpose. One of the themes that emerged from Stage One of the project was that regulation is often a process of learning and experimentation. This suggests that reporting needs to be developed with the emphasis placed on enabling learning, rather than to provide external accountability. Putting measurement and reporting in a learning frame is challenging because learning about the effect of regulations is itself very difficult. These difficulties arise because of a number of reasons including that effects are often separated in time and space and attribution of causation is difficult. Regulation, as Michael Greenstone observed in the introductory quote, "is largely based on faith rather than evidence."

If regulation is viewed as a process of learning and experimentation, then there are significant limits to the review of new regulation. Management of the flow of new regulations needs to be accompanied by more effective management of existing regulations. Focusing on the stock will raise organisational management questions about the capability of regulatory organisations rather than policy questions about the design of individual interventions. The focus on management of the stock needs to include the factors that enable learning from "fast failure". In the public sector learning from failures is particularly hard in the face of an authorising environment that is hostile to down-side risk.[83]

15.8 Conclusion

Regulatory management is a form of meta-regulation that involves responding to the special features of regulation with special measures – tools, procedures, institutions – that specifically apply to legislation, secondary and tertiary regulation. Regulatory management attempts to regulate the regulation makers by requiring a demonstration of the linkage between clearly articulated problems, the preferred response (from a range of regulatory policy options), and how that response leads to well-articulated impacts on behaviour and hence to a desired ultimate outcome.

This chapter has applied the logic of regulatory management to itself. Different theories of change were outlined which were designed to change different actors behaviours in different ways. The logic and desired impacts on behaviour from regulatory management are different in New Zealand from comparable jurisdictions. A stylised CBA was prepared for the RIA process in New Zealand which suggested that the RIA imposes low costs and has the potential to pay its way by identifying more effective interventions. Occasionally the RIS process may avoid significant harm.

The main policy thrust of this chapter is that regulatory management is not solely a policy design problem that lends itself to policy tools. The main focus of the RIA programmes has been to improve the quality of regulatory analysis and policy capability in order to enable better advice to inform decision making and improve the quality of the design of new regulations.

Improving the quality of the flow of new regulations is only the first step. The next challenge is to develop the capability of regulatory institutions to manage the stock of regulations over their life-cycle. This is a management problem, not a policy design problem. This is a different sort of problem that requires a different set of tools associated with smart scanning of regulations, incentives and capability building. Regulatory management focused on scrutinising existing regulations will inevitably be subject to diminishing returns over time.

The next big step – to improve the toolkit available for managing the stock of existing regulations – will be the subject of the next stage of the project.

ANNEX 1 – Summary of the Studies Assessing the Impact of Regulatory Impact Assessments (RIAs)[84]

James Zuccollo^[85]

A15.1 Introduction

This Annex summarises previous studies on the impact of regulatory impact assessments in OECD countries. The studies are separated according to the stage of the process that they focus on:

• Content evaluation papers discuss the extent to which analyses comply with the RIA requirements.
• Output evaluation papers measure the quality of the analysis undertaken.
• Impact evaluation studies assess the influence of the RIA on decision making.
• Outcome evaluations examine whether there was a change in the quality of regulation as a consequence of the RIA process.

A15.2 Content evaluation

The OECD review of RIA practice in all 30 OECD countries (as at 2009), based on a survey of literature and practitioners' experience, is very concerned with the lack of quantification.[86] Often costs and benefits are only qualitatively described, preventing them from being weighed against each other. When these are not quantified and weighed it is easy for the RIA to become a tool for rationalising and justifying a piece of legislation after the fact. Indeed, there are reports of RISs being prepared in the United Kingdom after a piece of legislation has already been passed.[87] Further, they believe the lack of quantification prevents regulators from envisioning the scale of the problem being addressed by a piece of regulation.

The OECD also found a tendency for RIAs to be analytically incomplete. That may mean that plausible regulatory options are not canvassed, costs and benefits are not fully listed, or that the weighing of options is not performed.

A study of European Union (EU) members' RIA practices found that "there is a large gap between requirements set out in official documents and actual … practice. … [T]ypically assessments are narrow, partial and done at a late stage".[88]

More recent EU evidence that scores RIAs and benchmarks against the United States "find[s] that recent EU impact assessments include more economic information" than they did in the past, although important items are still missing.[89] It also "provide[s] evidence that the quality of EU impact assessment increases with the expected cost of [a] proposal."[90] Table 1.1 provides a comparison of whether RIAs prepared in the United States and the EU, include point and range estimates of total costs and benefits. Estimates of costs (in the United States in particular) are provided more consistently than estimates of benefits and net benefits. It also suggests that United States RIAs report costs, benefits and net benefit estimates, much more frequently than their EU counterparts.

Table 1.1: Comparison of whether US & EU RIAs include estimates of total costs and benefits

Source: Hahn (2009),[91] United States data from Hahn and Dudley (2007),[92]  EU data from Renda (2006)[93]

In Australia, Banks reviewing RISs tabled in Parliament found that only 10 per cent of tabled RISs considered compliance costs and only 20 per cent of those attempted to quantify them.[94]

The Australian Productivity Commission found only 80 per cent of required RISs complied with guidelines in 2004 and 2005.

A15.3 Output evaluation

In 2004, the AEI-Brookings Center reviewed the quality of RIA in the United States and concluded that "there is no clear trend in the quality of cost-benefit analysis across administrations."[95] They found that over 70 per cent of the analyses failed to provide any quantitative information on net benefits.[96]

The United Kingdom's National Audit Office, in their 2005 review of the United Kingdom, found that only 40 per cent of RIAs quantified benefits.[97] That year's review by the Better Regulation Task Force judged 75 per cent of RIAs to be of concerning quality.

Jacobs' review of the quality of RIA across the OECD concludes that the "quality of analysis continues to disappoint".[98] In particular, he points to Australia and the European Commission (EC) as governments that have seen a decrease in quality over time.

A15.4 Impact evaluation

A British study of RIAs produced for the House of Commons found only two examples of regulations being withdrawn as a result of an unfavourable RIA.[99]

Work by Renda at the Centre for European Policy Studies in 2006 found few visible improvements to the efficiency of British regulation and concluded that "the cost-saving and efficiency-enhancing potential of the RIA model is still not confirmed by any empirical evidence."[100]

Ambler reviewed the British experience of impact analysis and concluded that "RIAs continue to be used to facilitate regulation rather than challenge the need for it or the quantum."[101]

Radaelli interviewed analysts and key civil servants in a number of EU nations known for their leadership in promoting RIA. In the United Kingdom he found that RIAs are "are not a major component of policy formulation".[102] In Sweden, Denmark and the Netherlands he struggled to discern recognisable RIA systems. He concluded that the United Kingdom is a champion of RIA in Europe, but "if we move from discourse to reality, RIAs are less important tha[n] one would think". Interviews with specialists confirm that "the salience of RIA in the policy formulation process varies from low to moderate".[103]

Hahn and Tetlock in the United States similarly found "insufficient evidence that economic analysis of regulatory decisions has actually had any substantial impact".[104]

Sudiana, in Australia, shows that CBA has been used in a pragmatic way to support decisions rather than as a decision making tool.[105]

Hahn found that in the United States, quantitative studies found negligible impact of RIA on the quality of regulatory decisions.[106] However, he cites a qualitative survey in the United States that used case studies based on interviews with participants in the process. It found that in the majority of cases the participants believed that the RIA process resulted in the selection of a more effective intervention to achieve an unchanged policy objective. Analysts at the United States Environmental Protection Agency subjectively believed that RIA had improved the quality of the rules implemented; however, they did not believe that it had reframed the problem.[107]

A15.5 Outcome evaluation

Farrow examined 69 proposed regulations between 1967–1991 in the United States for their effect on the cost per statistical life saved,[108] finding that regulatory evaluation had, at best, a slight effect on the cost. Rejected rules were marginally more expensive in terms of cost per statistical life saved.

RIA processes were expected by many to cause the quantity of regulation to decline over time, or the cost of regulation to decrease. Evidence from countries that have taken RIA seriously is that neither of these outcomes has eventuated.[109] In Australia, the Business Council's report of May 2005 calculated that the volume of regulation was growing by 10 per cent per year. In the United Kingdom, the Better Regulation Task Force (BRTF) found that, even eight years after introducing RIA, the volume, complexity and cost of regulation continued to grow. They said that in 2005 they found too few examples of better regulation in principle leading to less costly regulation in practice.[110]

A15.6 Conclusions

Peter Carroll surveys the available evidence on the effectiveness of RIA and concludes:[111]

… in all jurisdictions actual practice has shown that the performance of the impact assessment systems has been very limited, with proposed regulation continuing to offer little in the way of a rigorous and convincing evidence base. This paper explores the reasons for the poor performance of evidence-based approaches. Reasons include the varying levels of ministerial and head of department/agency commitment, poor integration of impact assessment systems with existing policy development processes, variable capacity for rigorous, evidence-based policy in departments, and a lack of data on which evidence-based policy can be developed.