Document G5eLDbp8aMrD1e1N7zxRxe3RV

September 29, 1999 MEMORANDUM SUBJECT: Region 8 Needs to Improve Its Performance Partnership Grant Program to Ensure Accountability and Improved Environmental Results Report No. 1999-000209-R8-100302 FROM: Bennie S. Salem Divisional Inspector General TO: William P. Yellowtail Regional Administrator Region 8 Attached is our final report on Region 8's implementation and oversight of its performance partnership grant (PPG) program. The report discusses issues that we believe will help Region 8 build a more effective PPG program leading to improved environmental results. We held an exit conference with your staff on September 23, 1999, where we discussed the Region's response to the draft report and changes we would make based on the Region's response. We summarized the Region's comments at the end of each chapter highlighting those significant issues on which we and Region 8 disagreed. We also included the full text of the Region's comments as Appendix I. The Region suggested that we substitute "National Environmental Performance Partnership System (NEPPS)" for "PPG" in many places throughout the report, because our review included parts of NEPPS, including the Performance Partnership Agreement (PPA) process. Although PPGs and NEPPS share some of the same goals, we did not specifically or comprehensively evaluate the Region's implementation of NEPPS. Therefore, we did not make this change. Instead, we modified the report to clarify our audit scope and the basis for our definition of the "PPG program." Because Region 8 viewed all state PPG work plans as PPAs, we did substitute the term "PPA" for "PPG work plan" where appropriate. We agree that because we did review the portion of PPAs that represented PPG work plan commitments, our findings do address some elements of Region 8's overall NEPPS implementation. While Region 8 PPG project officers provided guidance and coordinated the PPG program, the Region could have more effectively implemented its program. Region 8 and PPG recipients had not fully achieved PPG program goals because some regional program staff disagreed with how the Region implemented the program. As a result, all regional staff were not committed to the Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00001 program and did not actively participate in the PPG process. Region 8 also did not have agreement or consensus on what work plans should include to ensure accountability. Further, some regional program staff did not use or value the end-of-year reports because they were based upon work plans that they stated were inadequate. Without widespread regional staff commitment and participation, Region 8 will have difficulty fully achieving program goals and ultimately, achieving improved environmental results. Action Required In accordance with Environmental Protection Agency (EPA) Order 2750, you, as the action official, are required to provide this office with a written response within 90 days of the final report date. For corrective actions planned, but not completed by the response date, reference to specific milestone dates will assist in deciding whether to close this report. We appreciate the cooperation and assistance your staff provided throughout the audit. The staff showed a genuine interest in working with us to help improve the PPG program. We have no objections to the release of this report to the public. This audit report contains findings that the Office of Inspector General (OIG) has identified and corrective actions OIG recommends. This audit report represents the opinion of OIG, and the findings in this report do not necessarily represent the final EPA position. Final determinations on matters in this audit report will be made by EPA managers in accordance with established EPA audit resolution procedures. If you have any questions, please call Jeff Hart, Audit Manager, at (303) 312-6169. Please refer to report number 1999-000209-R8-100302 on any correspondence. Attachment Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00002 sr% ^ PR^ Office of Inspector General Report of Audit PERFORMANCE PARTNERSHIP GRANTS Region 8 Needs to Improve Its Performance Partnership Grant Program to Ensure Accountability and Improved Environmental Results Report No. 1999-000209-R8-100302 September 29,1999 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00003 Inspector General Division Conducting the Audit: Region Covered: Program Offices Involved: Auditors Conducting Audit: Central Audit Division Denver, Colorado Region 8 Office of Partnerships and Regulatory Assistance Enforcement, Compliance, and Environmental Justice Kim Victor Jean Finn Garrett Westfall Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00004 INTRODUCTION Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results EXECUTIVE SUMMARY The Environmental Protection Agency (EPA) Region 8 awarded approximately $29.7 million in fiscal 1998 performance partnership grants (PPG). PPGs are designed to provide recipients flexibility, reduce administrative burdens and costs, strengthen partnerships, and most importantly, improve environmental results while continuing to address core program requirements. EPA's 1997 strategic plan states that the President's "performance partnership" reinvention initiative provides for increased flexibility in how a program is run in exchange for increased accountability for results. A PPG is a multi program grant made to a recipient from funds allocated and otherwise available for specific environmental programs. PPGs are critical tools for implementing performance partnerships. OBJECTIVES Our specific audit objectives were to answer the following questions: Is Region 8 effectively implementing its PPG program to ensure that recipients accomplish overall PPG program goals? Has Region 8 negotiated PPG work plans that include adequate financial and programmatic accountability? Does Region 8's oversight ensure that recipients meet PPG work plan commitments and use PPG funds efficiently and effectively? 1 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00005 RESULTS IN BRIEF Region 8 Needs to Im prove Its PPG Program to Ensure Accountability and Im proved Environmental Results While Region 8 PPG project officers provided guidance and coordinated the PPG program, the Region could have more effectively implemented its program. Region 8 and PPG recipients had not fully achieved PPG program goals because some regional program staff disagreed with how Region 8 implemented the program. As a result, some program staff were not committed to the program and did not actively participate in the PPG process. Further, Region 8 had not developed clear goals and performance measures to gauge progress in implementing its program. Without widespread regional staff commitment and participation, Region 8 will have difficulty fully achieving program goals. Without specific regional PPG program goals and performance measures, regional and state staff cannot track PPG program progress and demonstrate whether PPGs have resulted in improved environmental results and human health. Work plan commitments were not all time-specific, measurable, quantifiable, and verifiable, and end-of-year reports did not address or provide descriptive responses to all work plan commitments. While Region 8 monitored and evaluated recipients' environmental programs, the Region's oversight did not ensure that all recipients accomplished all work plan commitments and used the funds efficiently and effectively. Region 8 did not have agreement or consensus on what work plans should include to ensure accountability. Further, regional program staff did not use end-of-year reports to ensure work plan commitments were met. Without work plans that clearly demonstrate adequate accountability and descriptive endof-year reports, the Region could not determine whether recipients used PPG funds efficiently and effectively and accomplished improved environmental results. Sierra Club v. EPA 18cv3472 NDCA 11 Report No. 1999-000209-R8-100302 Tier 10 ED 002061 00178457-00006 R EC O M M EN D A TIO N S Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results We recommend that the Regional Administrator (RA) establish a forum where regional program staff can discuss their disagreements and concerns with senior regional managers regarding the Region's PPG program implementation. During the forum, regional staff should brainstorm possible solutions that address their concerns as well as support achieving PPG program goals. Agreement or consensus should be reached on: (1) whether core performance measures are sufficient by themselves, and (2) what amount of detail in work plan commitments provides adequate accountability. Based on the results of this forum, the RA should develop regional guidance on what should be included in a work plan. The RA should also require recipients to submit end-of-year reports that specifically address each work plan commitment and provides information on whether recipients are improving environmental results under PPGs. AGENCY COMMENTS AND OIG EVALUATION Region 8 generally agreed with the factual accuracy of the report and, as a result of the exit conference, agreed with all but 3 of the 32 specific recommendations. Region 8 offered comments to clarify some issues and recommendations, and we modified our report as appropriate. We summarized Region 8 comments at the end of each chapter highlighting those significant issues on which we and Region 8 disagreed. We also included the full text of Region 8's comments as Appendix I. Sierra Club v. EPA 18cv3472 NDCA ill Report No. 1999-000209-R8-100302 Tier 10 ED 002061 00178457-00007 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results This page intentionally left blank. Sierra Club v. EPA 18cv3472 NDCA IV Report No. 1999-000209-R8-100302 Tier 10 ED 002061 00178457-00008 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results TABLE OF CONTENTS Page EXECUTIVE SU M M A R Y ....................................................................................................................i CHAPTERS 1 IN TRO D U C TIO N ........................................................................................................................ 1 P U R P O S E ........................................................................................................................ 1 B A C K G R O U N D ............................................................................................................ 2 SCOPE AND METHODOLOGY ............................................................................... 7 PRIOR AUDIT COVERAGE ......................................................................................8 2 IMPROVEMENTS IN REGION 8'S PPG PROGRAM WOULD HELP ACCOMPLISH PROGRAM GOALS ....................................................................................11 PPG GUIDANCE DEFINES PROGRAM GOALS ................................................11 PPG PROJECT OFFICERS COORDINATED PPG PR O G R A M .........................12 PPG PROGRAM GOALS NOT FULLY A C H IE V E D ........................................... 13 PPG PROGRAM NEEDED FULL REGIONAL COMMITMENT AND PA R TIC IPA TIO N ...........................................................................................18 REGION 8 NEEDED TO DEVELOP REGIONAL PPG PROGRAM GOALS AND PERFORMANCE MEASURES AND DEFINE STAFF ROLES AND AUTHORITY ................................................................... 21 C O N C L U S IO N ............................................................................................................ 22 R EC O M M EN D A TIO N S............................................................................................ 23 AGENCY COMMENTS AND OIG EV A LU A TIO N .............................................25 3 BETTER WORK PLANS NEEDED TO ENSURE ADEQUATE A CCO U N TA BILITY ............................................................................................................ 27 REGULATIONS AND GUIDANCE DEFINE WORK PLAN REQUIREMENTS ................................................................................................... 27 REGION 8 NEEDED CONSENSUS ON WORK PLAN A C C O U N TA B ILITY ...............................................................................................29 WORK PLANS DID NOT CLEARLY IDENTIFY CORE PERFORMANCE MEASURES .............................................................................31 MOST COMMITMENTS WERE NOT TIME-SPECIFIC, MEASURABLE, QUANTIFIABLE, AND V ER IFIA B L E....................................31 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00009 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results STAFF CONTINUED TO RELY ON ACTIVITY OR OUTPUT M E A SU R E S...............................................................................................................33 CONCLUSION ............................................................................................................ 35 R EC O M M EN D A TIO N S.............................................................................................36 AGENCY COMMENTS AND OIG EV A LU A TIO N ............................................. 37 4 IMPROVED PPG END-OF-YEAR REPORTS NEEDED TO HELP ENSURE WORK PLAN COMMITMENTS WERE MET .............................................39 REGULATIONS AND PPG GUIDANCE DEFINE EPA GRANT O V E R S IG H T ............................................................................................................ 40 PPG END-OF-YEAR REPORTS NOT USED BY ALL REGIONAL S T A F F ................................................................................................. 40 SOME PPG END-OF-YEAR REPORTS WERE INCOMPLETE ...................... 41 MIXED REACTIONS ON MID-YEAR PROCESS AND JOINT END-OF-YEAR REVIEWS ................................................................................... 43 REGION UNSURE IF RECIPIENTS SPENT PPG FUNDS EFFICIENTLY AND E F F E C T IV E L Y ................................................................. 44 EFFECTIVE COORDINATION NEEDED TO ENSURE CORRECT MATCH C A L C U L A T IO N ......................................................................................45 C O N C L U S IO N ............................................................................................................ 45 R EC O M M EN D A TIO N S............................................................................................ 46 AGENCY COMMENTS AND OIG EV A LU A TIO N .............................................48 EXHIBITS 1 SCOPE AND METHODOLOGY .......................................................................................... 49 2 CONTRASTING EXAMPLES OF WORK PLAN COM M ITM ENTS............................. 52 APPENDICES I AGENCY RESPONSE .............................................................................................................53 II A B BR EV IA TIO N S................................................................................................................... 63 III D IST R IB U T IO N ........................................................................................................................64 Sierra Club v. EPA 18cv3472 NDCA Report No. 1999-000209-R8-100302 Tier 10 ED 002061 00178457-00010 PURPOSE Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results CHAPTER 1 INTRODUCTION The Environmental Protection Agency's (EPA) 1997 strategic plan states that the President's "performance partnership" reinvention initiative provides for increased flexibility in how a program is run in exchange for increased accountability for results. Performance partnerships are EPA's new approach to working with states and tribes (recipients) to better implement environmental programs and achieve environmental results. A performance partnership grant (PPG) is a critical tool for EPA's implementation of performance partnerships. Both performance partnerships and PPGs represent a significant shift in how EPA and its partners work together to address human health and environmental protection. This audit is part of the Office of Inspector General (OIG) PPGs and National Environmental Performance Partnership System Issue Area Plan dated September 1998. OIG made a commitment to assess EPA's PPG program implementation and determine what improvements were needed to ensure EPA implemented the program effectively and achieved environmental results. Our specific audit objectives were to answer the following questions: Is Region 8 effectively implementing its PPG program to ensure that recipients accomplish overall PPG program goals? Has Region 8 negotiated PPG work plans that include adequate financial and programmatic accountability? 1 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00011 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results Does Region 8's oversight ensure that recipients meet PPG work plan commitments and use PPG funds efficiently and effectively? BACKGROUND PPG Authority and Purpose Beginning in fiscal 1996, EPA received authority to award PPGs. PPGs were part of EPA's effort to improve how EPA and its partners protect the environment. A PPG is a multi-program grant made to a recipient from funds allocated and otherwise available for a specific environmental program. PPGs provide recipients with the option to combine grant funds from two or more specific environmental grants (such as those authorized under the Clean Water Act and the Clean Air Act) into one or more PPGs. Recipients may then use PPGs to fund activities that are within the scope of the programs included in their PPG. Recipients may include any of the 17 eligible grant programs in their PPG. EPA developed PPGs as a response to recommendations from various internal and external stakeholders to increase recipient flexibility, achieve administrative savings by streamlining the grants process, strengthen EPA partnerships with recipients, and help recipients improve environmental results. These recommendations formed the basis for the PPG program's purposes and goals. PPGs are intended to improve the performance of environmental protection programs by providing recipients with greater flexibility in how they manage and implement programs that use grant funds. This increased flexibility will enable recipients to better coordinate and integrate activities that were fragmented in many statutes, regulations, and programs. PPGs will facilitate funding of activities such as pollution prevention, multi-media permits and enforcement, and community-based environmental protection. PPGs are also intended to reduce 2 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00012 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results administrative burden and ultimately, improve environmental results. EPA designed PPGs to encourage improved environmental results by linking program goals with program outcomes and by increasing the use of environmental indicators and program performance measures. Environmental indicators provide information on the environmental conditions, trends, and results. Program performance measures gauge progress in meeting agreed upon goals. Indicators and performance measures create a foundation for better reporting, monitoring, and assessment of state, tribal, and national environmental conditions. All PPGs must contain a legally binding set of work plan commitments. PPG guidance defines work plan commitments as a description of the PPG program goals and objectives, results and benefits expected, a plan of action, and quantifiable projections of the program and environmental accomplishments to be achieved and the performance measures to be used. ..PPG work plan commitments are the legal basis for the expenditure of federal grant funds and the recipient's matching requirement. PPG program commitments may be contained in categorical or PPG work plans, in a Performance Partnership Agreement (PPA), or in a tribal environmental agreement. A categorical work plan that is used for a PPG work plan would be the same as that prepared for a categorical grant. A PPA is a negotiated agreement between a region and a state that describes jointly developed goals, objectives, and priorities and may, but not necessarily, include work plan commitments that are the basis for the PPG. The agreements typically include the strategies to be used in meeting the PPG commitments, the 3 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00013 Relationship between PPGs, PPAs, and the National Environmental Performance Partnership System Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results roles and responsibilities of the state and EPA, and the measures to be used in assessing progress. Similarly, a tribal environmental agreement is a strategic planning document negotiated between a region and a tribe that sets out environmental goals, objectives, outcomes, outputs, priorities, actions to be taken, and measures of performance. Region 8 consists of six states--Colorado, Montana, North Dakota, South Dakota, Utah, and Wyoming. Region 8 also has within its boundaries 27 tribes residing on 26 reservations. In fiscal 1998, Region 8 allocated $109.6 million for state and tribal environmental program grants. O f this amount, $52.3 million was available to be reprogrammed into PPGs and Region 8 awarded approximately $29.7 million (about 56.8 percent) in PPGs. All 6 Region 8 states and 3 of 27 tribes received 1998 PPGs. Four of the six states used a PPA that included their work plan. The other two states and all three tribes used work plans that were similar to their prior categorical work plans. Because Region 8 viewed all state PPG work plans as PPAs, we used the term "PPAs" where appropriate, when referring to any of the six state PPG work plans. As described m EPA's Five-Year Strategic Plan, the National Environmental Performance Partnership System (NEPPS) provides a framework for defining a new EPA/State relationship and PPGs are a key tool for implementing this framework. On May 17, 1995, state and EPA leaders signed a Joint Commitment to Reform Oversight and Create a National Environmental Performance Partnership System (NEPPS). The objective of this agreement was to accelerate the transition to a new working relationship between EPA and states-one which reflected the advancement made in environmental protection over the preceding 2 decades by both states and EPA. In addition, the agreement recognized that existing policies and management approaches must be modified to ensure continued environmental progress. 4 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00014 PPG Program Organizational Structure Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results While PPAs are listed as one of the seven specific NEPPS components, PPGs are not. Although NEPPS and PPGs share many of the same goals, recipients may apply for PPGs without implementing NEPPS and vice-versa. EPA PPG guidance, dated July 1996 and October 6, 1998, states that the key goals that NEPPS shares with PPGs are: to allow States and EPA to achieve improved environmental results by directing scarce public resources toward the highest priority, highest value activities; to provide States with greater flexibility to achieve those results; to improve public understanding of environmental conditions and choices; and to enhance accountability to the public and taxpayers. Although PPGs and NEPPS share some of the same goals, we did not specifically evaluate the Region's implementation of NEPPS. Elowever, because we did review the portion of PPAs that represented the PPG work plan commitments, our findings do address some elements of Region 8's overall NEPPS implementation. While NEPPS was an agreement between states and EPA, many of the concepts of this performance partnership also apply to EPA's relationship with tribes. Under performance partnerships, EPA and its partners are expected to achieve more integrated environmental management and enhanced environmental results. This new partnership should also enable EPA and its partners to move progressively beyond relying on numbers of permits issued, inspections made, or other similar measures, to performance measures that more directly reflect changes in environmental quality. EPA developed interim guidance in 1996 and 1998 that served as the operating guidance for states and tribes interested in applying for PPGs. Regional offices were to 5 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00015 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results use this operating guidance to develop and implement their PPG programs. The guidance states that the Regional Administrator should designate a single PPG project officer for each PPG award. Region 8's State Assistance Program and Tribal Assistance Program within its Office of Partnerships and Regulatory Assistance were responsible for implementing Region 8's PPG program. State program managers and tribal program managers within these programs, respectively, served as PPG project officers. These project officers were the primary point of contact for PPG grant recipients and were responsible for coordinating and facilitating programmatic and technical aspects of PPGs, PPAs, and tribal PPG work plans within the Region. Region 8's PPG project officers were not program experts and relied upon regional program staff to perform programmatic and technical activities such as work plan negotiations and evaluating recipients' performance. The Region had separate PPG project officers for the two states that had pesticide PPGs. These project officers were located within the Region's Pollution Prevention, Pesticides and Toxics Program, and they coordinated with the State Assistance Program. The State Assistance Program initially had responsibility for the two pesticide PPGs, but the Region determined that it was more efficient for the pesticides program staff to handle the pesticide PPGs. In addition, the two pesticide PPGs were awarded to the individual state's agricultural department rather than its environmental agency. While the Region's PPG project officers were responsible for coordinating the Region's PPG program, regional grants specialists were responsible for PPG administrative functions. Region 8's grants specialists within the Grants, Audit, and Procurement Program in the Technical and Management Services Office were responsible for 6 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00016 SCOPE AND METHODOLOGY Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results reviewing PPG applications and related forms and certifications, preparing documentation related to the grant award and amendments, reviewing financial status reports, and calculating recipients' PPG match. Region 8 staff in the regional suboffice in Helena, Montana, were responsible for Montana recipients' PPGs. Staff included two PPG project officers, one for the State of Montana and one for tribes within Montana, as well as a grant specialist. For the purposes of this audit, we defined "PPG program" as the administrative and programmatic processes and activities Region 8 uses to implement PPG goals including: (1) the consolidation of categorical grant funds into a single grant document (the PPG itself), (2) the development of and documentation of specific work plan commitments supporting the awarding of PPG funds, and (3) regional oversight and recipients' documentation of the accomplishment of those work plan commitments. We performed our audit in accordance with Government Auditing Standards (1994 Revision) issued by the Comptroller General of the United States. Our review included an analysis of program records and other auditing procedures we considered necessary. We conducted our fieldwork from January through July 1999. Our review included five of six states' fiscal 1998 and all three tribes' fiscal 1998 and 1999 PPGs, work plans, and end-of-year reports. We did not verify whether the five state and two tribes actually met their work plan commitments or performed the work they said they did in their end-of-year reports. We performed our fieldwork at Region 8 in Denver, Colorado. We also visited agencies in two states: (1) 7 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00017 PRIOR AUDIT COVERAGE Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results Utah's Department of Environmental Quality and Department of Agriculture in Salt Lake City, Utah; and (2) South Dakota's Department of Environment and Natural Resources and Department of Agriculture in Pierre, South Dakota. See Exhibit 1 for scope and methodology details. OIG has not issued any audit reports related to Region 8's PPG program implementation. However, OIG and Regional staff conducted a joint management assistance review of Colorado's 1997 PPG, and 1997 and 1998 PPAs. As a result of the management assistance review, OIG and regional staff suggested that the State and Region 8 staff: (1) continue developing partnerships but recognize that some State and regional staff had not accepted the PPG program; (2) ensure work plan commitments were timespecific, measurable, quantifiable, and verifiable; (3) agree on end-of-year reporting detail; (4) ensure end-of-year reports assess how well the State was accomplishing its work plan commitments and goals; and (5) agree on regional oversight. In 1996, OIG issued two special reports on demonstration grants awarded to North Dakota and New Hampshire. In 1997, OIG issued a special report on a demonstration grant awarded to Massachusetts. These grants were predecessors to PPGs and contained similar objectives including providing states with more flexibility to address their priorities. The demonstration grant audits found that: (1) regional internal "turf battles" and statutory restrictions were barriers to moving resources to address priority environmental problems; (2) states still used activity-based performance measures due to inadequate guidance or incentive; and (3) some measures were not time-specific, quantifiable, or adequate to provide accountability. The reports found positive aspects of the demonstration grants including some administrative efficiencies and increased 8 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00018 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results cross-media enforcement activities. The U.S. General Accounting Office issued, ENVIRONMENTAL PROTECTION: Collaborative EPAState Effort Needed to Improve New Performance Partnership System, dated June 21, 1999. The audit found that the development of outcome measures has been impeded by an absence of baseline data, inherent difficulty in quantifying certain results, difficulty of linking program activities to environmental results, and considerable resources needed for high-quality performance measurement. The audit also found that EPA's oversight of states participating in the new system had been realized only to a limited degree. Among the factors complicating EPA's oversight were: (1) statutory and/or regulatory requirements in some cases prescribed the kind of oversight required by EPA over states, (2) reluctance by EPA staff to reduce oversight without the measures in place to ensure that environmental quality would not be compromised, and (3) the inherent difficulty in "letting go" on the part of some EPA staff that have implemented the existing EPAstate oversight arrangement for years. The audit also identified a number of reported benefits associated with the new system that included improving communication about program priorities among EPA and state program staff and allowing states the option to shift resources under the PPG program. The U.S. General Accounting Office also issued, ENVIRONMENTAL INFORMATION: EPA Is Taking Steps to Improve Information Management, but Challenges Remain, dated September 1999. The audit found that EPA's ability to evaluate the outcomes of its programs in terms of changes in the environment is limited by gaps and inconsistencies in the quality of its data. The report stated that of the 357 measures of performance that EPA has developed for use during fiscal year 2000 to report its accomplishments under the Government Performance and 9 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00019 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results Results Act, EPA reports that only 71 measures will reflect environmental outcomes. EPA's program offices will have to overcome: (1) difficulties in establishing cause-and-effect relationships between program activities and environmental outcomes, (2) a lack of reliable baseline data against which to measure progress and a more generalized lack of reliable data about the environment, and (3) constraints on the resources for gathering and analyzing the data. Sierra Club v. EPA 18cv3472 NDCA 10 Report No. 1999-000209-R8-100302 Tier 10 ED 002061 00178457-00020 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results CHAPTER 2 IMPROVEMENTS IN REGION 8'S PPG PROGRAM WOULD HELP ACCOMPLISH PROGRAM GOALS Region 8 could have more effectively implemented the PPG program. The PPG program goals are to provide flexibility, reduce administrative burdens and costs, strengthen partnerships, and most importantly, improve environmental results. Region 8 PPG project officers provided guidance and coordinated the PPG program. However, Region 8 and PPG recipients had not fully achieved PPG program goals primarily because some regional program staff disagreed with how Region 8 implemented the program. As a result, some program staff were not committed to the PPG program and did not actively participate in the PPG process. Further, Region 8 had not developed clear goals and performance measures to gauge progress in implementing its PPG program. Without widespread regional staff commitment and participation, Region 8 will have difficulty fully achieving PPG program goals. Without specific regional PPG program goals and performance measures, regional and state staff cannot track PPG program progress and demonstrate whether PPGs have resulted in improved environmental results and human health. PPG GUIDANCE DEFINES PROGRAM GOALS PPG guidance, dated July 1996 and October 6, 1998, describes four goals of the PPG program. Flexibility. Recipients will have the flexibility to address their highest environmental priorities, while continuing to address core program requirements. Administrative Savings. Recipients and EPA can reduce administrative burdens and 11 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00021 PPG PROJECT OFFICERS COORDINATED PPG PROGRAM Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results costs by reducing the number of grant applications, budgets, work plans, and reports. Strengthened Partnerships. EPA will develop partnerships with recipients where both parties share the same environmental and program goals and deploy their unique resources and abilities to jointly accomplish those goals. Improved environmental results. PPGs will encourage recipients to improve environmental results and more effectively link program activities with environmental goals and program outcomes; and develop innovative pollution prevention, ecosystem, and community-based strategies. Region 8's PPG project officers provided guidance on PPGs and coordinated meetings related to the PPG process. They provided guidance on PPA negotiation process, endof-year reviews, and carryover funds. The project officers also provided a detailed schedule of major milestones related to the PPG program and helped regional staff meet those milestones. PPG project officers, working with regional grant specialists, also provided PPG grant training workshops. They provided regional kickoff meetings at the beginning of each fiscal year that addressed the annual planning, development, and evaluation process. They also coordinated mid-year meetings between senior regional and state managers as well as the annual state directors' meeting with regional managers. Recipient staff stated that an important function PPG project officers served was providing a single point of contact for recipients. One state representative stated that 12 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00022 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results the main reason his program achieved success with the PPG was because of the working relationship between the PPG project officer and his staff. Another state representative stated that coordinating with a single liaison within the Region rather than having to coordinate with several regional staff had been very beneficial in the PPG process. PPG PROGRAM GOALS NOT FULLY ACHIEVED Full Flexibility Potential Not Achieved Despite PPG project officers' guidance and coordination, Region 8 did not fully achieve PPG program goals. PPG recipients: (1) had not achieved the full flexibility potential to address their highest priorities, (2) had realized varying degrees of administrative savings, (3) had strengthened partnerships, and (4) were uncertain about whether the PPG program resulted in improved environmental results. While we found some positive examples of success, barriers within Region 8 and recipient agencies prevented PPG program goals from being fully achieved. States had not fully achieved one of the most attractive aspects of the PPG to recipients-flexibility. According to regional and state staff, most states had not shifted funds from one program to another to address their highest environmental priorities. Both regional and state program staff identified insufficient resources, strong constituencies (i.e., state water boards and commissions), state legislative reporting requirements, internal turf battles, and lack of a process to identify priorities and make investments or disinvestments as barriers to flexibility. Some recipients stated, however, that while they had not yet utilized PPG flexibility, the fact that the potential flexibility existed was one of the strong incentives for keeping a PPG. Some state program staff said that insufficient resources prevented them from shifting funds to strategically plan, identify priorities, and address environmental priorities 13 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00023 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results beyond basic core environmental program requirements. For example, Wyoming's resource limitations did not allow staff time for strategic planning. Insufficient resources kept Wyoming focused on trying to achieve the basic core program requirements which left no time or resources for strategic planning. According to some state staff, federal grant funds barely supported basic core program requirements. For example, one state program director said his program was "anorexic" in terms of funding. Strong constituencies and state legislative reporting requirements prevented state staff from shifting funds from one program to another. Each program had strong constituencies that put pressure on states to ensure no reduction in funding occurred in the programs. Some state legislatures required states to report exactly how funds were spent and the source of the funds. As a result, some states traced expenditures by specific program and were discouraged from shifting funds from one program to another. State internal turf battles also hindered state flexibility to shift funds among programs. For example, one state program director said that if his program funds were put into a large pot with all of the other programs' funds, then another program director would get his funds. To ease program staff fears, some state agency directors had decided not to shift funds across programs. Some recipients had achieved some flexibility by using carryover funds to address unfunded priorities. For example, South Dakota had used some of its PPG carryover funds to implement projects such as concentrated animal feeding permitting and geographical information systems mapping. South Dakota staff stated that PPG carryover funds provided funding to perform special projects that would not have been possible under categorical grants. 14 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00024 Recipients Achieved Varying Degrees of Administrative Savings Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results Another barrier to shifting funds was the lack of a well thought out disinvestment/investment process. While work plan negotiations considered states' strategic plans, several state and regional staff said that establishing a clear disinvestments/investment process to address environmental priorities was difficult and had not yet been done. In consultation with the Region, recipients should strategically plan their annual priorities upfront, identify funding needed to accomplish those priorities, and determine where disinvestments/investments can be made to benefit the environment. In contrast to states, tribes realized substantial flexibility moving from categorical grants to PPGs. Tribes did not have strong constituencies, legislative reporting requirements, or internal turf battles that hindered flexibility. One tribal staffmember said that the Tribe's Council did not require staff to continue tracking funds by individual program once the Tribe received a PPG. A tribe's environmental office often consisted of one person working on several different programs, eliminating any turf issues. As a result, at least one tribe was able to more easily create a multi-media program that could address their environmental priorities. Recipients realized varying degrees of administrative savings. While most states realized minimal administrative savings, tribes realized significant savings. Recipients said that generally, PPGs reduced the number of grant applications and budget information submitted to the Region. For example, North Dakota identified in its fiscal 1998 end-of-year report that it had experienced administrative savings in the preparation, negotiation, and administration of one grant versus nine separate grants. However, most states still had to maintain program-specific cost information because of legislative requirements. In contrast, one tribal representative stated that they did not have to track program-specific cost information. Tribal staff also said that PPGs simplified the administrative 15 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00025 Some Improvements In Strengthening Region 8 and Recipients' Partnerships Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results process resulting in significant administrative savings. One tribal representative said PPGs reduced administrative costs by 700 percent. Neither states nor tribes realized reduced administrative burdens or costs in preparing work plans and reports. While some state staff stated that PPAs included fewer commitments than categorical work plans, staff still prepared the same number of work plans. The state staff simply consolidated these work plans into a single document. Some state and regional program staff said that this new PPG process was burdensome because PPA negotiations were frustrating and time consuming. In addition, some state staff said that they had to report more on PPGs than categorical grants. For example, they now had to report on core performance measures, statutory and regulatory requirements, and additional performance measures related to the state's overall goals. In addition, tribal staff found it burdensome to report on environmental accomplishments. Of the four PPG program goals, the Region has made the most progress in strengthening its partnerships with recipients. Recipient managers and program staff stated that a key benefit of the PPG program was the improved partnerships that had developed between recipients and EPA. For example, two state program staff said that the PPG helped some regional staff move from a command and control approach to more of a joint partnership. Other regional and recipient staff said that under the PPG process, both Region 8 and recipients were better informed about the other's priorities. State representatives stated that the Region was now recognizing the state's priorities as valid. In addition to improving the relationship between EPA and recipients, another expected benefit of the PPG program is to develop partnerships where both parties deploy their unique resources and abilities to jointly accomplish those 16 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00026 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results goals. For example, Region 8, Utah Department of Environmental Quality, and local health departments in southern Utah formed a three-way partnership with an emphasis on problem solving rather than oversight. The partnership included a community-based concept empowering elected officials and citizens to develop a list of local environmental priorities and problems to be addressed jointly. Utah's Deputy Director for the Department of Environmental Quality attributed the success of the partnership to each partner bringing knowledge, assistance, and possible solutions to the discussion table and focusing on the problem, rather than oversight roles. Despite the improvements cited by recipients and PPG project officers, some regional program staff stated that partnerships had not greatly improved overall. In some instances, regional program staff stated that the PPA process had strained relationships between regional and state staff. One regional program manager stated that the partnership with the states had definitely not improved and that some regional staff viewed the partnership as nonexistent. He stated that PPA negotiations were difficult and many times resulted in shouting matches with other state staff. Several state representatives said that regional enforcement staff did not consistently act in the spirit of partnerships. For example, one state director said that regional enforcement staff were primarily focused on oversight and were not willing to provide flexibility or compromise on enforcement issues. A regional enforcement manager stated that while regional enforcement staff agreed with the concept of performance partnerships, recipients needed to recognize that regional enforcement staff still had oversight responsibilities to ensure that states and regulated industry complied with environmental laws. While the relationship between regional enforcement and 17 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00027 Improved Environmental Results Uncertain Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results state staff was still contentious, regional staff have taken some important steps to improve the relationship. According to regional staff, enforcement managers visited each of Region 8's six states to discuss how the partnership between regional and state staff could be improved. For example, regional enforcement and South Dakota state staff agreed on a process to coordinate enforcement actions and allow the Region to perform its oversight role. The most important PPG program goal, in our opinion, was improved environmental results. However, Region 8 and states were uncertain whether the PPG program had improved environmental results. According to PPG guidance, improved environmental performance will be achieved by increasing the use of environmental indicators and program performance measures, and decreasing the reporting of inputs and activities. While Region 8 and some states have made efforts to develop and use indicators and outcome measures, they had primarily relied upon outputs and activities. Regional staff agreed that until good outcome measures were developed with appropriate indicators, environmental results would be difficult to determine. One tribal representative stated that her tribe had not yet moved toward measuring environmental results and that the tribe was still focused on meeting the programs' core requirements. Further, a tribal PPG project officer stated that PPGs emphasized reporting on accomplishments and results, but tribes were accustomed to focusing on work plan development rather than end-of-year reporting. PPG PROGRAM NEEDED FULL REGIONAL COMM ITM ENT AND PARTICIPATION Many regional program staff disagreed with how Region 8 implemented the PPG program. Specifically, they disagreed with: (1) senior regional management's definition of partnership as demonstrated by their actions; (2) the lack of detail in work plan commitments; and (3) as with categorical grants, the Region not enforcing any 18 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00028 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results consequences for noncompliance with grant conditions. As a result, some regional staff did not actively participate in the PPG process. We concluded that a lack of active participation among regional staff lead to a lack of commitment to the PPG program. Some regional program staff disagreed with senior regional management's definition of partnership as demonstrated by their actions. For example, one regional staffmember stated that senior managers considered everything about the PPG as negotiable and that regional staff should be flexible with recipients. Flowever, regional program staff believed that some things, such as following statutory and regulatory requirements, were not negotiable. According to regional program staff, senior management in the Region and headquarters said that EPA Government Performance and Results Act commitments should not be included in work plans because core performance measures provided all necessary reporting information. Flowever, regional program staff stated that states did not formally report all necessary information and regional program staff had to contact states to obtain additional information. As a result, regional staff stated that they were "stuck in the middle" between EPA reporting requirements and what senior managers were negotiating with recipients. Some regional program staff disagreed with the lack of detailed work plan commitments. For example, in negotiations with recipients, senior regional managers had decided that if a state chose to include only national core performance measures as their work plan commitments, then regional program staff could not include any additional measures. According to regional program managers and staff, the PPG program became a means to provide states with grant funds without adequate financial and programmatic accountability. In fact, several regional program staff said that states wanted "EPA to leave the money on the stump and walk away." We discuss regional program staffs concerns regarding work plan commitments 19 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00029 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results in greater detail and provide related recommendations in Chapter 3. Some regional program staff disagreed with the Region's approach of not enforcing any consequences for noncompliance with grant conditions. For example, one regional program manager stated that the Region would not withhold grant funds even when recipients' submitted endof-year reports 4 months late. A regional program staffmember stated that senior regional managers also did not withhold grant funds when one state intentionally did not report on all its work plan commitments. As a result of regional program staff disagreements, some staff did not actively participate in many PPG activities. For example, some regional program staff quit attending PPG related meetings and provided only cursory reviews of PPAs and end-of-year reports. Region 8's PPG project officers recognized that regional program staff disagreed with many aspects of the Region's PPG program. PPG project officers identified in a 1997 summary of lessons learned from the PPA development process that one major problem was a lack of strong commitment throughout the Region and states. However they stated, and we agreed, that it would take both PPG project officers and regional program managers and staff working together to have a successful PPG program. While PPG project officers could coordinate and champion the PPG program, the program would not be successful until all regional program staff made a commitment to the program. The success of the PPG program requires Region 8 staff letting go of the traditional EPA oversight role and focusing on how best to work with recipients to achieve environmental results. However, the process of "letting go" involves trust in the new system, and this can only happen if regional staff disagreements are addressed. 20 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00030 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results REGION 8 NEEDED TO DEVELOP REGIONAL PPG PROGRAM GOALS AND PERFORMANCE MEASURES AND DEFINE STAFF ROLES AND AUTHORITY Regional PPG Program Goals and Performance Measures Needed to Ensure Effective Implementation and Progress Regional Staff Unclear About Roles and Authority The Region recognized the importance of establishing goals and evaluating progress, but had not developed specific regional PPG program goals and measures to determine whether it was progressing toward achieving those goals. In addition, regional staff were unclear about their roles and authority. Furthermore, some regional program staff suggested additional responsibilities for PPG project officers. While PPG project officers recognized the importance of establishing goals and performance measures for the Region's PPG program, Region 8 had not developed specific regional PPG program goals and performance measures. The Region did not have performance measures and related milestones in place to periodically measure how the Region and recipients were progressing toward PPG program goals. For example, Region 8 had not specified a measurable and quantifiable goal for administrative savings. As a result, the Region could not evaluate recipients success in cost reduction. As part of its 1998-1999 priority-setting process, the Office of Partnerships and Regulatory Assistance stated that without measuring and assessing the Region's actions, taking success stories to the public would be difficult. All regional staff should have input in developing the Region's PPG program goals and performance measures to ensure commitment from all regional staff. Some regional staff were unclear about the respective coordination roles of PPG project officers and program contacts. For example, one PPG project officer and a state program contact each assumed that the other incorporated changes into a state's draft PPA. However, when the PPA was routed for final concurrence, the issues had not been 21 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00031 Regional Staff Suggested Additional Responsibilities for PPG Project Officers CONCLUSION Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results resolved or included in the PPA. PPG project officers stated that they were also often confused about the authority of the program contacts. PPG project officers stated that they encountered problems when: (1) a program contact's role and authority were not clear to them or other program staff, (2) a contact's other duties affected the degree of follow-up that was completed with staffmembers, and (3) PPG project officers had conflicting expectations of what program contacts were going to do. Regional program staff also stated that PPG project officers could do more to assist the regional program staff. Several regional program staff provided suggestions. One regional program staffm ember stated that project officers could review PPG end-of-year reports for completeness and responsiveness to all commitments. While the program staffmember recognized that the project officers did not have all the needed technical expertise, she believed that PPG project officers could more thoroughly review the reports to ensure that recipients responded to all commitments. In addition, if the program staff decided that additional information was required, PPG project officers could contact the recipient. Another regional staffmember suggested that project officers become more knowledgeable about the different programs. She suggested that the project officers attend program staff meetings and attend state directors' teleconferences when annual priorities and grant allocations were discussed. One PPG project officer stated that when the Region reorganized, PPG project officers attended program staff meetings. However, he stated that PPG project officers stopped attending the meetings because project officers determined that their attendance was no longer necessary. Not all of the benefits that EPA thought the PPG program would provide were realized in terms of seeing and 22 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00032 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results measuring environmental improvements. Regional and state staff could not determine or demonstrate whether the PPG had resulted in improved environmental results and human health. Region 8 and recipients had identified several examples of progress toward achieving each PPG goal. However, various barriers within the Region and recipients' agencies prevented them from fully achieving those goals. Some regional staff disagreed with the Region's implementation of the PPG program and did not commit to the PPG program or actively participate in the PPG process. Without widespread regional staff commitment and participation, Region 8 will continue to have difficulty fully accomplishing PPG program goals. Without specific regional PPG program goals and performance measures, regional and state staff could not track PPG program progress and demonstrate whether PPGs have resulted in improved environmental results and human health. R EC O M M EN D A TIO N S We recommend the Regional Administrator: 2-1. Work with recipients to develop a strategy to overcome barriers to accomplishing the four PPG program goals. Regional staff assistance could include obtaining examples from other regions and states on their successful efforts. 2-2. Strongly encourage states to initiate a joint EPAstate strategic planning process to identify state environmental priorities and help make decisions on investments and disinvestments. 2-3. Highlight and communicate successful efforts to improve partnerships throughout the Region, particularly enforcement partnerships. 23 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00033 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results 2-4. Emphasize during senior regional and program staff meetings, that the PPG program requires both PPG project officers and regional program staff working together to achieve program success. 2-5 Establish a forum where regional program staff can discuss their disagreements and concerns with senior managers regarding the Region's PPG program implementation. While addressing regional staff concerns, regional staff should brainstorm possible solutions that support achieving PPG program goals. 2-6. Seek to reach consensus or a common understanding among EPA and recipient staff on the definition of partnership and on the behaviors that demonstrate partnership. 2-7. Direct staff to clearly identify and elevate recipients' noncompliance with grant conditions to senior regional staff. 2-8. Require regional program staff to attend and actively participate in all appropriate PPG related meetings. 2-9. Develop and periodically adjust specific, realistic goals and related performance measures with specific milestones for the Region's PPG program. All appropriate regional staff should participate in developing these goals and performance measures (not just PPG project officers). 2-10. Emphasize Region 8's PPG program goals at divisional staff meetings and regional all-employees meetings. In addition, the Regional Administrator should consider identifying Region 8's PPG program goals as part of the Region's strategic plan and priorities. 24 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00034 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results 2-11. Consider including the Region's PPG program goals and performance measures as part of Region 8's employee evaluation and reward system. 2-12. Clearly define the roles and authority of all those involved with the PPG program, as recommended in the State Assistance Program's General Assessment o f the Performance Partnership Process (undated) and distribute to all regional staff. 2-13. Identify additional responsibilities PPG project officers could perform to assist regional program staff such as those suggested by regional program staff. (See page 22 of this report.) AGENCY COMMENTS AND OIG EVALUATION Region 8 concurred with all the recommendations in Chapter 2 and offered some comments. We agreed with some of the Region's comments and made some minor modifications to our original recommendations. We also included the full text of the Region's comments as Appendix I. Region 8 disagreed with our definition of the "PPG program." The Region suggested that we substitute "NEPPS" for "PPG" in the title of Chapter 2, and in many other places throughout the report, because our review included parts of NEPPS including the PPA process. We modified the report to clarify the basis for our definition. Specifically, we clarified the basis of our definition of the "PPG program" in Chapter 1, and added some clarifying information in Exhibit l(Scope and Methodology). The Region suggested that recommendation 2-2 could be strengthened and used as a guiding principle for midyear meetings with recipients. The Region stated that tying program priorities to budget priorities will encourage better strategic planning. The Region also stated that some changes need to be made at the National Program Management level first. 25 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00035 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results With regard to recommendation 2-9, Region 8 stated that headquarters should develop PPG program goals and performance measures with specific regional goals as a subset. Region 8 also said it should have a system to measure its results. Sierra Club v. EPA 18cv3472 NDCA 26 Report No. 1999-000209-R8-100302 Tier 10 ED 002061 00178457-00036 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results CHAPTER 3 BETTER WORK PLANS NEEDED TO ENSURE ADEQUATE ACCOUNTABILITY We could not determine whether Region 8 work plans included adequate financial and programmatic accountability. Although we found criteria on what components should be included in work plans, we did not find specific criteria that defined what a work plan should include to ensure adequate financial and programmatic accountability. Further, Region 8 did not have agreement or consensus on what work plans should include to ensure accountability. Recipients' work plans varied in format and language, making it difficult to determine whether they included required core performance measures. In addition, not all of the core performance measures and other measures used were time-specific, measurable, quantifiable, and verifiable. Regional and state staff relied primarily on activity or output measures because staff found it difficult to develop outcome measures, did not have good examples of outcome measures, and had not developed or identified baseline data. Also, some regional program staff stated that they and EPA National Program Managers continued to ask for output data needed to meet statutory and regulatory requirements. Without work plans that clearly demonstrate adequate accountability, the Region could not determine whether recipients used PPG funds efficiently and effectively and accomplished improved environmental results. REGULATIONS AND GUIDANCE DEFINE WORK PLAN R EQ U IR EM EN T S The 1996 and 1998 PPG guidance outlines work plan requirements as the basis for the management and evaluation of performance under the grant agreement. The work plan should include core program commitments (goals, performance measures, program activities) derived 27 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00037 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results from statutes, regulations, and standing legal agreements between EPA and recipients. Performance measures that are PPG program commitments must be time-specific, measurable, quantifiable, and verifiable. In addition, EPA encourages all recipients to adopt outcome and outputorientated performance measures that track program performance as well as environmental conditions and trends. Appropriate accountability provisions are essential in designing the new PPG program. The PPG guidance states that: A fundamental goal of EPA's efforts to design accountability provisions into PPGs is to begin moving Federal, State, and Tribal programs toward the use of results-oriented measures of environmental and program performance that are understandable and meaningful to the public. Further, the work plan should also establish procedures (e.g., mid-year and end-of-year reviews, reporting requirements, joint activities) that EPA and the recipient will use for evaluating accomplishments, discussing progress, and making adjustments to meet milestones. The PPG guidance states that EPA should work with recipients to balance the need to maintain core program requirements with the need to incorporate program flexibility and move toward program performance measures and environmental indicators. Core performance measures are a limited set of national measures designed to help gauge progress toward protection of the environment and human health as well as provide a national picture of the status of the environment. EPA and the Environmental Council of the States developed core performance measures to help focus EPA 28 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00038 REGION 8 NEEDED CONSENSUS ON W ORK PLAN ACCOUNTABILITY Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results and state attention on improving how they measured the effectiveness of their environmental protection efforts. Region 8 did not have agreement, or at least consensus, on what should be included in work plans to ensure adequate financial and programmatic accountability. EPA guidance established general work plan requirements, such as work plans specifying commitments and a time frame for their accomplishment. However, we did not find criteria or guidance that specifically defined what regional and state staff should include in work plans to ensure adequate financial and programmatic accountability. Regional staff disagreed on: (1) whether core performance measures were sufficient by themselves, and (2) what amount of detail in work plan commitments provided adequate accountability. As a result, some regional program staff said they did not use and derived no value or benefit from PPAs. Region 8 staff did not have agreement or consensus on whether core performance measures were sufficient by themselves to ensure adequate financial and programmatic accountability. Some senior regional and state managers had determined that if a state chose to include only core performance measures as their work plan commitments, then regional program staff could not include any additional measures. In contrast, some regional program managers and staff stated that core performance measures did not provide sufficient detail to ensure adequate accountability and wanted to include additional work plan commitments. They stated that core performance measures did not encompass all basic core program requirements such as those contained in delegation agreements. For example, one program manager said the State of Montana declined to input any information into the National Toxic Inventory System because the core performance measures did not specifically require such reporting. 29 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00039 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results Region 8 staff did not have agreement or consensus on whether the amount of detail in work plan commitments provided adequate financial and programmatic accountability. For example, senior regional management and PPG project officers determined that commitments related to statutes and regulations did not need to be repeated in the PPA and that referencing them was sufficient. Flowever, several regional program staff stated they pushed for commitments that included specific details, such as those required by statutes and regulations, but ultimately senior regional management disagreed and did not allow the additional specific details to be included in the work plans. As a result, some regional program staff quit participating in regional-state PPA related meetings and did not use PPAs to hold recipients accountable. Regional program staff had identified PPA accountability problems in 1997. In a 1997 Region 8 summary of lessons learned related to the PPA development process, regional program staff identified that work plans should: (1) provide clear measures of accountability, (2) act as both the grant agreement and a management or tracking tool, and (3) clearly state how PPAs fit in with other agreements such as delegation and enforcement agreements. Some regional program staff stated that prior categorical grant work plans were very detailed and state staff viewed that as micro-managing state programs. However, regional program staff stated that the Region's implementation of the PPG had swung the pendulum too far in the opposite direction with few details in work plan commitments. Regional program staff stated that PPAs should be a management tool for both regional and state staff and should reflect specific program activities and goals. One regional program manager stated that PPAs were not serving all Region 8 staff needs. Sierra Club v. EPA 18cv3472 NDCA 30 Report No. 1999-000209-R8-100302 Tier 10 ED 002061 00178457-00040 WORK PLANS DID NOT CLEARLY IDENTIFY CORE PERFORMANCE MEASURES Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results Work plans varied in format and language from program to program and from state to state, which made it difficult to determine whether work plans included all required and appropriate core performance measures. Only one of the five state work plans clearly identified the core performance measures for each of the three programs we reviewed. In the other four work plans, core performance measures were not clearly identified in one or more programs. For example, North Dakota's work plan included all the surface water quality program core performance measures, but were not identified as such. The hazardous waste program included some of the core performance measures among its commitments but did not clearly identify the core measures, or did not clearly tie work plan commitments with the national core measures. Some of the commitments may have been core measures that had been appropriately modified. However, without a clear tie, it was difficult to distinguish between such a commitment and a commitment that the state and the Region had added. In contrast, Utah clearly tied each work plan commitment to its respective national core performance measure even if the language had been drastically modified. MOST COMMITMENTS WERE NOT TIMESPECIFIC, MEASURABLE, QUANTIFIABLE, AND VERIFIABLE Most of the work plan commitments we reviewed were not time-specific, measurable, quantifiable, and verifiable. Of the 727 commitments in the five state work plans reviewed, only 76 (or about 10.5 percent) met all four attributes. PPG guidance states that performance measures included in PPG work plans as commitments must be measurable, quantifiable, and verifiable. In a memorandum dated December 13, 1995, OIG stated commitments must also be time-specific. The PPG guidance did not define any of these terms; therefore, we established our own definitions for the purpose of our analysis. Time-specific means a date or a 31 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00041 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results clear statement that the commitment is to be accomplished within a specific timeframe. Measurable means the relative change in something can be determined. We defined quantifiable as having a numerical value. Verifiable means it could be proved true or accurate. Many work plan commitments were not time-specific. Each of the five state work plans we reviewed included commitments that had "on going," "as necessary," "as needed," "as required," and "as requested" for milestone dates. For example, Colorado's ambient water quality monitoring program had a measure, "Where requested, work with local communities to identify solutions to impending planning/wastewater management issues...." South Dakota's surface water quality program had "on going" as the milestone for its measure, "Review criteria for toxicants other than priority pollutants and adopt criteria to protect the designated use." In addition, some of the work plans contained commitments that did not include annual measures or targets to quantify annual program performance. The following are examples from several PPAs: "Provide facility specific compliance information through automated data systems," "Implement federally approved requirements on schedule and encourage higher uses of oxygenates through Reg N o.13," "Investigate waterborne disease, spills, chemical contamination events, and other water quality issues affecting PWS," and "Target five laboratories and meet with them so they know what resources we can 32 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00042 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results offer them." Most commitments in work plans were not measurable, quantifiable, and verifiable as shown in the examples below: "Improve customer service through operation of the automobile inspection and maintenance program," "Improve automation and achieve further streamlining of the entire permit process," and "Monitoring capability is enhanced along Wasatch Front " Many of the work plan commitments we reviewed would be measurable, quantifiable, and verifiable if specific targets and/or baselines were identified as part of the commitment. See Exhibit 2 for contrasting examples of commitments that met these attributes with those that did not. STAFF CONTINUED TO RELY ON ACTIVITY OR OUTPUT MEASURES Regional and recipient staff continued to rely primarily on activity or output measures because staff found it difficult to develop and report on outcome measures, did not have good examples of outcome measures, and had not developed or identified baseline data. Commitments for the programs we reviewed in the five state and the three tribal work plans were primarily activity or output measures. Some regional staff said that EPA National Program Managers in addition to regional program staff continued to ask for output data needed to meet statutory and regulatory requirements. Regional and state staff said that they relied on activity or 33 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00043 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results output measures because developing and reporting on outcome measures was difficult. For example, one state program staffmember stated that developing outcome measures for preventive programs was extremely difficult because staff were unsure how they would measure the amount of pollution they prevented and validate it. A regional program staffmember stated that reporting on outcome measures for the pesticides program was difficult because some data did not exist, such as the reduction in pesticide poisonings. In addition, some regional program staff stated that program staff had not made any effort to develop outcome measures because staff perceived it as requiring time away from their real work. Regional and recipient staff relied on activity measures because they stated that they did not have good examples of outcome measures. For example, one state program staffmember stated no one had provided or developed good examples of outcome measures. Regional and recipient staff said they relied on activity measures because they had not developed or identified baseline data. A PPG project officer said that states and EPA needed to establish a baseline and measure progress from that baseline. For example, one state's work plan commitments included "Monitoring capability is enhanced along Wasatch Front." However, without a baseline that first establishes what the State's capability is, the State would have difficulty measuring and demonstrating "enhanced" capability. EPA National Program Managers and some regional program staff continued to ask for output data needed to meet statutory and regulatory requirements. One regional program staffmember said that the reason for the continued use of output measures was that these types of measures were required by statutes, regulations, delegation agreements, national program guidance, and in some cases, by national mandate. EPA guidance states, and we agreed, 34 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00044 CONCLUSION Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results that work plans should include a mixture of output and outcome measures. We could not determine whether recipient work plans included adequate financial and programmatic accountability. We did not find, and Region 8 had not developed, specific criteria that defined what a work plan should include to ensure accountability. Further, Region 8 did not have agreement, or at least consensus, on the amount of detail necessary in work plan commitments. Two different views existed in Region 8 regarding work plan commitments. One view was that everything was negotiable and another was that statutory and regulatory requirements were not negotiable. We agree that statutory and regulatory requirements are not negotiable and should be included as part of the work plan commitments. Work plans varied in format and language, making it difficult to determine whether they included required core performance measures. In addition, not all measures in the work plans were time-specific, measurable, quantifiable, and verifiable. Without work plans that clearly demonstrate adequate accountability, the Region could not use them to hold recipients accountable. In addition, some regional staff did not view PPAs as providing value or benefit as a day-to-day working document. Recipient work plans we reviewed included commitments that were primarily output measures. Regional and state staff relied primarily on activity or output measures because staff found it difficult to develop outcome measures, did not have good examples of outcome measures, and had not developed or identified baseline data. Also, some regional program staff stated that EPA National Program Managers and regional program staff continued to ask for output data needed to meet statutory and regulatory requirements. Although activity or output measures provide valuable information, they do not reflect 35 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00045 R EC O M M EN D A TIO N S Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results environmental improvements and results. Outcome measures with appropriate indicators and baseline data are needed in addition to output measures to adequately assess environmental results. We recommend that the Regional Administrator: 3-1. Develop regional guidance on what should be included in a PPA or PPG work plan based upon regional staff agreement, or at least consensus. 3-2. Work with senior regional managers, regional program staff, and PPG project officers to come to agreement or consensus on: (1) whether core performance measures are sufficient by themselves, and (2) what amount of detail in work plan commitments provides adequate accountability. Consider using an impartial facilitator for this process. 3-3. Require that regional staff adhere to the following guidelines during the development of a regional agreement of accountability provisions for the work plan: a. that the ultimate goal is to develop an agreement that ensures adequate accountability and flexibility, b. that everyone's perspectives are heard, and c. that senior regional managers and staff will support the agreement or consensus reached. 3-4. Require regional and state staff to clearly identify all core performance measures, including modified measures, included in the work plans. 36 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00046 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results 3-5. Establish working definitions for the terms timespecific, measurable, quantifiable, and verifiable. Direct regional staff to ensure that all annual work plan commitments are time-specific, measurable, quantifiable, and verifiable. 3-6. Help regional and state staff move toward using outcome measures by requiring that: a. Regional program managers increase the priority of developing outcome measures with recipients to ensure that staff have the time and resources needed to adequately establish these measures, b. PPG project officers and regional program staff develop and seek out good examples of outcome measures for reference by all regional and recipient staff, c. Regional program managers and staff work with recipients to develop baseline data needed to adequately measure outcomes, and d. Senior regional managers continue encouraging National Program Managers and regional program staff to move more toward outcome measures rather than just activity or output measures. AGENCY COMMENTS AND OIG EVALUATION Region 8 concurred with recommendation 3-1 but did not concur with most of the rest of our recommendations because they were, in the Region's opinion, prescriptive subparts of 3-1. However, senior regional managers concurred at the audit exit conference with the revised recommendations 3-2 and 3-3 as shown above. We 37 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00047 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results highlighted below those significant issues on which we and Region 8 disagreed. We also included the full text of Region 8's comments as Appendix I. Region 8 disagreed with recommendation 3-4 on clearly identifying all core performance measures in recipient work plans because the Region believed states would refuse to agree. While the Region stated that all core performance measures were in the work plans, we were unable to determine whether all appropriate measures were included and we believe the general public would have similar difficulty. We strongly believe that clearly identified core performance measures are essential to providing full accountability. Region 8 disagreed that all work plan commitments could be time-specific, measurable, quantifiable, and verifiable as discussed in recommendation 3-5. However, at the exit conference, senior Region 8 management agreed that all annual commitments could and should meet these four attributes and we modified our recommendation accordingly. Region 8 disagreed with recommendation 3-6. The Region stated that it would move more aggressively toward developing and using outcome measures once headquarters identifies better outcome measures. We believe that the Region can and should begin helping regional and state staff to develop more and better outcome measures and not wait for additional headquarters guidance. Sierra Club v. EPA 18cv3472 NDCA 38 Report No. 1999-000209-R8-100302 Tier 10 ED 002061 00178457-00048 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results CHAPTER 4 IMPROVED PPG END-OF-YEAR REPORTS NEEDED TO HELP ENSURE WORK PLAN COMMITMENTS WERE MET While Region 8 monitored and evaluated recipients' environmental programs, the Region's oversight did not ensure that all recipients accomplished all work plan commitments and used PPG funds efficiently and effectively. Some regional program staff evaluated recipients' programs by assessing compliance with delegation, enforcement, and other agreements, but spent little time analyzing PPG end-of-year reports. Some regional program staff placed little importance on end-ofyear reports because these reports were based on work plans that staff stated contained inadequate commitments. All five of the state and one of the two tribal PPG end-ofyear reports we reviewed did not address all commitments nor did they include descriptive responses to all commitments. As a result, some regional staff were not using end-of-year reports to help ensure work plan commitments were met, and could not determine or demonstrate whether recipients were improving environmental results and human health. Regional staff had mixed reactions to the Region's new approach to the mid year review process and joint end-of-year reviews. Some regional program staff stated that they were unsure if recipients spent PPG funds efficiently and effectively. In addition, Region 8 did not correctly calculate the match requirement for two of the five PPGs we reviewed, resulting in under-match of approximately $150,000. Sierra Club v. EPA 18cv3472 NDCA 39 Report No. 1999-000209-R8-100302 Tier 10 ED 002061 00178457-00049 REGULATIONS AND PPG GUIDANCE DEFINE EPA GRANT OVERSIGHT Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results PPG recipients are required to prepare end-of-year reports. PPG project officers are required to review the recipient's end-of-year of report, provide evaluation findings to the recipient, and include those findings in the official PPG file. As part of the Region's and recipients' negotiation of work plan commitments, PPG guidance states that the Region and recipients should establish procedures that EPA and the recipient will use for evaluating accomplishments, discussing progress, and making adjustments to meet milestones. The 40 Code of Federal Regulations 31.40 states that endof-year reports will include a comparison of actual accomplishments to the objectives established for the grant period and reasons for not meeting objectives. The 1996 and 1998 PPG guidance state that in addition to evaluating performance based on work plan commitments, the recipient should identify any benefits that enabled the recipient to perform better than expected. The 1998 PPG guidance states that the recipient should also address EPA's performance in helping the recipient achieve work plan commitments. Further, the 1996 and 1998 guidance state that the recipient should provide information on whether the work undertaken under the grant: (1) addressed the stated strategic priorities and goals, (2) achieved administrative cost savings, (3) improved EPA-recipient working relationships, and (4) improved environmental results. PPG END-OF-YEAR REPORTS NOT USED BY ALL REGIONAL STAFF Some regional program staff evaluated recipients' programs by assessing compliance with delegation, enforcement, and other agreements, but spent little time analyzing PPG end-of-year reports. Some regional program staff stated that they placed little importance on end-of-year reports because these reports were based on work plans that contained inadequate commitments. 40 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00050 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results Some regional and state program staff did not value or use PPG end-of-year reports. A regional program manager and staffmember stated that end-of-year reports for their program addressed only core performance measures and did not provide information on basic core program requirements. They explained that end-of-year reports for their program reflected work plan commitments which excluded any measures other than core performance measures. Regional program staff said that they concentrated on day-to-day activities while only briefly reviewing end-of-year reports. One regional enforcement staffmember stated that enforcement staff only superficially evaluated end-of-year reports because staff viewed the PPA as simply a "checklist with no deliverables" and thus provided little benefit or value. She stated that enforcement staff put more effort into determining state compliance with delegation and enforcement agreements. A state program director said that while end-of-year reports provided good documentation, his staff did not use them. Further, he stated that his staff perceived regional staff as using mid year meetings, phone discussions, and site visits rather than end-of-year reports for evaluating state accomplishments. In contrast, one regional program manager stated that she used end-of-year reports because it was the only thing her program could use to hold states accountable. Another regional program staffmember said that PPG end-of-year reports were useful for reporting on core performance measures. SOME PPG END-OFYEAR REPORTS WERE INCOMPLETE All five of the state and one of the two tribal PPG end-ofyear reports we reviewed were incomplete. The end-ofyear reports did not address all work plan commitments and did not include descriptive responses for some commitments. PPG guidance states that recipients' end-ofyear reports should evaluate performance based on work plan commitments. 41 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00051 Not All Work Plan Commitments Were Addressed Some Reports Did Not Include Descriptive Responses Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results All five of the state and one of the two tribal end-of-year reports we reviewed did not address all work plan commitments. For example, Colorado's air division had a commitment to make seven presentations to trade groups. In the State's end-of-year report, the air division did not address whether it met this commitment. Utah's end-ofyear report addressed drinking water program goals, but did not report on the commitments under these goals. For example, the drinking water program discussed accomplishments under its goal "Evaluate the feasibility of forming a Regional Water Quality Alliance for small treatment plants..." but did not report on the specific commitments. Crow Creek Sioux Tribe's end-of-year report did not address any of the Tribe's work plan commitments related to its General Assistance Program. For example, the Tribe did not report on whether it developed a training plan and developed position descriptions. The regional tribal PPG project officer stated that she used site visits to supplement tribes' end-of-year reports. All five state and one of the two tribal PPG end-of-year reports we reviewed did not include descriptive responses to all work plan commitments. Montana's end-of-year report included very detailed narrative responses for most, but not all, work plan commitments. The lack of descriptive responses made it difficult for a reviewer to determine if the recipient met its commitments. Four out of the five state end-of-year reports included vague comments such as "ongoing," "done," or "as required." For example, South Dakota's end-of-year report mostly included responses of "yes," "no," or "ongoing." In addition, North Dakota's air program responded with "done" on 91 out of 140 work plan commitments. Most of these commitments had "ongoing" for their milestones. Regional program staff stated these types of responses were insufficient and resulted in regional program staff contacting recipients for further information. One regional program director stated end-of-year reports were not 42 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00052 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results sufficient to evaluate program performance because the reports were extremely vague. While regional program staff recognized the importance of reduced reporting burden, they stated that recipients' responses needed to provide sufficient detail to evaluate recipients' performance. Region 8 and recipients needed to negotiate and clearly agree on the amount of detail that recipients should include in end-of-year reports. Colorado and Region 8 have already taken strides to improve the State's end-of-year report through a pilot effort that focuses on a joint EPAState assessment that includes determining how the State will report on work plan commitments. MIXED REACTIONS ON MID-YEAR PROCESS AND JO IN T END-OFYEAR REVIEWS Regional and state staff had mixed reactions to the Region's new approach to the mid-year review process and joint end-of-year reviews. The Region changed the mid year review process in 1997 from a meeting to discuss program status and results to a strategic planning meeting. This strategic planning focused on joint prioritization and planning as a prelude to PPA negotiations for the following fiscal year. Some regional and state program staff preferred the old process because it provided an opportunity to formally review and discuss states' performance. Other regional and state staff preferred the new process because it was more important to focus on the future and to improve the Region 8-state partnership. The State Assistance Program proposed replacing separate Region 8 and state end-of-year reviews with joint reviews for several reasons. The program would like to improve the timeliness and usefulness of end-of-year reviews, assist in developing a planning document for the following year, and generally improve the partnership. The State Assistance Program director stated that joint reviews would avoid the lengthy process of Region 8 and state staff 43 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00053 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results reviewing and commenting on separate review reports. Although some regional program staff were uncertain about the feasibility of joint end-of-year reviews, Region 8 planned to pilot them with Colorado for fiscal 1999. For fiscal 1998, states and Region 8 still conducted separate end-ofyear reviews. Some regional program staff have stated there was neither additional time nor resources to jointly discuss and write end-of-year reports. Region 8 and Colorado have agreed to conduct a joint review of fiscal 1999 PPG program performance in the fall of 1999. REGION UNSURE IF RECIPIENTS SPENT PPG FUNDS EFFICIENTLY AND EFFECTIVELY Some regional program staff stated that they were unsure if recipients spent PPG funds efficiently and effectively. They stated that once the recipient's program dollars were included in a PPG, the program had no way of gauging how a state used the dollars because work plans did not specifically indicate how the recipient was going to spend program dollars. Some regional program staff stated that they did not receive copies of PPGs. One regional program staffmember stated that it would be beneficial to review the PPG for how much was allocated for each program and compare it to states' work plan commitments. PPG guidance states that the PPG budget must accurately reflect the grant agreement and be able to be tracked to support the program outcomes and outputs cited in the grant agreement. One regional program director said that a state's work plan commitments were not equivalent with the amount of funding that the state received. Region 8 could test a sample of PPG expenditures to help ensure recipients spent PPG funds efficiently and effectively as part of its management assistance program reviews. Region 8's Grants, Audit, and Procurement Program periodically conducted management assistance program reviews at recipient agencies. These reviews provided technical assistance on sound management 44 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00054 EFFECTIVE COORDINATION NEEDED TO ENSURE CORRECT MATCH CALCULATION CONCLUSION Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results practices and assessed recipients' compliance with EPA's administrative grant requirements, but did not evaluate whether recipients spent PPG funds efficiently and effectively. Region 8's Montana suboffice staff incorrectly calculated the PPG match requirements for Montana's 1998 and Fort Peck Tribes' 1997 and 1998 PPGs. Region 8 and Montana suboffice PPG program staff did not effectively coordinate with each other regarding the PPG process. Montana suboffice staff stated that they received incorrect grant information from a senior regional staffmember. In addition, Region 8's State Assistance Program did not include Montana suboffice PPGs in their database used to review match calculations. As a result, the state of Montana's and the Fort Peck Tribes' PPGs were under matched by about $150,000. Montana suboffice staff did not consistently participate in Region 8's PPG project officer meetings, retreats, and grants specialist meetings. Some regional program staff evaluated recipients' programs by assessing compliance with delegation, enforcement, and other agreements, but spent little time analyzing PPG end-of-year reports. Some regional program staff placed little importance on end-of-year reports because these reports were based on work plans that staff stated contained inadequate commitments. All five state and one of the two tribal end-of-year reports we reviewed did not address or include descriptive responses to all work plan commitments. As a result, some regional staff were not using end-of-year reports to help ensure work plan commitments were met, and could not determine or demonstrate whether recipients were improving environmental results and human health. Some regional 45 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00055 R EC O M M EN D A TIO N S Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results program staff had mixed reactions to the Region's new approach to the mid-year review process because it focused on strategic planning rather than reviewing recipients' program performance. While we agree that strategic planning is important, the Region should ensure that endof-year reports or other means provide an opportunity to formally review and discuss recipients' performance. Some regional program staff stated that they were unsure if recipients spent PPG funds efficiently and effectively. In addition, Region 8 did not correctly calculate the match requirement for two of the five PPGs we reviewed, resulting in under-match of about $150,000. While work plans with inadequate commitments were one of the reasons some regional program staff did not use endof-year reports for oversight, we included recommendations in Chapter 3 designed to correct this problem. We recommend the Regional Administrator: 4-1. Require recipients to submit end-of-year reports that specifically address each and every work plan commitment. 4-2. Work with senior regional managers, program staff, and PPG project officers to come to agreement or consensus on the minimum amount of detail that all recipients should include in end-of-year reports. Use this agreement or consensus as the basis for negotiating recipient end-of-year reporting requirements. Similar to recommendation 3-2, consider using an impartial facilitator for this process. 4-3. Encourage recipients to provide and assist them in providing information in end-of-year reports on 46 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00056 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results whether the PPG helped to improve environmental results. 4-4. Provide examples of how the PPG program (i.e., through the use of a good work plan and end-ofyear report) can be used to better evaluate program performance and obtain environmental results. These examples should demonstrate that PPGs are effective tools to obtain improved environmental results and are more than just mechanisms to provide grant funds. 4-5. Revise the PPG end-of-year report format to make the report more valuable and useful to regional and state program staff. 4-6. Ensure mid-year reviews provide an opportunity to formally review recipients' past performance in addition to strategically plan for the future. 4-7. Evaluate the results of the planned Region 8Colorado fiscal 1999 end-of-year review and expand to other states if it is successful. 4-8. Ensure regional program staff receive copies of or have access to all PPG and PPA documents that include their programs. 4-9. Ensure recipients and regional program staff negotiate work plan commitments that are commensurate to the amount of PPG funds awarded to the recipient. 4-10. Consider expanding Region 8's management assistance program reviews to include testing a sample of PPG expenditures and determining whether recipients spent PPG funds efficiently and effectively and only for eligible costs. 47 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00057 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results 4-11. Direct Montana suboffice to submit PPG grant information to the State Assistance Program to ensure match calculations are correct. 4-12. Require Montana suboffice PPG staff to consistently participate in Region 8 PPG project officer meetings and retreats, and grants specialist meetings. 4-13. Take appropriate action to address Montana's and Fort Peck Tribes' fiscal 1998 PPG under-match. AGENCY COMMENTS AND OIG EVALUATION Region 8 concurred with all but one of the recommendations in Chapter 4 and offered some comments. We agreed with most of the Region's comments and made some minor modifications to our original recommendations. Region 8 disagreed with recommendation 4-9 because the Region believed that information about the amount of funds available for each program area was unnecessary to negotiate work plan commitments. We disagreed because we believe that it is impossible to even begin such negotiations with recipients without such information. Without such information, the Region could not know how many or what type of commitments should be included in the work plan. We agree that one of the purposes of PPGs is to allow flexibility. Flowever, we believe that flexibility is not limited by ensuring that the initial negotiations of work plans include consideration of the amount of PPG funds. Sierra Club v. EPA 18cv3472 NDCA 48 Report No. 1999-000209-R8-100302 Tier 10 ED 002061 00178457-00058 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results EXHIBIT 1 SCOPE AND METHODOLOGY Our first objective was to determine if Region 8 effectively implemented its PPG program to ensure that recipients accomplished overall PPG program goals. Our review of Region 8's PPG program included: Interviewing all six of Region 8's state PPG project officers, two of seven tribal PPG project officers, and both pesticide PPG project officers regarding their roles and responsibilities related to the PPG program; Reviewing national and regional PPG guidance and policy to determine program requirements and consistency between national and regional guidance; Interviewing regional program managers and staff regarding their knowledge of PPG program goals and Region 8's implementation of those goals; Reviewing PPG project officers' performance standards to determine if they have measures related to effectively implementing Region 8's PPG program; Evaluating Region 8's PPG program coordination and communication among regional and state staff to determine if the Region's process was effective; Evaluating Region 8's organizational placement of PPG program responsibilities to determine whether the placement facilitated effective program implementation; Interviewing state managers and program staff regarding the Region's actions to effectively implement its PPG program and accomplish program goals; Reviewing information from the July 1998 Management Assistance Review of Colorado's 1997 PPG, and 1997 and 1998 work plans to determine if similar issues existed; and Reviewing Region 8's Federal Managers' Financial Integrity Act reports for fiscal 1994 through 1998 to determine whether the Region disclosed any material weaknesses related to its PPG program and the Region's actions to address those weaknesses. We interviewed staff from Utah's Department of Environmental Quality and Department of Agriculture in Salt Lake City, Utah. We also interviewed staff at South Dakota's Department of Environmental and Natural Resources and Department of Agriculture in Pierre, South Dakota. Our interviews with state staff primarily focused on how they perceived the Region had implemented the PPG program including barriers and successes in accomplishing the four program goals. We did not evaluate whether the states met their work plan commitments or effectively implemented their PPG program. 49 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00059 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results Our second objective was to determine if Region 8 negotiated PPG work plans that included adequate financial and programmatic accountability. Our analysis of recipients' work plans included: Reviewing national and regional guidance to determine the requirements for PPG work plans; Reviewing documentation on Region 8's PPA negotiation process to determine how work plan commitments were negotiated, who participated, and how disputes were addressed; Reviewing a judgmental sample of PPAs and tribal work plans for fiscal 1998 to determine whether the required work plan elements were included such as core performance measures; and Interviewing regional program staff to determine whether they used work plans and how they ensured the work plans had adequate programmatic and financial accountability. We also reviewed the work plans to determine whether: (1) commitments included outcome measures; and (2) these measures were time-specific, measurable, quantifiable, and verifiable. Our analysis of the work plan commitments for these four required attributes included some core performance measures as part o f our judgmental sample. However, we did not specifically evaluate the usefulness and effectiveness of these core performance measures in providing a national environmental picture. We reviewed five out of six 1998 PPAs: Colorado, Montana, North Dakota, South Dakota, and Utah. We reviewed the 1999 Colorado PPA and evaluated whether Colorado implemented suggestions from the July 1998 Colorado Management Assistance Review. We limited our review of the PPAs to commitments related to air, water, and hazardous waste programs. We also reviewed the Region's three fiscal 1998 tribal work plans for the Fort Peck Tribes, Crow Creek Sioux Tribe, and Yankton Sioux Tribe. We also reviewed the fiscal 1999 Fort Peck Tribes' work plan. We did not review the entire PPA for each of the five states, only the portions of the agreement that were identified as the PPG work plan. Further, we did not review any documents referenced in the PPAs such as delegation and enforcement agreements. Our third objective was to determine if Region 8's oversight ensured that recipients met PPG work plan commitments and used PPG funds efficiently and effectively. Our review of Region 8's oversight of PPG work plans and funds included: Reviewing PPG project officers' files to obtain information on the methods and actions used to monitor and evaluate PPG activities, 50 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00060 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results Interviewing three grant specialists and reviewing their files and determining whether they reviewed and properly calculated PPG match requirements, Interviewing regional staff and reviewing PPG project files to determine whether Region 8 had an agreement with recipients on the extent of PPG oversight and end-of-year reporting requirements, Reviewing regional management assistance program reviews to determine whether the Region had found any problems with recipients' controls and grant documentation that could affect their PPGs, Interviewing regional program managers and staff regarding their oversight of PPG commitments and whether they reviewed and used PPG end-of-year reports, and Evaluating a judgmental sample of PPG end-of-year reports for fiscal 1998 to determine if recipients accomplished all work plan commitments and described progress toward environmental goals. We reviewed five of six 1998 state end-ofyear reports: Colorado, Montana, North Dakota, South Dakota, and Utah. We also reviewed the Fort Peck Tribes' and Crow Creek Sioux Tribe's fiscal 1998 end-of-year reports. We did not verify whether the five states and two tribes actually performed the work they said they did in the end-of-year reports. Sierra Club v. EPA 18cv3472 NDCA 51 Report No. 1999-000209-R8-100302 Tier 10 ED 002061 00178457-00061 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results EXHIBIT 2 CONTRASTING EXAMPLES OF WORK PLAN COMMITMENTS Time-Specific Commitments Non Time-Specific Commitments "Finalize rule requiring dispersion modeling for air toxics by 9/30/97" "Provide facility specific compliance information through automated data systems" "An official written pesticide use policy is to be adopted by the division by June 30, 1998" "Resolve consecutive system issue by December 1, 1997" "Use EPA economic computer models to assist in evaluation" "Target five laboratories and meet with them so they know what resources we can offer them" "Complete QAT recommendations by January 31, 1998" "Conduct two customer service surveys" Measurable Commitments Non-Measurahle Commitments "Increase the laboratories electronic transfer of data by 5% by June 30, 1998" "Percentage of population served with approved ratings" "Business and local partnerships are developed" "Experiment with providing six LHD's and all District Engineer's convenient access to our live database via the WAN, Internet or telephone lines" "Establish 2-3 ozone episodes for 1996: Model performance is evaluated using enhanced air quality, meteorology, and emissions data sets." "Educate the public, as well as planners and county officials, who may be associated with approval of developments, on the necessity of getting approval for public supply wells from DDW prior to drilling...." Quantifiable Commitments Non-Quantifiable Commitments "Establish 2-3 ozone episodes for 1996. Model performance is evaluated using enhanced air quality, meteorology, and emissions data sets" "Number of complete source protection plans implemented by drinking water systems" "...By June 30, 1998 measure to determine if 250 surveys have been performed and data input into the computer" "Coordinate partnership efforts among the Division, local health departments, and EPA" "Educate the public and county officials...." "A port-of-entry and/or roadside diesel I'M program is implemented, if feasible" "Contact CLEHA leadership by August 31, 1997 to discuss possible schedule and advertising issues" Verifiable Commitments Non-Verifiable Commitments "PM 2.5 monitoring is conducted and PM 2.5 data are collected as finding is available..." "NSR regulations are reviewed, and revised as necessary', annually" "Monitoring capability is enhanced along Wasatch Front" "Regulated community compliance status improves" "The inventory process reflects stability, accuracy and thoroughness" "Maintain, for review by EPA/State, Compliance of stationary' Sources Through the Compliance Monitoring Strategy by November 15" * Although the above exhibit illustrates commitments with one attribute (i.e., time-specific, measurable, quantifiable, and verifiable), some ofthese commitments could contain more than one attribute. In addition, many ofthe PPG work plan commitments we reviewed would be measurable, quantifiable, and verifiable if specific targets and/or baselines were identified as part ofthe commitment. 52 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00062 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results APPENDIX I AGENCY RESPONSE UNITED 5 S ENVIRONMENTAL PROTECTION AGENCY REGION 8 999 18th STREET - SUITE 500 DENVER, CO 80202-2466 Ref: 8TMS-G September 24, 1999 MEMORANDUM SUBJECT: Region 8 Needs to Improve its Performance Partnership Grant Program To Ensure Accountability and Improved Environmental Results Draft Report No. 1999-XXXXXX-CAD8-XXXXXXX FROM: William P. Yellowtail /signed by Kerry Clough/ Regional Administrator Region 8 TO: Bennie S. Salem Divisional Inspector General Central Audit Division Attached are the Region's comments to the draft report on the Office of Inspector General's (OIG) review of the regional Performance Partnership Grant (PPG) Program. We would like to thank the OIG for giving us the opportunity to provide comments. We recognize the extensive effort the OIG staff made in reviewing this relatively new program. We would like to note that we agree with the three global recommendations listed in the Executive Summary as stated below. The Regional Administrator (RA) establish a forum where regional program staff can discuss their disagreements and concerns with senior regional managers regarding the Region's NEPPS program implementation. During the forum, regional staff should brainstorm possible solutions that address their concerns as well as support achieving NEPPS program goals. Sierra Club v. EPA 18cv3472 NDCA 53 Tier 10 Report No. 1999-000209-R8-100302 II Printed on Recycled Paper ED 002061 00178457-00063 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results The RA should develop regional guidance on what should be included in a PPA work plan based upon regional staff agreement or consensus. Agreement or consensus should be reached on: (1) whether core performance measures are sufficient by themselves, and (2) what amount of detail in work plan commitments provides adequate accountability. The RA should also require recipients to submit end-of-year reports that specifically address each work plan commitment and provide information on whether recipients are improving the environment as a result of the money spent through the PPGs. By necessity, your review of PPGs lead to an examination of PPAs, which most of our States use to meet the PPG grant requirement concerning work plans. However, it appears to the Region that the report is focused on PPAs and not on the narrower aspects of the PPGs. Most of the findings, comments and recommendations appear to apply to broader PPA issues rather than PPG implementation. We strongly urge you to clarify the distinctions between NEPPS, PPA, and PPGs. Attached are the Region's comments on the specific recommendations as requested. We have communicated and met separately with you covering a number of factual issues in the draft report. These issues have been resolved satisfactorily. If you have any questions, please contact Barbara Rodriguez, Regional Audit Coordinator, at 303/312-6360 or Wayne Anthofer, Director, Grants, Audit and Procurement Program, at 303/312-6305. Attachment cc: Jeff Hart, 80IG Kerry Clough, 8P Max Dodson, 8EPR Aundrey Wilkins, 8TMS Wayne Anthofer, 8TMS-G Kim Victor, 80IG Barbara Rodriguez, 8TMS-G Pat Hull, 8TMS Carol Rushin, 8ENF Michael Gaydosh, 8ENF John Warded, 8MO Steve Burkett, 8P-SA Chris Lehnertz, 8ENF Sierra Club v. EPA 18cv3472 NDCA 54 Report No. 1999-000209-R8-100302 Tier 10 ED 002061 00178457-00064 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results Recommendations - Chapter 2: Improvements in Region 8's PPG Program Would Help Accomplish Program Goals General Comment: The Region suggests you change this title to " . . . Region 8's NEPPS Program 2-1 Work with recipients to develop a strategy to overcome barriers to accomplishing the four PPG program goals. Regional staffs assistance could include obtaining examples from other regions and states on their successful efforts. The Region concurs (except NEPPS should be substituted for "PPG"). 2-2 Strongly encourage states to initiate a joint EPA-State strategic planning process to identify State environmental priorities and help make decisions on investments and disinvestments. The Region concurs with comments. This recommendation could be strengthened and used as a guiding principle for pre-work and as a specific outcome for each of the midyear meetings, which would give the midyear meetings more meaning. Tying program priorities to budget priorities will encourage better strategic planning. We also note that change needs to be made at the National Program Management (NPM) level first. Core priorities, enforcement goals and other initiatives are defined in national guidance with little room for the regions and states to do strategic planning, other than moving the level of effort a little from one area to another. 2-3 Highlight and communicate successful efforts to improve partnerships throughout the Region, particularly enforcement partnerships. The Region concurs. 2-4 Emphasize during senior regional and program staff meetings, that the PPG program requires both PPG project officers and regional program staff working together to achieve program success. The Region concurs (except NEPPS should be substituted for the first use of the term "PPG"). 55 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00065 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results 2-5 Establish a forum where regional program staff can discuss their disagreements and concerns with senior managers regarding the Region's PPG program implementation. While addressing regional staff concerns, regional staff could brainstorm possible solutions that support achieving PPG program goals. The Region concurs (except NEPPS should be substituted for "PPG"). 2-6 Reach consensus among EPA and recipient staff on the definition of partnership and on the behaviors that demonstrate partnership. The Region concurs with the following modification: "Attempt to work toward development o f a common understanding among EPA and recipient staff on the definition . . . 2-7 Identify rewards/disincentives for recipient compliance/noncompliance with grant conditions. The Region concurs with the following modification: "Direct staff to clearly identify compliance or noncompliance in the grant conditions." 2-8 Require regional program staff to attend and actively participate in all appropriate PPG related meetings. The Region concurs (except NEPPS should be substituted for "PPG"). 2-9 Develop and periodically adjust realistic goals and related performance measures with specific milestones for the Region's PPG program. All regional staff who are involved in the PPG process should participate in developing these goals and performance measures (not just PPG project officers). The Region concurs with comments. In light of GPRA, this will be critical to every program. Goals and performance measures for the NEPPS program should be developed at the national level with Region 8 specific goals and measures as a subset. This could be addressed in the work of the proposed NEPPS forum. (Also, NEPPS should be substituted for the first and second use of the term "PPG" in your recommendation.) The four goals for NEPPS stated in the PPG guidance have been used as the Region 8 goals, but the Region should have a system to measure its results. 56 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00066 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results 2-10 Emphasize Region 8's PPG program goals at divisional staff meetings and regional all-employees meetings. In addition, the Regional Administrator should consider identifying Region 8's PPG program goals as part of the Region's strategic plan and priorities. The Region concurs (except NEPPS should be substituted for "PPG"). 2-11 Consider including the Region's PPG program goals and performance measures as part of Region 8's employee evaluation and reward system. The Region concurs with comments. The Region now has a modified "evaluation and reward" system. The performance criteria are generally broader and less specific. However, this is an area that we will consider for review. (Also, change PPG to NEPPS in your recommendation.) 2-12 Clearly define the roles and authority of all those involved with PPG program, as recommended in the State Assistance Program's General Assessment of the Performance Partnership Process (undated) and distribute to all regional staff. The Region concurs. Although the roles of all those involved in the NEPPS program have been defined in Regional Orders R8.5700.1 and R8.5700.03, the orders need to be reviewed, revised and consolidated based on the Region's past four years' experience implementing the NEPPS program. Then these orders need to be identified and made known to those who are affected by them. 2- 13 Identify additional responsibilities PPG project officers could perform to assist regional program staff such as those suggested by regional program staff. (See page 20 of this report.) The Region concurs. Recommendations - Chapter 3: Better Work Plans Needed To Ensure Adequate Accountability 3-1 Develop regional guidance on what should be included in a PPG work plan based upon regional staff agreement, or at least consensus. The Region concurs with comments. Regional guidance would supplement whatever guidance regarding PPA work plans is specified by the Agency and/or by National Program Managers. (Also, change PPG to PPA in your recommendation.) 57 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00067 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results 3-2 Obtain an impartial facilitator to work with senior regional managers, regional program staff and P P G project officers to come to agreement or consensus on whether: (1) core performance measures are sufficient by themselves, and (2) what amount of detail in work plan commitments provides adequate accountability. 3-3 Require that regional staff adhere to the following guidelines during the development of a regional agreement of accountability provisions for the PPG work plan: y. Ultimate goal is to develop an agreement that ensures adequate accountability and flexibility, z. Everyone's perspectives are valid and will be acknowledged, aa. Flexibility should be a priority but not at the expense of accountability, and bb. Senior regional managers and staff will support the agreement or consensus reached. 3-4 Require regional and state staff to clearly identify all core performance measures, including modified measures, included in the PPG work plans. 3-6 Help regional staff move toward using outcome measures by requiring that: a. Regional program managers increase the priority of developing outcome measures with recipients to ensure that staff have the time and resources needed to adequately establish these measures, b. PPG project officers and regional program staff develop and seek out good examples of outcome measures for reference by all regional and recipient staff, c. Regional program mangers and staff work with recipients to develop baseline data needed to adequately measure outcomes, and d. Senior regional managers continue encouraging national program managers and regional program staff to move more toward outcome measures rather than just activity or output measures. The Region does not concur with Recommendations 3-2. 3-3. 3-4. and 3-6. These recommendations are all prescriptive subparts of 3-1. We agree with Recommendation 3-1 and 58 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00068 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results we further agree that the issues identified in Recommendations 3-2, 3-3, 3-4, and 3-6 are important. The Region commits to addressing these issues by performing Recommendation 3-1. Regarding 3-4, the Region would like to point out that the core performance measures are in the PPA work plans. We recognize that some are easier to identify than others. Regarding 3-6, the Region will move as aggressively as possible once Headquarters identifies the outcome performance measures and provides them to us in their guidance. As a State wants to develop them, we would welcome their interest and provide assistance to them. States currently express difficulty in meeting outcome measures that we do have. 3-5 Establish working definitions for the terms "time-specific", "measurable", "quantifiable", and "verifiable". Direct regional staff to ensure that all PPG work plan commitments are time-specific, measurable, quantifiable, and verifiable. Concur with the first part of the recommendation. "Establish working definitions for the terms "time-specific", "measurable", "quantifiable", and "verifiable"." We will consider adopting the definitions made by the OIG in the Management Assistance Review (MAR) of the Colorado PPA program. Do not concur with the second part of the recommendation. "Direct regional staff to ensure that all PPG work plan commitments are time-specific, measurable, quantifiable, and verifiable." Certain output data are required to meet statutory or regulatory requirements that are augmented, but not superceded by GPRA and its requirements for outcome-based measures. In some situations, the lack of baseline data or lack of reporting mechanisms beyond EPA control prevent truly quantifiable outcome measures. (Also, change "PPG" in your recommendation to "PPA" ) Recommendations: Chapter 4: Improved PPG End-of-Year Reports Needed to Help Ensure work Plan Commitments Were Met 4-1 Require recipients to submit end-of-year reports that specifically address each and every work plan commitment. The Region concurs. 4-2 Similar to recommendation 3-2, use an impartial facilitator to work with senior regional managers, program staff, and PPG project officers to come to agreement 59 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00069 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results or consensus on the amount of detail recipients should include in end-of-year reports. Use this agreement or consensus as the basis for negotiating recipient end-of-year reporting requirements. The Region concurs with the modification: "Determine the minimum amount of detail recipients should include in end-of-year reports. Use this agreement or consensus as the basis for negotiating recipient end-of-year reporting requirements. Consider using a facilitator to work with senior regional managers, program staff, and PPG project officers to come to agreement or consensus on the minimum amount." The Region will then include this into regional PPA guidance. The end-of-year reports should fit the situation. If a State is doing a good job, the amount of detail is not going to be the same as for a State not doing a good job. 4-3 Encourage recipients to provide and assist them in providing information in recipients' end-of-year reports on whether recipients improved environmental results under PPGs. The Region concurs. We support this recommendation and would like to note that this is one of the NEPPS goals which we have been striving for in the Region. We have a few examples of endof-year reports which describe the improved environmental results under PPAs. (Also, change "PPGs" to "NEPPS" in your recommendation.) Clarification of the meaning of this sentence is needed to confirm our interpretation. 4-4 Provide examples of how the PPG program (i.e., through the use of a good work plan and end-of-year report) has or can be used to better evaluate program performance and obtain environmental results. These examples should demonstrate that PPGs are an effective tool to obtain improved environmental results and more than just a mechanism to provide grant funds. The Region concurs. (The wording should be changed from "PPG program" to "NEPPS.) 4-5 Identify alternatives to PPG end-of-year reports if regional and state program staff continue to not use the reports. The Region concurs with comments. The Region supports this recommendation, with the following rewording: I-d m tffy-att&matives-t o R e v ise the format for PPA end-of-year reports 60 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00070 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results to m a k e th e m v a lu a b le a n d u s e fu l to if- regional and state program staff, continue to not use the-reports.- 4-6 Ensure mid-year reviews provide an opportunity to formally review recipients' past performance in addition to strategically plan for the future. The Region concurs. The Program and Enforcement offices are already doing this and are moving to joint reviews when possible during media-specific mid-years. The Senior Leadership Team (SLT) deals with priority issues during SLT/State Director mid-years. 4-7 Evaluate the results of the planned Region 8/Colorado fiscal 1999 end-of-year review and expand to other states if it is successful. The Region concurs. 4-8 Ensure regional program staff receive copies of all PPGs that include their programs. The Region concurs with the following modification: Replace "PPG" with "PPA". 4-9 Ensure recipients and regional program staff negotiate work plan commitments that reflect the allocation of PPG funds among programs. The Region does not concur. The purpose of the PPGs is to allow flexibility to change priorities if needed. If we add back in the allocation of program funds and accountability of funding levels to each program, we remove the flexibility and increase administrative overhead costs. We suggest rather that the benefits of "letting go" of this control by the NPMs and technical program staff be identified, discussed, and consensus reached that we are all one agency, environmental achievement is the Agency's goal regardless of which program gets the achievement. We should have the accountability built into the PPAs and yet maintain the flexibility as the States need. However, since funds lose their identity when they become part of a PPG, allocations cannot be used to track work plan commitments. 4-10 Consider expanding Region 8's management assistance program reviews to include testing a sample of PPG expenditures and determining whether recipients spent PPG funds efficiently and effectively and only for eligible costs. The Region concurs with comments. The Region will consider this modification and expansion of its Management Assistance Program (MAP) reviews to include a testing of PPG expenditures; a test for "eligible" costs is already included in reviews. However, it should be understood that 61 Report No. 1999-000209-R8-100302 Sierra Club v. EPA 18cv3472 NDCA Tier 10 ED 002061 00178457-00071 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results our purpose in the MAP reviews is technical assistance in terms of good grants management practices (financial recordkeeping, timely submission of reports, etc.) Your recommendation would be a major modification to our MAP reviews and would take on the appearance of a programmatic review rather than technical assistance. 4-11 Direct Montana suboffice to submit PPG g ran t information to the State Assistance program to ensure match calculations are correct. The Region concurs. 4-12 Require Montana suboffice PPG staff to consistently participate in Region 8 PPG project officer meetings and retreats, and grants specialist meetings. The Region concurs. The Montana office grants specialist already participates via teleconference the weekly grants specialists meetings. 4-13 Take appropriate action to address the State of Montana's and Fort Peck Tribes' fiscal 1998 PPG under-match. The Region concurs. These actions are underway. Sierra Club v. EPA 18cv3472 NDCA 62 Report No. 1999-000209-R8-100302 Tier 10 ED 002061 00178457-00072 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results APPENDIX II ABBREVIATIONS EPA NEPPS OIG PPA PPG RA Environmental Protection Agency National Environmental Performance Partnership System Office of Inspector General Performance Partnership Agreement Performance Partnership Grant Regional Administrator Sierra Club v. EPA 18cv3472 NDCA 63 Report No. 1999-000209-R8-100302 Tier 10 ED 002061 00178457-00073 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results APPENDIX III DISTRIBUTION Office of Inspector General Inspector General (2410) Deputy Assistant Inspector General for External Audits (2421) Deputy Assistant Inspector General for Internal Audits (2421) Headquarters Audit Liaison (2421) Divisional Inspectors General for Audit Headquarters Offices Agency Followup Official, Attn: Assistant Administrator for Administration and Resources Management (3101) Agency Followup Coordinator (2724) Associate Administrator for Congressional and Intergovernmentai Relations (1301) Deputy Associate Administrator for Office of State and Local Relations (1306) Director for Office of Regional Operations (1108) Associate Administrator for Communications, Education, and Media Relations (1701) Director for Office of Planning, Analysis, and Accountability (2721) Chief Financial Officer (2710) Headquarters Library EPA Region 8 Assistant Regional Administrator, Office of Partnerships and Regulatory Assistance Assistant Regional Administrator, Office of Enforcement, Compliance, and Environmental Justice Assistant Regional Administrator, Technical & Management Services Assistant Regional Administrator, Ecosystems Protection and Remediation Director, State Assistance Program Director, Montana Operation Office Regional Offices Regional Administrators Sierra Club v. EPA 18cv3472 NDCA 64 Report No. 1999-000209-R8-100302 Tier 10 ED 002061 00178457-00074 Region 8 Needs to Improve Its PPG Program to Ensure Accountability and Improved Environmental Results States Colorado Department of Public Health and Environment Director, Office of Environment Montana Department of Environmental Quality Director North Dakota Department of Health Chief, Environmental Health Section Utah Department of Environmental Quality Deputy Director Utah Department of Agriculture and Food Director, Division of Plant Industry South Dakota Department of Environment and Natural Resources Director of Environmental Services South Dakota Department of Agriculture Department Secretary Administrator for Office of Agronomy Services Tribes Assiniboine and Sioux Tribes Environmental Program Manager Crow Creek Sioux Tribe Environmental Director Yankton Sioux Tribe Director, Environmental Protection Program Sierra Club v. EPA 18cv3472 NDCA 65 Report No. 1999-000209-R8-100302 Tier 10 ED 002061 00178457-00075