26.0 Expert Judgment (40 CFR § 194.26)

26.1 Requirements

§ 194.26 Expert Judgment

(a) Expert judgment, by an individual expert or panel of experts, may be used to support any compliance application, provided that expert judgment does not substitute for information that could reasonably be obtained through data collection or experimentation.

(b) Any compliance application shall:

(1) Identify any expert judgments used to support the application and shall identify experts (by name and employer) involved in any expert judgment elicitation processes used to support the application.

(2) Describe the process of eliciting expert judgment, and document the results of expert judgment elicitation processes and the reasoning behind those results. Documentation of interviews used to elicit judgments from experts, the questions or issues presented for elicitation of expert judgment, background information provided to experts, and deliberations and formal interactions among experts shall be provided. The opinions of all experts involved in each elicitation process shall be provided whether the opinions are used to support compliance applications or not.

(3) Provide documentation that the following restrictions and guidelines have been applied to any selection of individuals used to elicit expert judgments:

(i) Individuals who are members of the team of investigators requesting the judgment or the team of investigators who will use the judgment were not selected; and

(ii) Individuals who maintain, at any organizational level, a supervisory role or who are supervised by those who will utilize the judgment were not selected.

(4) Provide information which demonstrates that:

(i) The expertise of any individual involved in expert judgment elicitation comports with the level of knowledge required by the questions or issues presented to that individual; and

(ii) The expertise of any expert panel, as a whole, involved in expert judgment elicitation comports with the level and variety of knowledge required by the questions or issues presented to that panel.

(5) Explain the relationship among the information and issues presented to experts prior to the elicitation process, the elicited judgment of any expert panel or individual, and the purpose for which the expert judgment is being used in compliance applications(s) [sic].

(6) Provide documentation that the initial purpose for which expert judgment was intended, as presented to the expert panel, is consistent with the purpose for which this judgment was used in compliance application(s).

(7) Provide documentation that the following restrictions and guidelines have been applied in eliciting expert judgment:

(i) At least five individuals shall be used in any expert elicitation process, unless there is a lack or unavailability of experts and a documented rationale is provided that explains why fewer than five individuals were selected.

(ii) At least two-thirds of the experts involved in an elicitation shall consist of individuals who are not employed directly by the Department or by the Department's contractors, unless the Department can demonstrate and document that there is a lack or unavailability of qualified independent experts. If so demonstrated, at least one third of the experts involved in an elicitation shall consist of individuals who are not employed directly by the Department or by the Department's contractors.

(c) The public shall be afforded a reasonable opportunity to present its scientific and technical views to expert panels as input to any expert elicitation process.

26.2 Background

According to 40 CFR § 194.26 (U.S. EPA 1996), the expert judgment by an individual expert or panel of experts may be used to support any compliance application, provided that expert judgment does not substitute for information that could reasonably be obtained through data collection or experimentation.

The U.S. Environmental Protection Agency's (EPA's) Certification Decision (U.S. EPA 1998a) provides the following explanation of the use of the expert judgment process in demonstrating compliance with 40 CFR Part 194 (U.S. EPA 1996):

The requirements of 40 CFR § 194.26 apply to expert judgment elicitation. Expert judgment is typically used to elicit two types of information: numerical values for parameters (variables) that are measurable only by experiments that cannot be conducted due to limitations of time, money, and physical situation; and essentially unknowable information, such as which features should be incorporated into passive institutional controls to deter human intrusion into the repository (61 FR 5228). Quality assurance (QA) requirements (specifically 40 CFR § 194.22(a)(2)(v)) must be applied to any expert judgment to verify that the procedures for conducting and documenting the expert elicitation have been followed.

The requirements of 40 CFR Part 194 prohibit expert judgment from being used in place of experimental data, unless the Department of Energy (DOE) can justify that the necessary experiments cannot be conducted. Expert judgment may substitute for experimental data only in those instances in which limitations of time, resources, or physical setting preclude the successful or timely collection of data.

26.3 1998 Certification Decision

26.3.1 Expert Judgment for Performance Assessment Parameters

The Compliance Certification Application (CCA) (U.S. DOE 1996) does not identify any formal expert judgment activities related to the performance assessment (PA) parameters. During the EPA's review of the PA parameters, the EPA found inadequate explanation and information for 149 parameters that the U.S. Department of Energy (DOE) claimed had been derived using professional judgment. The compliance criteria do not provide for utilization of "professional judgment." Input parameters are to be derived from data collection, experimentation, or expert elicitation. The EPA requested that the DOE provide additional information on the derivation of the 149 parameters (Trovato 1997a;Trovato 1997b;Trovato 1997c).

The DOE responded to the EPA's request by adding information to and improving the quality of the records to enhance the traceability of parameter values. The EPA deemed the documentation provided by the DOE adequate to demonstrate proper derivation of all but one of the professional judgment parameters-the waste particle size distribution parameter. The EPA required the DOE to use the process of expert elicitation to develop the value for the waste particle size distribution parameter (Trovato 1997c).

The DOE conducted the expert judgment elicitation May 5-9, 1997. The results of the expert elicitation consisted of a model for predicting waste particle size distribution as a function of the processes occurring within the repository, as predicted by the PA. The DOE completed a final report entitled, Expert Elicitation on WIPP Waste Particle Size Distributions(s) During the 10,000-Year Regulatory Post-Closure Period (Carlsbad Area Office Technical Assistance Contractor (CTAC 1997). The particle size distribution derived from the expert elicitation was considered in the PA Verification Test parameterization.

The EPA's review of the DOE's compliance with the requirements of section 194.26 principally focused on the conduct of the elicitation process, since section 194.26 sets specific criteria for the performance of an expert judgment elicitation. The EPA observed the DOE's elicitation process and conducted an audit of the documentation prepared in support of the DOE's compliance with section 194.26. The scope of the audit covered all aspects of the expert judgment elicitation process, including panel meetings, management and team procedures, curricula vitae of panel members, background documents, and presentation materials. The EPA also assessed compliance with the quality assurance requirements of 40 CFR § 194.22(a)(2)(v) (U.S. EPA 1996). The EPA found that the documentation was appropriate, that the panel members were appropriately qualified, and that the results of the elicitation were used consistently with the stated purpose; the EPA, therefore, found the DOE in compliance with section 194.26 (U.S. EPA 1998a).

Comments on the EPA's proposed decision for section 194.26 related to questions concerning two main issues: (1) DOE's statement that it did not conduct any expert judgment activities in developing the CCA, and (2) the use or role of professional judgment in the development of input parameters used in the CCA. In response, the EPA stated that the DOE's understanding of expert judgment was consistent with the EPA's use of the term "expert judgment" in the compliance criteria, namely a formal, highly structured elicitation of expert opinion. The EPA further stated that while the CCA initially did not contain adequate information to ascertain whether a large number of the input parameters had been properly derived, the DOE subsequently provided additional information that enabled the EPA to confirm that all but one of the parameters (i.e., particle size) was adequately supported (U.S. EPA 1998b).

Based on its review of documentation developed by the DOE and its contractors, the results of the EPA's audit, and consideration of public comments, the EPA concluded that the DOE complied with the requirements of section 194.26 in conducting the required expert elicitation. For further information on the EPA's evaluation of compliance with section 194.26 in the CCA, see Compliance Application Review Document (CARD) 26 (U.S. EPA 1998c).

26.3.2 Expert Judgment for Passive Institutional Control Credit

In the CCA, Appendix EPIC, the DOE proposed a 700-year credit for the passive institutional controls (PICs) to prevent human intrusion at the Waste Isolation Pilot Plant (WIPP) and argued that the PA for the WIPP need not consider human intrusion for the first 700 years due to the postulated effective active and passive institutional controls. Such credit is allowed by 40 CFR § 194.43(c) (U.S. EPA 1996).

In its discussion on the 1998 decision on the CCA, CARD 43, (U.S. EPA 1998d), the EPA did not allow the requested credit, based in part on the argument that the DOE did not conduct an expert judgment process in the manner prescribed by section 194.26 (Expert Judgment) to derive the PICs credit. EPA stated that instead of a formal expert judgment, the DOE prepared a credit proposal and submitted it to a peer review panel.

The EPA did not consider the peer review to be equivalent to an expert judgment elicitation, as prescribed in section 194.26. For instance, the EPA stated, the PIC peer review panel was composed of three members, whereas EPA's expert judgment requirements call for at least five members on a panel (40 CFR § 194.26(b)(7)(i)).

The EPA provided the following detailed discussion in CARD 43 for its decision:

DOE undertook two expert judgment exercises related to PICs prior to the promulgation of the final compliance criteria. In one exercise, DOE asked groups of experts to predict the likelihood of various intrusion scenarios in the future. In another, DOE asked an expert panel to identify the elements of a marker system and to estimate the probability that such system would deter inadvertent intrusion. In neither case did DOE present the panel with the conceptual design for PICs that is in the CCA and ask the panel to derive a credit proposal based on that design. EPA therefore noted that the results of either exercise may not be viewed as directly relevant to DOE's credit proposal, and DOE has not requested that EPA consider them in this way.

26.4 Changes in the CRA-2004

No formal expert judgment elicitations were performed between the original certification decision (U.S. EPA 1998a) and the 2004 Compliance Recertification Application (CRA-2004) (U.S. DOE 2004).

26.5 EPA's Evaluation of Compliance for the 2004 Recertification

Because no activity relating to formal expert judgment had taken place after the original certification decision (U.S. EPA 1998a) and before submission of the CRA-2004, the EPA did not identify any issues relating to section 194.26 in the evaluation of compliance for the 2004 recertification. During its review of the CRA-2004, the EPA received no public comments on the DOE'scontinued compliance with the expert judgment requirements of section 194.26.

Based on its review of the material pertaining to the CRA-2004, the EPA concluded that the DOE demonstrated continued compliance with the requirements of section 194.26 (U.S. EPA 2006).

26.6 Changes or New Information Between the CRA-2004 and the CRA-2009 (Previously: Changes or New Information Since the 2004 Recertification)

No formal expert judgment elicitations were performed for the WIPP project between the CRA-2004 and the CRA-2009 (U.S. DOE 2009).

26.7 EPA's Evaluation of Compliance for the 2009 Recertification

Because no activity relating to formal expert judgment had taken place between the CRA-2004 and the CRA-2009, the EPA did not identify any issues relating to section 194.26 in the evaluation of compliance for the 2009 recertification. During its review of the CRA-2009, the EPA received no public comments on the DOE's continued compliance with the expert judgment requirements of section 194.26.

Based on its review of the material pertaining to the CRA-2009, the EPA concluded that the DOE demonstrated continued compliance with the requirements of section 194.26 (U.S. EPA 2010a;U.S. EPA 2010b).

26.8 Changes or New Information Since the CRA-2009

No formal expert judgment elicitations have been performed for the WIPP project since the CRA-2009 and EPA's second recertification decision (U.S. EPA 2010b). Information pertaining to expert judgment as provided for the CCA and the CRA-2004 remains unchanged. Therefore, the DOE believes it has demonstrated continued compliance with the provisions of section 194.26.

26.9 References

(*Indicates a reference that has not been previously submitted.)

Carlsbad Area Office Technical Assistance Contractor. 1997. Expert Elicitation on WIPP Waste Particle Size Distribution(s) During the 10,000-Year Regulatory Post-Closure Period (Final Report, June 3). ERMS 541365. Carlsbad, NM: Carlsbad Field Office. [PDF / Author]

Trovato, E.R. 1997a. Letter to A. Alm (6 Enclosures). 19 March 1997. ERMS 245835. U.S. Environmental Protection Agency, Washington, DC: Office of Air and Radiation. [PDF / Author]

Trovato, E.R. 1997b. Letter to G. Dials (2 Enclosures). 17 April 1997. ERMS 247196. U.S. Environmental Protection Agency, Washington, DC: Office of Air and Radiation. [PDF / Author]

Trovato, E.R. 1997c. Letter to G. Dials (2 Enclosures). 25 April 1997. ERMS 247206. U.S. Environmental Protection Agency, Washington, DC: Office of Air and Radiation. [PDF / Author]

U.S. Department of Energy (DOE). 1996. Title 40 CFR Part 191 Compliance Certification Application for the Waste Isolation Pilot Plant (October). 21 vols. DOE/CAO 1996-2184. Carlsbad, NM: Carlsbad Area Office. [Author]

U.S. Department of Energy (DOE). 2004. Title 40 CFR Part 191 Compliance Recertification Application for the Waste Isolation Pilot Plant (March). 10 vols. DOE/WIPP 2004-3231. Carlsbad, NM: Carlsbad Field Office. [Author]

U.S. Department of Energy (DOE). 2009. Title 40 CFR Part 191 Compliance Recertification Application for the Waste Isolation Pilot Plant (March). DOE/WIPP 2009-3424. Carlsbad, NM: Carlsbad Field Office.* [Author]

U.S. Environmental Protection Agency (EPA). 1996. "40 CFR Part 194: Criteria for the Certification and Recertification of the Waste Isolation Pilot Plant's Compliance with the 40 CFR 191 Disposal Regulations; Final Rule." Federal Register, vol. 61 (February 9, 1996): 5223-45. [PDF /Author]

U.S. Environmental Protection Agency (EPA). 1998a. "40 CFR Part 194: Criteria for the Certification and Recertification of the Waste Isolation Pilot Plant's Compliance with the Disposal Regulations: Certification Decision; Final Rule." Federal Register, vol. 63 (May 18, 1998): 27353-406. [PDF /Author]

U.S. Environmental Protection Agency (EPA). 1998b. Response to Comments: Criteria for the Certification and Recertification of the Waste Isolation Pilot Plant's Compliance with the 40 CFR Part 191 Disposal Regulations (May). Washington, DC: Office of Radiation and Indoor Air. [PDF / Author]

U.S. Environmental Protection Agency (EPA). 1998c. "CARD No. 26: Expert Judgment." Compliance Application Review Documents for the Criteria for the Certification and Recertification of the Waste Isolation Pilot Plant's Compliance with the 40 CFR Part 191 Disposal Regulations: Final Certification Decision (May) (pp. 26-1 through 26-9). Washington, DC: Office of Radiation and Indoor Air. [PDF / Author]

U.S. Environmental Protection Agency (EPA). 1998d. "CARD No. 43: Passive Institutional Controls." Compliance Application Review Documents for the Criteria for the Certification and Recertification of the Waste Isolation Pilot Plant's Compliance with the 40 CFR Part 191 Disposal Regulations: Final Certification Decision (May) (pp. 43-1 through 43-47). Washington, DC: Office of Radiation and Indoor Air. [PDF / Author]

U.S. Environmental Protection Agency (EPA). 2006. "40 CFR Part 194: Criteria for the Certification and Recertification of the Waste Isolation Pilot Plant's Compliance with the Disposal Regulations: Recertification Decision; Final Notice." Federal Register, vol. 71 (April 10, 2006): 18010-21.* [PDF /Author]

U.S. Environmental Protection Agency (EPA). 2010a. "2009 Compliance Recertification Application (CRA-2009) Compliance Application Review Document (CARD) No. 26, Expert Judgment." EPA Docket FDMS Docket ID No. EPA-HQ-OAR-2009-0330. Washington, DC: Office of Radiation and Indoor Air.* [PDF / Author]

U.S. Environmental Protection Agency (EPA). 2010b. "40 CFR Part 194 Criteria for the Certification and Recertification of the Waste Isolation Pilot Plant's Compliance with the Disposal Regulations: Recertification Decision; Final Notice." Federal Register, vol. 75. (November 18, 2010): 70584-95.* [PDF / Author]

Table of Contents

Acronyms and Abbreviations