Formal Expert Judgement: An Overview

Introduction:
This paper, prepared at the Joint Research Centre of the European Commission within the frame of the institutional action SAFELIFE - Safety of Ageing Components in Nuclear Power Plants. Acknowledging that formal expert judgement has become a relatively well-established tool in connection to risk assessments, the paper aims to apply formal expert judgement to assess the structural integrity of nuclear power plant components. In the past, the Delphi method was one of two key methods that were used to convey structured expert opinion. The authors offer a description of the method, including the process and it's limitations.

The Delphi Method: Description
Developed in the early 1950s, it was mainly applied to technology forecasting, but also to many types of policy analyses. The method is based on a structured process for collecting and distilling
knowledge from a group of experts by means of a series of questionnaires combined with controlled opinion feed back.

Process:

• A questionnaire is sent to experts.
• Each expert gives his answers to the questions in an independent and anonymous way.
• The responses of each expert are analysed by the monitoring team. Thelower 25% and the upper 25 % of responses are excluded.
• The set of responses is then sent back to experts and they are asked ifthey wish to revise the initial predictions.
• The process is reiterated until a degree of consensus is reached by experts.

Weaknesses:

• A low level reliability of judgements among experts and therefore
  dependency of forecasts on the particular judges selected
• Sensitivity of the results to ambiguity in the questionnaire that is used for
  data collection in each round
• Difficulty in assessing the degree of expertise incorporated in the forecast
• Responses can be altered by monitors in the attempt of moving the
  following round of responses in the desired direction.

Despite these weaknesses, the authors note, "it must be acknowledged that there have been many poorly conducted Delphi applications and there is in fact an important conceptual distinction between evaluating a technique and evaluating an application of a technique." Therefore, they conclude that in general the Delphi method is useful in answering one
specific and single-dimension question. They argue; however, that it may not be effective for
determining complex forecasts concerning multiple factors since the collation of expert judgements suffers from the possibility that interactions between forecasted items may not be fully considered.

Improving the Delphi Method:
To improve forecasting reliability in the Delphi method, it is necessary to consider
the possibility that the occurrence of one event may change the probability of occurrence of other events included in the surveys. Cross impact analysis was therefore developed as an extension of Delphi method.

Source:
K. Simola, A. Mengolini, R. Bolado-Lavin. "Formal Expert Judgement: An Overview", Institute for Energy, Joint Research Center of the European Commission (2005).