Scoping the field of disaster exercise evaluation - A literature overview and analysis

The evaluation of emergency, disaster and crisis management exercises supports both individual and organisational learning, facilitates the development of response capabilities, and helps to determine whether the current level of preparedness is good enough. Nevertheless, despite its importance in the field of disaster risk management, there is a lack of a comprehensive overview of research in the area. The aim of the paper is to provide such an overview. A scoping study identified the key contributions on the topic of disaster exercise evaluation, provides an overview of research in the area, and analyses opportunities for future work. The purpose, function and form of the evaluation provided the framework for the analysis, which was applied to the scoping study results. The results indicate a lack of academic interest. Although exercises take place on a regular basis and are often used for research purposes, their evaluations are seldom the focus of attention per se. Moreover, contributions that do focus on evaluations are spread over several disciplines. Nevertheless, the results indicate that recent contributions are becoming more coherent as they build on each other (or at least refer to each other), even if they are produced within different disciplines. Despite encouraging signs of a more cohesive scientific corpus on the evaluation of disaster exercises, there is still room for improvement. The scientific discourse would benefit from greater clarity regarding: (1) the purpose and context in which a specific evaluation method is designed to be used; (2) what the method needs to do (or produce) in order for it to fulfil the purpose; and (3) how the method achieves its goal and thereby fulfils its purpose. Moreover, in order to help researchers to build on each other's work and suggest improvements to evaluation methods, it is urgent that the supporting evidence (for example, empirical data or logical reasoning) for claims regarding the usefulness of a specific method is clearly presented. This is likely to lead to a more vigorous scientific discourse, which will result in increasingly relevant and robust arguments related to how to approach the problem of evaluating disaster exercises in practice.