Infrastructures as an analytical framework for understanding research evaluation systems

When trying to map the bibliometrics based ‘research evaluation landscape’ in terms of for instance methods, technical systems and stakeholders; the complexities soon become obvious. Different methods are used at different national and local levels; there is a great variation of systems used for gathering, organizing, and analyzing data; and a multitude of stakeholders, many of which taking on a variety of roles within the context of evaluation practices. To be able to study and understand the evaluation landscape, an infrastructure perspective is suggested as an analytical framework, as proposed by Star and Bowker (2006). According to Star and Bowker, an infrastructure should be understood, not as a concrete technology, but as a system of contextual factors, representing “one of a number of possible distributions of tasks and properties between hardware, software and people" (Star & Bowker, 2006, p. 232). In the context of research evaluation studies, “hardware” could be operationalized as technical systems, “software” as the evaluation practices per se, and “people” as the stakeholders involved. The operationalization of the concepts will be further illustrated by examples from previous research on e.g. stakeholders involved in the research evaluation process, systems used in bibliometric evaluations, and indicators utilized for measuring academic performance; as well as the use of one empirical example, outlining how one stakeholder can be studied through its different roles in research evaluation – as well as its interactions with other stakeholder, and its use, and production, of both ‘hardware’ and ‘software’.