A Conceptual Framework and Recommendations for Open Data and Artifacts in Empirical Software Engineering

Open science aims to improve research accessibility, replicability, and consequently its quality. Empirical software engineering entails both data and artifacts, which may be shared more or less openly, to support transparency. However, the trade-offs involved in balancing the openness against integrity and secrecy concerns need methodological guidance. Aim. We aim to derive such advice, based on our own experiences from a research project, in the field of gaze-assisted code reviews – the Gander case. Method. We draw on literature about open data and artifacts in socio-technical research. Next, we describe our case project and derive a conceptual framework of steps in research data analysis and artifact development, using our data and artifacts as illustrating examples. Results. The conceptual framework contains 1) a categorization of humans involved as participants and their concerns, 2) four steps for data analysis, each resulting in corresponding data and meta-data, and 3) three steps of artifact distribution, matching different levels of openness. We derive a preliminary set of recommendations for open science practices for data and artifacts. Conclusion. The conceptual framework has proven useful in summarizing and discussing data and artifacts in the studied case project. We envision that the framework and recommendations will provide a foundation for further advancement of open science research practices in empirical, socio-technical software engineering.