How Do People Make Relevance Judgment of Scientific Data?

Abstract

Many efforts have been made to explore user relevance judgment for documents, images, web pages and music in the field of information retrial. However, there is a lack of attention to scientific data even when scientists and researchers are facing an increasing data deluge. In this study, we carried out a two-phase (first exploratory and then empirical) research to explore relevance judgment patterns of scientific data users. In the exploratory study, we interviewed 23 subjects who participated in a national competition related to scientific data. Five relevance criteria (RC) and seven paths of their usage were identified by content analysis of the transcribed records of the interview. Based on the results of the first phase, seven hypotheses were proposed and verified by partial least squares structural equation modelling (PLS-SEM). The study identified five RC, i.e. topicality, accessibility, authority, quality and usefulness used by scientific data users. Three patterns were identified including 1) data topicality judgment as the first step or starting point, 2) data reliability judgment as the necessary process and 3) data utility judgment as final purpose. These findings provide new understanding of relevance judgement and behaviours of scientific data users, and could benefit the design for cognitive retrieval systems and algorithms specific to scientific data.