Search results

Technology is sometimes used to support assessments of academic research in the form of automatically generated bibliometrics for reviewers to consult during their evaluations or by replacing some or all human judgements. With artificial intelligence (AI), there is increasing scope to use technology to assist research assessment processes in new ways. Since transparency and fairness are widely considered important for research assessment and AI introduces new issues, this review investigates their implications.

This article reviews and briefly summarises transparency and fairness concerns in general terms and through the issues that they raise for various types of Technology Assisted Research Assessment (TARA).

Whilst TARA can have varying levels of problems with both transparency and bias, in most contexts it is unclear whether it worsens the transparency and bias problems that are inherent in peer review.

This is the first analysis that focuses on algorithmic bias and transparency issues for technology assisted research assessment.

Scholars often aim to conduct high quality research and their success is judged primarily by peer reviewers. Research quality is difficult for either group to identify, however and misunderstandings can reduce the efficiency of the scientific enterprise. In response, we use a novel term association strategy to seek quantitative evidence of aspects of research that are associated with high or low quality.

We extracted the words and 2–5-word phrases most strongly associated with different quality scores in each of 34 Units of Assessment (UoAs) in the Research Excellence Framework (REF) 2021. We extracted the terms from 122,331 journal articles 2014–2020 with individual REF2021 quality scores.

The terms associating with high- or low-quality scores vary between fields but relate to writing styles, methods and topics. We show that the first-person writing style strongly associates with higher quality research in many areas because it is the norm for a set of large prestigious journals. We found methods and topics that associate with both high- and low-quality scores. Worryingly, terms associated with educational and qualitative research attract lower quality scores in multiple areas. REF experts may rarely give high scores to qualitative or educational research because the authors tend to be less competent, because it is harder to do world leading research with these themes, or because they do not value them.

This is the first investigation of journal article terms associating with research quality.

The purpose of this study is to explore current practices, challenges and technological needs of different data repositories.

An online survey was designed for data repository managers, and contact information from the re3data, a data repository registry, was collected to disseminate the survey.

In total, 189 responses were received, including 47% discipline specific and 34% institutional data repositories. A total of 71% of the repositories reporting their software used bespoke technical frameworks, with DSpace, EPrint and Dataverse being commonly used by institutional repositories. Of repository managers, 32% reported tracking secondary data reuse while 50% would like to. Among data reuse metrics, citation counts were considered extremely important by the majority, followed by links to the data from other websites and download counts. Despite their perceived usefulness, repository managers struggle to track dataset citations. Most repository managers support dataset and metadata quality checks via librarians, subject specialists or information professionals. A lack of engagement from users and a lack of human resources are the top two challenges, and outreach is the most common motivator mentioned by repositories across all groups. Ensuring findable, accessible, interoperable and reusable (FAIR) data (49%), providing user support for research (36%) and developing best practices (29%) are the top three priorities for repository managers. The main recommendations for future repository systems are as follows: integration and interoperability between data and systems (30%), better research data management (RDM) tools (19%), tools that allow computation without downloading datasets (16%) and automated systems (16%).

This study identifies the current challenges and needs for improving data repository functionalities and user experiences.

The peer review history for this article is available at: https://publons.com/publon/10.1108/OIR-04-2021-0204

Data sharing is widely thought to help research quality and efficiency. Data sharing mandates are increasingly being adopted by journals and the purpose of this paper is to assess whether they work.

This study examines two evolutionary biology journals, Evolution and Heredity, that have data sharing mandates and make extensive use of Dryad. It uses a quantitative analysis of presence in Dryad, downloads and citations.

Within both journals, data sharing seems to be complete, showing that the mandates work on a technical level. Low correlations (0.15-0.18) between data downloads and article citation counts for articles published in 2012 within these journals indicate a weak relationship between data sharing and research impact. An average of 40-55 data downloads per article after a few years suggests that some use is found for shared life sciences data.

The value of shared data uses is unclear.

Data sharing mandates should be encouraged as an effective strategy.

This is the first analysis of the effectiveness of data sharing mandates.

To assess whether interdisciplinary research evaluation scores vary between fields.

The authors investigate whether published refereed journal articles were scored differently by expert assessors (two per output, agreeing a score and norm referencing) from multiple subject-based Units of Assessment (UoAs) in the REF2021 UK national research assessment exercise. The primary raw data was 8,015 journal articles published 2014–2020 and evaluated by multiple UoAs, and the agreement rates were compared to the estimated agreement rates for articles multiply-evaluated within a single UoA.

The authors estimated a 53% agreement rate on a four-point quality scale between UoAs for the same article and a within-UoA agreement rate of 70%. This suggests that quality scores vary more between fields than within fields for interdisciplinary research. There were also some hierarchies between fields, in the sense of UoAs that tended to give higher scores for the same article than others.

The results apply to one country and type of research evaluation. The agreement rate percentage estimates are both based on untested assumptions about the extent of cross-checking scores for the same articles in the REF, so the inferences about the agreement rates are tenuous.

The results underline the importance of choosing relevant fields for any type of research evaluation.

This is the first evaluation of the extent to which a careful peer-review exercise generates different scores for the same articles between disciplines.

This study investigates differences and commonalities in data production, sharing and reuse across the widest range of disciplines yet and identifies types of improvements needed to promote data sharing and reuse.

The first authors of randomly selected publications from 2018 to 2019 in 20 Scopus disciplines were surveyed for their beliefs and experiences about data sharing and reuse.

From the 3,257 survey responses, data sharing and reuse are still increasing but not ubiquitous in any subject area and are more common among experienced researchers. Researchers with previous data reuse experience were more likely to share data than others. Types of data produced and systematic online data sharing varied substantially between subject areas. Although the use of institutional and journal-supported repositories for sharing data is increasing, personal websites are still frequently used. Combining multiple existing datasets to answer new research questions was the most common use. Proper documentation, openness and information on the usability of data continue to be important when searching for existing datasets. However, researchers in most disciplines struggled to find datasets to reuse. Researchers' feedback suggested 23 recommendations to promote data sharing and reuse, including improved data access and usability, formal data citations, new search features and cultural and policy-related disciplinary changes to increase awareness and acceptance.

This study is the first to explore data sharing and reuse practices across the full range of academic discipline types. It expands and updates previous data sharing surveys and suggests new areas of improvement in terms of policy, guidance and training programs.

The peer review history for this article is available at: https://publons.com/publon/10.1108/OIR-08-2021-0423.

Recently, the new application of DVD technology in multimedia reference sources provides opportunities to integrate a large amount of various media, as well as many kinds of reference sources, on a single disc. This paper provides an overview of the recent trends towards publishing DVD multimedia reference sources and discusses the advantages of DVD‐ROMs and the gigabyte storage potential of the technology for publishing multimeia reference sources.

The purpose of this paper is to investigate the potential of altmetric and webometric indicators to aid with funding agencies’ evaluations of their funding schemes.

This paper analyses a range of altmetric and webometric indicators in terms of suitability for funding scheme evaluations, compares them to traditional indicators and reports some statistics derived from a pilot study with Wellcome Trust-associated publications.

Some alternative indicators have advantages to usefully complement scientometric data by reflecting a different type of impact or through being available before citation data.

The empirical part of the results is based on a single case study and does not give statistical evidence for the added value of any of the indicators.

A few selected alternative indicators can be used by funding agencies as part of their funding scheme evaluations if they are processed in ways that enable comparisons between data sets. Their evidence value is only weak, however.

This is the first analysis of altmetrics or webometrics from a funding scheme evaluation perspective.

More knowledge about open access (OA) scholarly publishing on the web would be helpful for citation data mining and the development of web‐based citation indexes. Hence, the main purpose of this study is to identify common characteristics of open access publishing, which may therefore enable us to measure different aspects of e‐research on the web.

In the current study, five characteristics of 545 OA citing sources targeting OA research articles in four science and four social science disciplines were manually identified, including file format, hyperlinking, internet domain, language and publication year.

About 60 per cent of the OA citing sources targeting research papers were in PDF format, 30 per cent were from academic domains ending in edu and ac and 70 per cent of the citations were not hyperlinked. Moreover, 16 per cent of the OA citing sources targeting studied papers in the eight selected disciplines were in non‐English languages. Additional analyses revealed significant disciplinary differences in some studied characteristics across science and the social sciences.

The OA web citation network was dominated by PDF format files and non‐hyperlinked citations. This knowledge of characteristics shaping the OA citation network gives a better understanding about their potential uses for open access scholarly research.

The main purpose of this study is to assess the citation advantage for self‐archived Open Access (OA) agriculture research against its non‐OA counterparts.

At the article level, the paper compared the citation counts of self‐archived research with non‐OA articles based upon a sample of 400 research articles from ISI‐indexed (ISI, Institute for Scientific Information) agriculture journals in 2005. At the journal level the paper compared impact factors (IFs) of OA against non‐OA agriculture journals from 2005 to 2007 as reported by the ISI Journal Citation Reports. The paper also sought evidence of citation impact based on a random sample of 100 OA and 100 non‐OA publications from the Food and Agriculture Organization (FAO) of the United Nations in 2005. It used both ISI and Scopus databases for citation counting and also Google and Google Scholar for locating the self‐archived articles published in the non‐OA journals.

The results showed that there is an obvious citation advantage for self‐archived agriculture articles as compared to non‐OA articles. Out of a random sample of 400 articles published in non‐OA agriculture journals, about 14 per cent were OA and had a median citation count of four whereas the median for non‐OA articles was two. However, at the journal level the average IF for OA agriculture journals from 2005 to 2007 was 0.29, considerably lower than the average IF for non‐OA journals (0.65). Finally it found that FAO publications which were freely accessible online tended to attract more citations than non‐OA publications in the same year and had a mean citation count of 1.73 whereas the mean for non‐OA publications was 0.28.

Self‐archived agriculture research articles tended to attract higher citations than their non‐OA counterparts. This knowledge of the citation impact of OA agricultural research gives a better understanding about the potential effect of self‐archiving on the citation impact.