Zheng Wang, Shuo Xu, Yibo Wang, Xiaojiao Chai and Liang Chen
The purpose of this study is to solve the problems caused by the growing volumes of pre-annotated literature and variety-oriented annotations, including teamwork, quality control…
Abstract
Purpose
The purpose of this study is to solve the problems caused by the growing volumes of pre-annotated literature and variety-oriented annotations, including teamwork, quality control and time effort.
Design/methodology/approach
An annotation collaboration workbench is developed, which is named as Bureau for Rapid Annotation Tool (Brat). Main functionalities include an enhanced semantic constraint system, Vim-like shortcut keys, an annotation filter and a graph-visualizing annotation browser. With these functionalities, the annotators are encouraged to question their initial mindset, inspect conflicts and gain agreement from their peers.
Findings
The collaborative patterns can indeed be leveraged to structure properly every annotator’s behaviors. The Brat workbench can actually be seen as an experienced-based annotation tool by harnessing collective intelligence. Compared to previous counterparts, about one-third of time can be saved on Xinhuanet military news and patent corpora with the workbench.
Originality/value
The various annotations are very popular in real-world annotation tasks with multiple annotators. Though, it is still under-discussed on variety-oriented annotations. The findings of this study provide the practitioners valuable insight into how to govern annotation projects. In addition, the Brat workbench takes the first step for future research on annotating large-scale text resources.