How to collect high quality segmentations: use human or computer drawn object boundaries?
Date
2013-12-23
DOI
Authors
Gurari, Danna
Wu, Zheng
Isenberg, Brett
Zhang, Chentian
Purwada, Alberto
Wong, Joyce
Betke, Margrit
Version
OA Version
Citation
Gurari, Danna; Wu, Zheng; Isenberg, Brett; Zhang, Chentian; Purwada, Alberto; Wong, Joyce; Betke, Margrit. "How to Collect High Quality Segmentations: Use Human or Computer Drawn Object Boundaries?", Technical Report BUCS-TR-2013-020, Computer Science Department, Boston University, December 23, 2013. [Available from: http://hdl.handle.net/2144/11428]
Abstract
High quality segmentations must be captured consistently for applications such as biomedical image analysis. While human drawn segmentations are often collected because they provide a consistent level of quality, computer drawn segmentations can be collected efficiently and inexpensively. In this paper, we examine how to leverage available human and computer resources to consistently create high quality segmentations. We propose a quality control methodology. We demonstrate how to apply this approach using crowdsourced and domain expert votes for
the "best" segmentation from a collection of human and computer drawn segmentations for 70 objects from a public dataset and 274 objects from biomedical images. We publicly share the library of biomedical images which includes 1,879 manual annotations of the boundaries of 274 objects. We found for the 344 objects that no single segmentation source was preferred and that human annotations are not always preferred over computer annotations.
These results motivated us to examine the traditional approach to evaluate segmentation algorithms, which involves comparing the segmentations produced by the algorithms to manual annotations on benchmark datasets. We found that algorithm benchmarking results change when the comparison is made to consensus-voted segmentations. Our results
led us to suggest a new segmentation approach that uses machine learning to predict the optimal segmentation source and a modified segmentation evaluation approach.