Satyam: An Open Source Groundtruth Annotation Platform for Vision
Automated Groundtruth Collection for Vision
While deep learning has led to machine vision systems that rival human abilities, their training requires massive amounts of annotated groundtruth data. Obtaining such data often presents a significant challenge as it requires employing and managing a large human workforce.
Satyam is an open-source automated groundtruth collection system for machine vision tasks, such as classification, detection, tracking, segmentation, etc. The system enables researchers to collect high-quality labels at scale and build custom datasets for novel applications with minimal effort, latency and cost.
System Overview
Satyam is designed to minimize friction for users when collecting groundtruth for ML-based machine vision systems. It eliminates the need for users to develop complex Web-UIs for vision annotation tasks, or manually intervene in quality control, or task launch and management.
A core module of Satyam is automated quality control, which provides high quality groundtruth data in the face of error prone workers and even spammers. Satyam obtains work for the same image/video from multiple workers and fuses their partially correct work using cross-task generic groundtruth-fusion techniques to generate high quality groundtruth.
Automated Quality Control
Human Annotations
Aggregated Result
Although each annotation is less than perfect, Satyam is able to automatically identify correct parts and fuse them to significantly improve the quality of groundtruth. Our key contribution in this paper is a unified fusion algorithm that can be parameterized across different kinds of tasks.
Result Demo
Performance Benchmark
Satyam is extensively evaluated across various tasks, including image/video classification, object counting, detection, tracking, segmentation, etc., on multiple public datasets (e.g. KITTI, PASCAL, ImageNet, CARPK, JHMDB, etc.). Results show that Satyam achieves ~95% match accuracy. Additionally, Satyam is able to identify extra ~5% missing or incorrect annotations among public datasets.
Dataset: The TrafficCam Challenge
We have used Satyam to collect human annotations for various machine vision tasks on public traffic camera streams. Two of the primary foci are on day time and seasonal changes.
Morning
Afternoon
Night
Spring
Summer
Fall
Winter
We are releasing our first dataset: FourSeaons!
Site Title 