Micro-Solutions for Micro-Adjustments - Reducing Micro-Adjustments During Annotation for Data Labeling Platforms: An Interaction Design Approach

Abstract

The autonomous vehicles sector has brought computerization into driving, a task that previously was solely for humans. This results in companies providing software and annotation services for machine learning, helping with labeling, categorizing, and tracking data. This Master’s thesis was conducted at one of these annotation companies, Kognic. The act of making adjustments to annotations of already sufficient quality, called micro-adjustments, is one optimization problem existing in Kognic’s annotation platform. However, annotations still need expert-level quality to generate as close as possible to ’ground truth’, a reality for the machine learning model to train on. Since the process handles huge datasets, the time spent on micro-adjustments results in major time losses. The thesis goal is to analyze annotator workflows, propose one or multiple solutions to improve the way annotators work with object annotation, and create general design guidelines for the reduction of micro-adjustments in annotationplatforms. This was achieved by utilizing interaction design following the research question: In which aspects can interaction design support the reduction of micro-adjustments (small repetitive unnecessary adjustments of annotations of already sufficient quality) in annotations of 3D point clouds? The study utilized aspects such as design thinking, user-centered design, and the triple diamond design process to address the wicked problem. The thesis showed that the iterative nature of interaction design is an efficient approach when designing to reduce micro-adjustments. The result was three final design solutions, Line Assist, Two Pointy, and Sequence Process Wizard, and 11 design guidelines. The design guidelines contribute to the field of annotation by outlining important considerations when designing for the reduction of micro-adjustments. These guidelines emphasize the importance of rotation micro-adjustments, feedback to annotators, improving existing tools over creating new ones, annotators culture, and how micro-solutions can minimize micro-adjustments. Additionally, the guidelines can assist in projects with users located far away from where the study takes place, highlighting the significance of gathering information from multiple sources.