Software-Engineered Human-AIAnnotation Systems for Safety-Critical Perception Pipelines

Abstract

AI-assisted annotation systems are increasingly used to support perception data generation for autonomous and ML-enabled systems. While modern AI models can improve annotation efficiency and detection coverage, integrating multiple AI components into human-in-the-loop workflows also introduces architectural tradeoffs related to functional correctness, performance efficiency, maintainability, human cognitive workload, and usability. Existing research often focuses primarily on model-level accuracy, while the broader impact of workflow orchestration and human-AI interaction remains less explored. This thesis investigates how different architectural design patterns influence the behavior and quality attributes of AI-assisted annotation systems. Building upon architectural design patterns for ML-enabled systems, six human-AI annotation variants were designed and evaluated, including sequential, parallel, single-model, and VLM-assisted workflows. A modular annotation platform integrating YOLO11n, OWL-ViT, VLM-based prompt generation, and human validation was developed following a design science research approach. The evaluation combined controlled annotation experiments and a complementary user study. Functional correctness, maintainability, performance efficiency, annotation consistency, human intervention effort, and cognitive workload were analyzed across architectural variants using both quantitative metrics and qualitative feedback from participants. The results demonstrated that no single architectural variant optimized all evaluated quality attributes simultaneously. The Parallel variant improved detection diversity and recall but increased cognitive workload and interaction complexity. The Sequential variant reduced cognitive effort and improved usability through simplified validation-oriented interaction, although it introduced stronger dependency propagation and orchestration complexity. The findings further showed that increasing architectural complexity or adding additional AI components does not automatically improve overall effectiveness. Overall, the thesis highlights that evaluating ML-enabled human-AI systems requires considering not only predictive performance, but also maintainability, interaction design, human oversight, and long-term sustainability. The findings provide practical architectural recommendations for future AI-assisted perception annotation systems.