Design and Empirical Evaluation of a Multi-Agent System for Engineering Workflows

Abstract

Early-phase truck integration requires engineers to coordinate vehicle data, regional regulations, engineering calculations, and CAD-related operations across heterogeneous tools. This thesis designs, implements, and empirically evaluates a toolorchestrated multi-agent system for a bounded set of these tasks. The research examines heterogeneous-resource integration through a traceable and extensible toolorchestrated architecture, correct workflow execution and contract conformance on the defined benchmark, and trade-offs in accuracy, evidence faithfulness, reliability, and latency across tasks and configurations. The prototype uses a LangGraph supervisor–specialist architecture comprising one orchestrator and five ReAct subagents for Database, Regulation, Payload, Agility, and CAD tasks. A separate non-agent utility node manages the RAG knowledge base. The study follows an artifact-centred design-science strategy. Evaluation is conducted at the offline, component, system, and implementation levels using frozen datasets. These datasets include a simulated vehicle database, a curated local regulation corpus, deterministic reference calculations, and explicit workflow checks. On the defined benchmark, routing accuracy was 91.0% across 100 requests, and 49 of 50 end-to-end tasks passed the required sequence and output checks. The Payload calculation results agreed with an independent oracle in all 60 evaluated cases. In the Database evaluation, the required values were retrieved more often than the generated result exactly matched the reference result. This difference demonstrates why capability-specific metrics are needed. At retrieval depth five, the annotated answer fact was found for 83.65% of regulation queries, whereas the specifically annotated source was retrieved for 64.42%. The paired automatic comparison between rule-only and RAG-only regulation answering did not detect a statistically significant difference in term match (p = 0.1263), while rule-only execution had a substantial latency advantage. The results support the feasibility of bounded multi-agent orchestration for engineering assistance. They do not establish production performance or superiority over a monolithic agent. The quantitative findings are limited to the synthetic benchmark and small document corpus. The CAD and Agility measurements cover planning rather than complete geometry execution. The study therefore supports a capability-specific interpretation in which deterministic operations, retrieved evidence, execution traces, and continued human oversight remain necessary for dependable engineering use