Walking the Talk: Vision-Language Orchestration of Navigation and Motion-Imitation Skills on a Quadruped Robot

Abstract

Campus guide robots are expected not only to guide visitors through complex environments but also to interact naturally, respond to changing intentions, and engage through expressive physical behaviour. Realising these capabilities requires tight coordination of conversational intelligence, navigation, and learned motor control. This thesis presents FIDO (Friendly and Interactive robot DOg), a quadruped campus guide built on the Unitree Go2 platform, integrating full-duplex conversation, vision-language orchestration, indoor–outdoor navigation, and learned motor skills within a unified perception–action framework. We develop ShallWeTalk for predictive turn-taking and interruption handling, and a Vision-Language Orchestrator that translates speech, visual observations, conversational context, and robot state into spoken responses and executable actions. A Robot Execution Backend executes actions and reports outcomes, while LiDAR localisation and RTK-GNSS/odometry fusion support indoor–outdoor navigation. To expand FIDO’s motor repertoire, we construct MyGO2, a library of 1,731 Go2 motion clips, and develop a general motion-tracking controller. We introduce two hand-designed reference motions: Crawl, which uses a lower body posture than any motion in MyGO2, and Moonwalk, an expressive backward-sliding movement. For Moonwalk, we train dedicated tracking policies using two approaches: asymmetric actor–critic learning and teacher–student distillation. We also develop EasyGO2, a common deployment framework supporting MuJoCo sim-to-sim validation and hardware execution. During live English conversations, FIDO stops speaking for eight of nine interruptions but continues through six of eight backchannels, tolerating pauses of up to 1.5 s without added response latency relative to a silence-based baseline. The general controller completes 163 of 176 unseen clips in simulation and executes Crawl zero-shot on the Go2; both Moonwalk approaches yield policies that succeed in all five hardware trials without hardware fine-tuning. FIDO relocalises in all 30 indoor trials and reaches all 20 navigation goals. Experts rate output quality of the Vision-Language Orchestrator at 98.5/100 across 149 tasks. Integrated campus demonstrations show FIDO following a 231 m outdoor route in response to a spoken request and, in a separate indoor run, changing its goal, pausing, and resuming on voice commands. Together, these results demonstrate how coordinated dialogue, navigation, and learned motor control enable natural, responsive, and physically expressive visitor interaction.