Sample Efficient Game Strategy Through Active Demonstrations

Abstract

AlphaZero-style learning trains agents in game-playing purely through self-play, but requires millions of games before converging. In domains like robotics, imitation learning has been used to increase sample efficiency. The goal of this thesis is to incorporate imitation learning and expert feedback into AlphaZero-style learning. We extend Gumbel AlphaZero to query an expert for demonstrations during self-play, measuring uncertainty as the variance of returns backed up through search under the agent’s selected action, and querying when this variance exceeds a sliding-window threshold. We propose two methods for using these demonstrations: corrective behavioral cloning, where the expert’s intervention adjusts the search policy target; and action reranking, where a discriminator score nudges the search prior toward expert-like moves. We implement the system in Rust using the Burn framework, making it feasible to train on consumer hardware. We train three chess agents and evaluate them in a head-to-head tournament, where the demonstration-trained agents consistently beat a self-play baseline at matched sample counts. However, the improvement is small relative to the cost of expert queries. Analysis reveals that both methods improve the learned prior, but in high-uncertainty positions, the value estimate dominates action selection, leaving the improved prior with little influence exactly where guidance is most needed. The uncertainty measure itself proves useful: across all agents, positions with high return variance concentrate mistakes, supporting it as a query criterion for future work.