Iterative Clustering for Memory-Assisted Reinforcement Learning in De Novo Drug Design Language Models

Abstract

Reinforcement learning for de novo molecular design often suffers from mode collapse, where the policy rapidly concentrates on narrow regions of chemical space at the expense of structural diversity. This thesis presents a modular approach for combining experience replay and clustering approaches to enhance diversity within REINVENT. Across three targets, an implementation based on BitBIRCH clustering generally improved fingerprint-based diversity and produced more diverse high scoring molecules than baseline and scaffold-based methods. The results also reveal a trade-off between diversity and score optimization. Experience replay accelerates early learning but may restrict exploration, while combining it with BitBIRCH partially mitigates this effect and can improve diversity further. Overall, the proposed method is effective but target dependent, emphasizing the need to evaluate molecular quality and multiple forms of diversity together.