Controlling Diffusion

Abstract

In this thesis, the primary focus is the exploration and evaluation of input-output mappings within the components of diffusion models, extending conventional methods of control like prompt engineering. The objective is not to propose a definitive solution for control within diffusion models, but rather to probe the underlying mappings that drive these processes. The project examines two main components: The attention maps within the CLIP model, and the input-output relationships in the diffusion process itself. By doing so, the intention is to increase understanding of the inherent complexity of the models and identify potential control opportunities. In the intricate domain of diffusion models, the evaluation of input-output maps might not always offer a clear-cut measure of success. This project contributes by examining the interplay between various components of the model, as viewed through the lens of input-output mappings. Instead of presenting conclusive control methods, it examines these components, setting a possible path for further exploration.