Explainable AI for the Transformer Model Used on Chemical Language

dc.contributor.authorBükk, Caroline
dc.contributor.authorHoang, Linda
dc.contributor.departmentChalmers tekniska högskola / Institutionen för data och informationstekniksv
dc.contributor.examinerStrannegård, Claes
dc.contributor.supervisorJohansson, Richard
dc.date.accessioned2022-06-21T11:40:07Z
dc.date.available2022-06-21T11:40:07Z
dc.date.issued2022sv
dc.date.submitted2020
dc.description.abstractOne of the main challenges in drug discovery is to find new molecules with desirable properties. In recent years, using deep learning models to change the properties of a molecule has shown promising results. This task is done by letting the model transform the original molecule, and is often referred to as molecular optimization. A problem with using deep learning models is that it is difficult to understand what the model bases its decisions on. In our project, understanding what the model basis its decision on could be valuable feedback to drug designers and chemists. It could both extend their understanding of suitable transformations in different scenarios and provide insight in how the model could be improved. In this thesis, we have focused on explaining the Transformer model, when used to perform molecular optimization. As the molecules in this task are expressed in a chemical language, this problem can be viewed as a machine translation problem. The predicted molecule then corresponds to the translation of the input molecule and the desirable property changes. To explain the model, we considered a set of assumptions of what the model would focus on. The assumptions were inspired by the chemists’ intuition regarding what should influence the transformation most. The attention weights of the cross-attention layer were then analysed to test if these assumptions were correct. In order to determine if a contribution to the transformation could be considered important, relative comparisons between different parts of the input and output were used. We found that in some regards, the chemists’ intuition agreed with our comparisons of the attention weights. However, in some cases, the absolute value of the attention weights on the important parts were still very low. For future work, we suggest additional assumptions based on the chemists’ intuition and experiments to test them. We also suggest to use the explainability technique, integrated gradient, that could be applied similarly and used to verify our results.sv
dc.identifier.coursecodeDATX05sv
dc.identifier.urihttps://hdl.handle.net/20.500.12380/304838
dc.language.isoengsv
dc.setspec.uppsokTechnology
dc.subjectExplainable AIsv
dc.subjectattention weightssv
dc.subjecttransformersv
dc.subjectNLPsv
dc.subjectmolecular optimizationsv
dc.subjectmachine translationsv
dc.subjectmachine learningsv
dc.titleExplainable AI for the Transformer Model Used on Chemical Languagesv
dc.type.degreeExamensarbete för masterexamensv
dc.type.uppsokH
Ladda ner
Original bundle
Visar 1 - 1 av 1
Hämtar...
Bild (thumbnail)
Namn:
CSE 22-39 Bukk Hoang.pdf
Storlek:
13.88 MB
Format:
Adobe Portable Document Format
Beskrivning:
License bundle
Visar 1 - 1 av 1
Hämtar...
Bild (thumbnail)
Namn:
license.txt
Storlek:
1.51 KB
Format:
Item-specific license agreed upon to submission
Beskrivning: