Weighted Ensemble Distillation in Federated Learning with Non-IID Data

dc.contributor.authorEriksson, Oscar
dc.contributor.departmentChalmers tekniska högskola / Institutionen för matematiska vetenskapersv
dc.contributor.examinerGebäck, Tobias
dc.contributor.supervisorGebäck, Tobias
dc.date.accessioned2022-06-19T17:05:51Z
dc.date.available2022-06-19T17:05:51Z
dc.date.issued2022sv
dc.date.submitted2020
dc.description.abstractFederated distillation (FD) is a novel algorithmic idea for federated learning (FL) that allows clients to use heterogeneous model architectures. This is achieved by distilling aggregated local model predictions on an unlabeled auxiliary dataset into a global model. While standard FL algorithms are often based on averaging local parameter updates over multiple communication rounds, FD can be performed with only one communication round, giving favorable communication properties when local models are large and the auxiliary dataset is small. However, both FD and standard FL algorithms experience a significant performance loss when training data is not independently and identically distributed (non-IID) over the clients. This thesis investigates weighting schemes to improve the performance with FD in non-IID scenarios. In particular, the sample-wise weighting scheme FedEDw2 is proposed, where client predictions on auxiliary data are weighted based on the similarity with local data. Data similarity is measured with the reconstruction loss on auxiliary samples when passed through an autoencoder (AE) model that is trained on local data. Image classification experiments with convolutional neural networks performed in this study show that FedED-w2 exceeds the test accuracy of FL baseline algorithms with up to 15% on the MNIST and EMNIST datasets for varying degrees of non-IID data over 10 clients. The performance of FedED-w2 is lower than FL baselines on the CIFAR-10 dataset, where the experiments display up to 5% lower test accuracy.sv
dc.identifier.coursecodeMVEX03sv
dc.identifier.urihttps://hdl.handle.net/20.500.12380/304781
dc.language.isoengsv
dc.setspec.uppsokPhysicsChemistryMaths
dc.subjectfederated learning, federated distillation, knowledge distillation, weighted ensembles, artificial intelligence, privacy, image classificationsv
dc.titleWeighted Ensemble Distillation in Federated Learning with Non-IID Datasv
dc.type.degreeExamensarbete för masterexamensv
dc.type.uppsokH
Ladda ner
Original bundle
Visar 1 - 1 av 1
Hämtar...
Bild (thumbnail)
Namn:
Master_s_thesis_finalversion110622.pdf
Storlek:
25.54 MB
Format:
Adobe Portable Document Format
Beskrivning:
License bundle
Visar 1 - 1 av 1
Hämtar...
Bild (thumbnail)
Namn:
license.txt
Storlek:
1.51 KB
Format:
Item-specific license agreed upon to submission
Beskrivning: