Generalizability of Representation Learning on Downstream Tasks

Abstract

We study the theoretical generalizability of representations learned by contrastive learning by analyzing their performance in downstream linear regression tasks. To quantify the quality of these learned features, we provide rigorous proofs for the worst-case and expected downstream performances under specified assumptions. These results offer theoretical results for analyzing the quality of features by looking at downstream tasks. To empirically verify the theory, we conduct experiments on both simulated and real-world data, following time-contrastive learning (TCL) strategies, which solve the problem of nonlinear independent component analysis (nonlinear ICA). In the real-world setting, we construct nonlinear source separation tasks using mixed audio signals from different instrument categories. Then, we propose some specific downstream tasks to verify our theories using our learned features from the TCL method and compare with the observed features. The results show that most of the defined downstream tasks are located in the confidence level of the expected performance and are bounded by the worst case, which indicates that our theory aligns with the real-world setting.