Utilizing heterogeneity to allocate ML tasks for increased efficiency

Abstract

There is a growing interest in using heterogeneous hardware and resource allocation to boost the efficiency of software applications. Proper use of both imposes additional burdens on software development. We look at characterizing some common machine learning tasks with regards to CPU-GPU systems, specifically for the NVIDIA Orin, in order to try and predict what conditions will give the highest performance and energy-efficiency. We then take an iterative approach for allocating said tasks to hardware and selecting resources based on our characterization, with either performance or energy-efficiency as a goal. We find that while there is room for improvement on per-task predictions there are various possibilities to gain significant benefits to performance and energy by properly utilizing hardware heterogeneity and resource allocation. Additional exploration of domain specific accelerators such as tensor cores shows significant potential for accelerating convolutions.