Design Space Exploration of Deep Neural Networks for Embedded Microcontroller Platforms

Deep neural networks (DNNs) and their ability to make accurate predictions when trained on large datasets are the backbone of today’s artificial intelligence. With advancing technologies such as the Internet of Things (IoT) and Industry 4.0, the ability to classify high dimensional data generated by sensors is becoming an increasingly important task. However, in many applications, data is recorded by sensors deployed in the field, while most artificial intelligence (AI) algorithms traditionally run in the cloud due to their massive computational requirements. This becomes a challenging problem, especially when there is no stable and high-throughput network connection between the sensor and the cloud to send recorded data to the server for classification.

As a result, efficient AI is a research trend that aims to enable the deployment and execution of DNNs on embedded devices directly at the sensor. In addition to the advantage of not needing a network connection to the cloud at all times to classify data, other advantages, such as reducing the energy and resource requirements of DNNs, are of great interest in the context of efficient AI.

This dissertation aims to investigate the efficient and automated design and deployment of DNNs on resource-constrained microcontroller units (MCUs). To this end, three major contributions are presented as part of this work. First, an end-to-end training, compression, and deployment methodology for DNNs is proposed, discussed, and evaluated, focusing on DNN pruning and quantization for network compression as well as graph-based and instruction set based optimizations to enable memory and compute-efficient deployment of DNNs on Cortex-M MCUs. In addition to these optimizations and to further help to minimize the memory requirements of deployed DNNs, a novel layer fusion technique is proposed as part of the methodology as well.

Second, a novel multi-objective optimization method for automated and target hardware-aware design of efficient DNNs is proposed. Novel compared to other work is the combination of Bayesian optimization with reinforcement learning (RL) as an informed and trainable sampling strategy, allowing for sample-efficient optimization of DNN compression, i.e., not requiring the training of hundreds of models, taking into account not only accuracy, but also memory, and inference latency. Third, an algorithm for memory and compute efficient on-device training of fully-quantized DNNs within the memory and compute constraints of common Cortex-M MCUs is introduced. Besides supervised training and fine-tuning of DNNs the proposed algorithm is also tested and evaluated for unsupervised training using variational autoencoders (VAEs). This, for one of the first times, tackles the problem of the unavailability of labels of sensor recorded data on the edge.

In summary, the results presented in this dissertation show that DNNs achieving state-of-the-art accuracy on classification tasks both for vision and time series datasets can be designed and optimized to be deployable on Cortex-M MCUs with resource constraints as low as 256 KB RAM and 1 MB ROM that run at a clock speed in the lower MHz range. Even more, this dissertation is one of the first works to show that besides inference only, DNNs deployed on an MCU can even be trained and fine-tuned on-device, enabling continuous learning and adaptation of DNNs to new and unseen data even after their initial deployment.