TinyML for IoT Sensing: A Comparative Review of Frameworks for Deploying Machine-Learning Models on Low-Power Microcontrollers
Keywords:
tiny machine learning (TinyML), microcontrollers, edge AI, Internet of Things, model compression, quantisation, embedded inferenceAbstract
The Internet of Things (IoT) generates vast amounts of data from sensors, but sending raw data to the cloud for analysis adds latency, bandwidth and energy costs and privacy considerations. To overcome these limitations, Tiny machine learning (TinyML) is being developed that runs inference directly on the microcontroller unit (MCU), which has a few hundred kilobytes of memory and a few milliwatts power budget. This paper reviews and compares some of the main solutions that have been developed for deploying machine-learning (ML) models on low-power microcontrollers (MCUs): TensorFlow Lite for Microcontrollers (LiteRT), CMSIS-NN, and STM32Cube.Together with model compression methods for which they rely, such as integer quantization, pruning and neural architecture search, AI (ST Edge AI) and Edge Impulse and the MCUNet/TinyEngine co-design approach. The frameworks are contrasted with regard to memory consumption, inference latency, energy consumption per inference, accuracy loss upon compression, and hardware portability and usability. The examined evidence reveals that vendor-optimised kernels and compiler-based solutions perform best when it comes to latency and memory consumption on the target machine while interpreter-based and platform-based solutions are more portable and provide quick development. Significant challenges are (1) the lack of consistency in how benchmarks are performed, (2) the lack of support for learning on-device, (3) the hardware and toolchain fragmentation and (4) maintaining accuracy in extreme memory constraints. The review identifies hardware–software co-design and the use of a standardised benchmark like MLPerf Tiny as key pillars towards scaling TinyML from prototypes to reliable, large-scale IoT deployments.
References
[1] R. Sanchez-Iborra and A. F. Skarmeta, “TinyML-enabled frugal smart objects: Challenges and opportunities,” IEEE Circuits and Systems Magazine, vol. 20, no. 3, pp. 4-18, 2020.
[2] P. Warden and D. Situnayake, TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers. O'Reilly Media, 2019.
[3] C. Banbury et al., “MLPerf Tiny benchmark,” in Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, vol. 1, 2021.
[4] P. P. Ray, “A review on TinyML: State-of-the-art and prospects,” Journal of King Saud University - Computer and Information Sciences, vol. 34, no. 4, pp. 1595-1623, 2022.
[5] B. Jacob et al., “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 2704-2713.
[6] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding,” in International Conference on Learning Representations (ICLR), 2016.
[7] R. David et al., “TensorFlow Lite Micro: Embedded machine learning for TinyML systems,” Proceedings of Machine Learning and Systems, vol. 3, pp. 800-811, 2021.
[8] L. Lai, N. Suda, and V. Chandra, “CMSIS-NN: Efficient neural network kernels for Arm Cortex-M CPUs,” arXiv preprint arXiv:1801.06601, 2018.
[9] T. Chen et al., “TVM: An automated end-to-end optimizing compiler for deep learning,” in Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), 2018, pp. 578-594.
[10] S. Hymel et al., “Edge Impulse: An MLOps platform for tiny machine learning,” Proceedings of Machine Learning and Systems, vol. 5, 2023.
[11] J. Lin, W.-M. Chen, Y. Lin, J. Cohn, C. Gan, and S. Han, “MCUNet: Tiny deep learning on IoT devices,” Advances in Neural Information Processing Systems, vol. 33, pp. 11711-11722, 2020.
