Low-power, low-complexity image classification with optical sensors
OA Version
Citation
Abstract
Convolutional Neural Networks (CNNs) have become a very effective tool in image estimation and inference (e.g., noise reduction, segmentation, object recognition). However, CNNs are difficult to deploy in resource-constrained scenarios, such as edge devices, IoT sensors, mobile embedded systems, because of high computational requirements and large memory footprint. Clearly, there is a growing need to investigate lightweight neural-network architectures designed to provide high performance at significantly reduced computational cost, memory usage, and power consumption. In this thesis, we investigate a low-power, low-complexity hybrid optical-digital neural network that leverages a novel metasurface sensor recently developed in Professor Paiella’s lab. Unlike typical image sensors, this sensor outputs an edge-like map of the scene akin to the output of the first convolutional layer of a CNN. We simulate this physical sensor in software and combine it with just a few digital layers to assure low computational load and power consumption. Since different pixels of the sensor capture different edge orientations, we organize the sensor array into groups of 2-by-2 or 3-by-3 pixels capturing either 4 or 9 edge orientations. This leads to either 4-channel or 9-channel convolutional layer simulation. We jointly optimize the optical parameters of this layer and digital parameters of the remaining layers for image classification of low-resolution images. Our best-performing designs approach classification performance of equivalent fully-digital network within 2% points, while reducing computational complexity and power consumption by a factor of 7.
Description
2025