A Computer Vision Sensor for Efficient Object Detection Under Varying Lighting Conditions

Convolutional neural networks (CNNs) have attracted much attention in recent years due to their outstanding performance in image classification. However, changes in lighting conditions can corrupt image segmentation conducted by CNN, leading to false object detection. Even though this problem can be mitigated using a more extensive CNN training set, the immense computational and energy resources required to continuously run CNNs during always‐on applications, such as surveillance or self‐navigation, pose a serious challenge for battery‐reliant mobile systems. To tackle this longstanding problem, a vision sensor capable of autonomously correcting for sudden variations in light exposure, without invoking any complex object detection software, is proposed. Such video preprocessing is efficiently achieved using photovoltaic pixels tailored to be insensitive to specific ranges of light intensity alterations. In this way, the pixels behave similarly to neurons, wherein the execution of object detection software is only triggered when light intensities shift above a certain threshold value. This proof‐of‐concept device allows for efficient fault‐tolerant object detection to be implemented with reduced training data as well as minimal energy and computational costs and demonstrates how hardware engineering can complement software algorithms to improve the overall energy efficiency of computer vision.

DOI: 10.1002/aisy.202100055 Convolutional neural networks (CNNs) have attracted much attention in recent years due to their outstanding performance in image classification. However, changes in lighting conditions can corrupt image segmentation conducted by CNN, leading to false object detection. Even though this problem can be mitigated using a more extensive CNN training set, the immense computational and energy resources required to continuously run CNNs during always-on applications, such as surveillance or self-navigation, pose a serious challenge for battery-reliant mobile systems. To tackle this longstanding problem, a vision sensor capable of autonomously correcting for sudden variations in light exposure, without invoking any complex object detection software, is proposed. Such video preprocessing is efficiently achieved using photovoltaic pixels tailored to be insensitive to specific ranges of light intensity alterations. In this way, the pixels behave similarly to neurons, wherein the execution of object detection software is only triggered when light intensities shift above a certain threshold value. This proof-of-concept device allows for efficient fault-tolerant object detection to be implemented with reduced training data as well as minimal energy and computational costs and demonstrates how hardware engineering can complement software algorithms to improve the overall energy efficiency of computer vision.

Fault-Tolerant Sensor Operational Framework
In this work, we propose one such device architecture and demonstrate how it may be used to reduce the computational load associated with object detection under varying lighting conditions. Here, our pixel sensor is envisioned to consist of two independent units, one being an object-detecting pixel (ODP) like silicon photodiodes found in commercial cameras and the other being a fault-tolerant pixel (FTP) that corrects for lighting alterations. Both ODP and FTP can be constructed next to each other or in a stacked tandem structure (provided the FTP is transparent enough to transmit sufficient light to the ODP). Such division of visual tasks to handle the complexity of vision resembles the layered construction of the retina where different photoreceptive nerves process certain features of vision as well. As glares and shadows are simply changes in greyscales, the FTP just needs to manage white light. The idea behind achieving insensitivity to glares and shadows is for the FTP's outputs to only change if alterations in light intensity exceed a certain threshold. This is reminiscent of neuron behavior, where a signal is only fired when the stimulus overcomes a fixed threshold. In this way, our neuromorphic sensor can serve as a video preprocessing filter layer for image sampling, signaling the computer to run CNN only for those pixels where light intensity changes have exceeded the preset threshold, thus reducing the overall computational load. In contrast to similar silicon-based technologies like neuromorphic event cameras, [10][11][12][13] our pixel sensors do not require additional circuitry that consumes energy and limits pixel size for high video resolution. Figure 1a shows a plausible use case for our pixel sensors when conducting synchronous image frame-based object detection based on CNN.
Step 1: First, the ODPs detect the new object entering the scene in the image frame (Frame 1) by running CNN.
Step 2: For the subsequent image frame, the computer compares pixel values from our FTPs with those from the previous frame to check if there are any changes. This step can be conducted at a much lower computational cost than running CNN analysis for every single frame.
Step 3 Case 1: If the FTP values remain within the preset threshold, it means that our sensor "sees" the same object, despite exposure to variations in lighting conditions. In this case, these FTPs will not signal the computer to conduct CNN on the ODPs in this frame and for all successive image frames, until such a time when the FTP threshold has been exceeded (introduction of new object or more extreme lighting changes, for instance). In this way, our neuromorphic FTPs minimize runtime of CNN, hereby reducing the overall energy and computational burden.
Step 3 Case 2: If, in contrast, some pixels vary due to more extreme lighting condition changes, triggered, for instance, by the appearance of a completely different object or by the onset Figure 1. Operational framework of the fault-tolerant sensor. a) Simplified illustration of the working principle of our video prefilter pixels. In Frame 1, the object is detected by running CNN analysis. In Frame 2, the prefilter pixels will output different voltages depending on the specific lighting conditions of this frame. If the variation is mild, as shown in Case 1, the voltage output of the FTP remains within the preset threshold. When light intensities vary significantly however, as shown in Case 2, the FTP will output a dramatically different voltage value. This can then be used to trigger the execution of object detection software (CNN). b) Schematic illustration of different types of shadows cast on a 24-pixel representation of a rectangle (white squares represent one pixel each). c) Image of our 24-pixel sensor and schematic architecture of a dye-sensitized solar cell (DSSC)-based single FTP. Red spheres represent the dye-sensitized TiO 2 mesoporous photoanode film. d) FTP's open-circuit voltage V OC response at various white LED light intensities of a representative pixel of the 24-pixel sensor. e) FTP's percent deviation in V OC with respect to the highest V OC value (at highest light intensity) as derived from graph d. Error bars were derived from 24 different FTPs.
www.advancedsciencenews.com www.advintellsyst.com of motion of the perceived stationary object, then the transformed FTP values will trigger CNN to conduct object detection (step 1). After this, step 2 of this algorithm resumes. According to this algorithm, our sensor is most efficient and impactful when dealing with moderate lighting alterations that do not severely mask key features of an object (Figure 1a and b). In these cases, CNNs do not have to be executed for each image frame, thus significantly reducing energy and computational costs. In contrast, for more "extreme" shadows leading to complete obscuring of a major piece of a rectangle for instance ( Figure 1b, bottom), the FTPs perceiving the shadow will output voltages exceeding the predefined tolerated threshold, triggering the computer to run CNN or other shadow detection and removal software to correctly identify the object.

FTP Characteristics
To attain the desired invariant FTP output with varying light intensities, we utilize dye-sensitized solar cells (DSSCs) in the open-circuit potential V OC mode as the V OC depends logarithmically on light intensity and is hence less sensitive to lighting variations than the short-circuit current. In general, any photovoltaic device such as silicon, perovskite, or organic photovoltaics can be utilized for this purpose. The reason for choosing DSSCs over other sensors such as silicon-based devices is the ease of modifying the V OC via simple electrochemical means, as will be discussed later on. The ability to tune the V OC response to the degree of light exposure is crucial as the FTP cannot be too insensitive to a wide range of intensities; otherwise, scene variations, such as the appearance of a new object, expressed as a major change in lighting contrast, cannot be perceived anymore. Before discussing how the light intensity range for V OC invariance can be tuned, the basic charge transfer mechanisms occurring in an illuminated DSSC-based FTP are first briefly reviewed. [14,15] Upon light exposure, the photoactive dye molecule absorbs a photon and promotes an electron to the dye's lowest unoccupied molecular orbital (LUMO), shown by process 1 in Figure 2a. The promoted electrons are then injected into the TiO 2 conduction band (CB), as shown in Figure 2a, process 2. Under open-circuit conditions, the injected charges accumulate in the TiO 2 film, leading to a shift in the TiO 2 quasi-Fermi level toward the CB. The difference between the quasi-Fermi level and the redox potential of the redox couple in the electrolyte determines the V OC . When the now-oxidized dye molecules gain an electron from a nearby reducing agent in the electrolyte through a process called dye regeneration (process 3 in Figure 2a), the dye molecules are restored to their initial state, thus enabling the whole cycle of photon absorption and electron injection to restart. The injected electrons in the TiO 2 CB may also be transferred to www.advancedsciencenews.com www.advintellsyst.com nearby oxidizing agents present in the electrolyte via a loss mechanism known as charge recombination (process 4). This process would result in a decrease in TiO 2 electron density, which leads to the lowering of the quasi-Fermi level, thus reducing the opencircuit potential of the pixel. For the FTPs to exhibit minimal deviation in V OC with changing light intensities, the number of electrons residing within the TiO 2 photoanode must be similar for all light intensities within the range of interest. Generally, the TiO 2 electron density, and hence the V OC , is dependent on both the light intensity and the recombination rate. For instance, the higher the intensity, the more photons available to be harvested by the dyes, resulting in more electrons being injected into the TiO 2 film. In contrast, when for example the recombination rate is high, the TiO 2 electron density decreases, leading to lower V OC . [16][17][18] Hence, understanding the interplay between light intensity and recombination rate is crucial for achieving V OC invariance under varying light exposure. [19,20] Therefore, we start with studying the charge recombination rates for our FTPs, using the open-circuit voltage decay (OCVD) method. [21,22] In the following sections, we detail how our sensor conducts fault-tolerance analysis by itself without computational operations from the computer, hence dramatically minimizing computational load. As a proof-of-concept, we fabricated sensors consisting of 24 pixels, each comprising electrically independent 500 μm by 500 μm square DSSCs, only sharing the same electrolyte and counter electrode (Figure 1c). Further scaling down of TiO 2 pixel sizes can potentially be achieved using inkjet printing [23] or 3D nano printing [24,25] to yield higher-resolution image detection. The individual DSSC FTP consists of an organic dye-sensitized mesoporous TiO 2 photoanode infiltrated by a liquid electrolyte containing 0.1 M Co(II)(bpy) 3 [TFSI] 2 as the reducing agent (a more detailed device architecture can be found in Supporting Information). For our sensors, fault-tolerant behavior is expressed as an invariance in open-circuit potential V OC delivered by the DSSC pixels in response to the specific range of light intensity alterations. Here, invariance means that the V OC does not exceed a user-defined threshold value-in this study, we define this threshold to be of maximum 10% V OC deviation limit upon changes in light intensity. Different V OC thresholds can be used, depending on the range of light intensity variation one intends to cover for the desired object-detection application. Figure 1d shows the V OC output of a single representative pixel as a function of illumination time at different light intensities (white LED light). The percent V OC deviations from the value obtained at the highest probed intensity of 57.2 k Lx are shown in Figure 1e. It is evident that for the intensity range from 57.2 to 7.1 k Lx, the pixel's maximum-saturated V OC values stay within a 10% margin, thus exhibiting fault-tolerant object detection within a 50.1 k Lx intensity variation. As the difference in light intensity increases, i.e., toward lower values (1.5 k Lx in Figure 1d, e), the V OC response grows further apart as well. This demonstrates that this particular FTP is most useful within the 57.2kÀ7.1 k Lx range, which corresponds well to outdoors daylight applications. Figure 2b shows the TiO 2 electron lifetimes as a function of light intensity, with the inverse of these electron lifetimes representing the charge recombination rates. For the intensity range where fault-tolerant behavior is exhibited (from 57.2 k Lx to 7.1 k Lx, black curve), the TiO 2 electron lifetimes, and hence the recombination rates, are similar despite a 50 k Lx intensity variation. This result suggests that, despite the differences in TiO 2 electron densities generated within the 50 k Lx range, the TiO 2 quasi-Fermi level, and hence the observed V OC , did not significantly shift when charge carrier losses are minimal, as mirrored by the similar recombination rates. At lower light intensities outside the fault-tolerant range where the V OC values decrease and are thus not invariant anymore, the TiO 2 electron lifetimes are prolonged (Figure 2b). Typically, extended electron lifetimes and hence slower recombination rates yield higher V OC due to more suppressed electron losses. [26,27] The observed lower V OC thus implies that the fewer electrons injected at lower light intensities play a more crucial role in determining the V OC as compared with the slower recombination rate, contrary to the fault-tolerant region where the light intensities have little impact on V OC .
To further investigate the effect of recombination rate, DSSC pixels were fabricated, containing, in addition to the 0.1 M Co(II)(bpy) 3 [TFSI] 2 , the oxidizing agent Co(III)(bpy) 3 [TFSI] 3 at the same 0.1 M concentration for the purpose of enhancing charge recombination. The red curve in Figure 2d shows the percent V OC deviation from the value obtained at the highest probed illumination (at 57.2 k Lx, Figure 2c) as a function of light intensity for these enhanced recombination pixels. In contrast to the Co(II)(bpy) 3 [TFSI] 2 -only FTP, the 10% V OC deviation threshold only holds for a narrower intensity range. That is, the lower fault-tolerant threshold occurs at higher light intensities in the presence of Co(III)(bpy) 3 [TFSI] 3 . For these pixels, the TiO 2 electron lifetime, as expected, is shorter than that in the Co(II)(bpy) 3 [TFSI] 2 -only FTP for all probed intensities, as shown by the red graph in Figure 2b. Based on these findings, it can be deduced that the pixels exhibiting the higher recombination rates cannot maintain a comparable number of TiO 2 electrons at lower light intensities in relation to the higher intensities, leading to the observed narrower fault-tolerant intensity range. That is, at low intensities, fewer electrons are being injected into the TiO 2 photoanode than at higher intensities, and if, in addition, the loss mechanism is more severe, as is the case for the pixels exhibiting faster recombination, then the V OC values will differ more significantly. In contrast, if recombination is efficiently suppressed, as in the Co(II)(bpy) 3 [TFSI] 2 -only FTPs, the differences in TiO 2 electrons generated within a larger intensity range do not impact the V OC as dramatically.
To provide further evidence for the crucial role of charge recombination rates on the fault-tolerant light intensity region, Co(II)(bpy) 3 [TFSI] 2 -only FTPs were fabricated without any dense TiO 2 blocking layer (Figure 1c). The absence of such a blocking layer creates additional recombination sites, leading to shorter TiO 2 electron lifetimes, as shown by the blue graph in Figure 2b. This device exhibits a similarly narrower V OC invariant light intensity range as the Co(III)(bpy) 3 [TFSI] 3 containing FTPs, thus underscoring the importance of charge recombination rates on the FTPs' fault-tolerant behavior. These findings suggest that simple manipulation of the recombination rates allows tailoring of the V OC -invariant light intensity region.

Object Detection Subject to Sudden Lighting Alterations
To demonstrate the proof-of-concept application of our FTP sensor, we exposed it to a simple scenario where a stationary object is www.advancedsciencenews.com www.advintellsyst.com subject to the sudden appearance of varying lighting conditions, using a similar intensity range as before. To simulate such a case, we first illuminate the entire 24-pixel sensor arranged as a rectangle, consisting of two rows of 12 pixels (Figure 1c) at low light intensity of 7.1 k Lx. Subsequently, the sensor was exposed to higher light intensities to simulate the emergence of glare. We use the most fault-tolerant Co(II)(bpy) 3 [TFSI] 2 -only FTPs as an example. All pixels were measured concurrently (without multiplexing) to probe their true time responses to light exposure without any delay caused by consecutive pixel sampling. Figure 3a shows that the V OC values of all sensor pixels are invariant within a 10% range in the presence of a glare at 33 k Lx. The brightest glare that can be tolerated occurs for an alteration of light intensity from 7.1 to 57.2 k Lx (Figure 3b), where the individual V OC pixel values remain within the 10% preset threshold as well, thus demonstrating highly efficient fault-tolerant behavior. Similarly, positive results were obtained for the introduction of shadows. Here, we begin with a bright object (represented by 57.2 k Lux) exposed to the sudden occurrence of a shadow. Figure 3c and d show the percent V OC deviation of all sensor pixels for shadows at 33 and 7.1 k Lx, respectively, indicating that the FTPs' V OC values still reside within the 10% variation limit required for fault-tolerant object detection.
Next, we studied the ability of our sensor to correct for a moving glare or shadow. The exposure of an object to moving glares or shadows is an important scenario that occurs frequently in everyday situations, which is particularly detrimental for motion-detecting algorithms using background subtraction techniques. Here, the computer compares successive image frames and subtracts their individual pixel values [6] to determine which objects or which pixels have undergone a change, signaling possible motion. In the presence of glare or shadows, this process may yield erroneous results, which can trigger high instances of false alarms and increased operational costs, especially for autonomous surveillance systems. To test whether our sensor is capable of correcting for a moving glare, the device was illuminated with white light from a projector (representing a stationary 24-pixel rectangle), together with a brighter two-pixel strip traveling from left to right at a 50 ms frame rate, representing a moving glare, as highlighted in the movie S1, Supporting Information. Figure 4a,b shows the percent V OC deviations of our sensor during this simulation. The pixels exhibit an increase in V OC at the instance of glare appearance, as indicated by the spikes in V OC . However, this V OC deviation remains within the 10% threshold for all pixels. Similarly, our sensor's ability to detect a stationary object exposed to a moving shadow was demonstrated by Figure 3. Detecting a 24-pixel object subject to varying lighting conditions. V OC deviation from the initial object's brightness of 7.1 k Lx, expressing the detection of a stationary 24-pixel rectangle illuminated at 7.1 k Lx and subject to the sudden appearance of a glare with intensity of a) 33 k Lx (room light is shown as reference to get a sense of the differences in various brightness) and b) 57.2 k Lx. Similarly, percent V OC deviation from the initial object brightness of 57.2 k Lx, expressing the detection of a stationary 24-pixel rectangle illuminated at 57.2 k Lx and subsequently exposed to a c) 33 k Lx and d) 7.1 k Lx shadow, respectively. www.advancedsciencenews.com www.advintellsyst.com conducting a similar experiment, but with a dimmer two-pixel strip, representing the traveling shadow instead. Figure 4c and d shows the percent V OC deviation of our sensor for a two-pixel strip shadow traveling at a frame rate of 50 ms. The negative percent change in V OC mirrors the registration of the moving shadow. For all probed pixels, the V OC deviation is well within the 10% threshold, demonstrating that our sensor is capable of detecting simple objects despite alterations in lighting conditions. Further increasing the number and density of sensor pixels could allow for the detection of more complex objects.

Conclusion
In summary, we have demonstrated a proof-of-concept neuromorphic sensor capable of detecting simple objects subject to light intensity variations, without incurring computationally expensive object recognition software such as CNNs. This sensor is most suitable for fault-tolerant object detection under low-tomoderate lighting alterations, where key features of the object are still visible and not completely obscured. Even though simple, such a scenario occurs nevertheless quite frequently in everyday life and our "noise filtering" hardware can dramatically minimize energy and computational burden for the computer. Our results serve to highlight the benefits of incorporating neuromorphic hardware within existing computer vision design frameworks as a strategy to boost energy and computational efficiency.

Experimental Section
Device Structure: Each sensor pixel consisted of a DSSC. This cell contained a screen-printed mesoporous transparent semiconductor photoanode sensitized with a photoactive dye (Dyenamo Red dye DN-F05, chemical structure shown in S6, purchased from Dyenamo and used as is). This photoactive layer was deposited on FTO coated on glass (TEC 7, purchased from Greatcell). Further, the cell contained an electrolyte sandwiched by a counter electrode. This sensor architecture contained photoactive films that were printed as individual pixels, where the photoactive films were electrically isolated via FTO etching. The counter electrode as well as the electrolyte were shared by all pixels. . Detecting a 24-pixel object exposed to moving glares and shadows. Percent V OC deviations of all 24 pixels subject to a a,b) vertical two-pixel glare strip and c,d) shadow moving at a frame rate of 50 ms (see Movie S1 and Movie S2, respectively, Supporting Information). Dashed lines indicate the peak V OC of the two pixels detecting the moving vertical glare or shadow strip. First 12 pixels from left to right, indicated by the orange and red arrows (a, c), 6 top and 6 bottom row, and last 12 pixels (b, d) are shown on separate graphs for clarity purposes.
www.advancedsciencenews.com www.advintellsyst.com Device Fabrication: Patterned FTO-coated glasses (purchased from Latech 14 ohm sq À1 ) for the 24-pixel sensors were cleaned by heating at 500 C for 10 min. Subsequently, a compact TiO 2 layer was coated onto these glasses by spin coating a solution of titanium isopropoxide (TTIP) (254 mL TTIP/5.6 mL HCl 35%/2 mL ethanol). Spin coating was conducted at 2000 rpm for 60 s. These spin-coated substrates were sintered at 500 C for 30 min. Then, TiO 2 paste (30NR-D Titania paste from Greatcell Solar) was deposited via screen printing to form a transparent mesoporous layer. Subsequently, the substrates were sintered at 500 C for 30 min. Once the glasses were cooled down, they were immersed for 1 day in an organic dye (Dyenamo Red DN-F05) solution (0.1 mM in tert-butanol/acetonitrile 1:1 v:v). These photoanodes were sealed with bare FTO-coated glass (Greatcell Solar TEC7) acting as the counter electrode, using a thermoplastic sealant film Surlyn (50 mm thin). The electrolyte was injected from previously drilled holes onto the counter electrode. For this study, two different electrolytes were used, 0.1 M Co(II)(bpy) 3 [LiTFSI] 2 and 0.1 M Co(II)(bpy) 3 [LiTFSI] 2 /0.1 M Co(III)(bpy) 3 [LiTFSI] 3 (purchased from Dyenamo DN-C14) in 50 mL methoxypropionitrile (MPN). As the final step, the counter electrode holes were sealed using Surlyn.
Device Characterization: Open-circuit voltage (V OC ) of fabricated solar cells were measured using a National Instruments NI PXIe-1071 24-channel source measurement unit (SMU). For the light source, a high-power LED day white light Solis-3C from Thor Labs was used. Moving glares and shadows were projected using an EPSON EH-TW3200 projector. Motion movies were created with Microsoft PowerPoint using different shades of gray to mimic varying light conditions.

Supporting Information
Supporting Information is available from the Wiley Online Library or from the author.