Adaptive Convolutional Neural Networks for Enhanced Memory Retention and Restoration in Optoelectronic Vision Devices

Optoelectronic devices based on optically responsive materials have gained significant attention due to their low cross talk and reduced power consumption. These devices rely on light‐induced changes in conductance states, which are used to create synaptic weights for image recognition tasks in neural networks. However, a major drawback of such devices is the rapid decay of conductance states after light stimulus removal, which hinders their long‐term memory and performance without a continuous external stimulus in place. To address this issue, a platform neural network scheme is proposed to counter the natural decay of conductance in optoelectronic devices. The approach restores the memory effect of the devices and significantly enhances their performance by several orders of magnitude without using additional energy‐intensive techniques like training pulses or gate fields. Herein, the model is validated experimentally using optoelectronic devices fabricated with two different materials, BP and doped In2O3, and demonstrates the restoration of memory/image retention ability to any material system being studied for optoelectronic synapses and vision. This approach has important implications for the practical application of neuromorphic visual processing technologies, bringing them closer to real‐world applications.

Optoelectronic devices based on optically responsive materials have gained significant attention due to their low cross talk and reduced power consumption. These devices rely on light-induced changes in conductance states, which are used to create synaptic weights for image recognition tasks in neural networks. However, a major drawback of such devices is the rapid decay of conductance states after light stimulus removal, which hinders their long-term memory and performance without a continuous external stimulus in place. To address this issue, a platform neural network scheme is proposed to counter the natural decay of conductance in optoelectronic devices. The approach restores the memory effect of the devices and significantly enhances their performance by several orders of magnitude without using additional energy-intensive techniques like training pulses or gate fields. Herein, the model is validated experimentally using optoelectronic devices fabricated with two different materials, BP and doped In 2 O 3 , and demonstrates the restoration of memory/image retention ability to any material system being studied for optoelectronic synapses and vision. This approach has important implications for the practical application of neuromorphic visual processing technologies, bringing them closer to real-world applications.
information, and then process this information, all using a single monolithic architecture while being fully light-controlled, offering the prospect of ultralow energy consumption for their operation. [1,4,15,21,22] These technologies typically deploy lightresponsive materials and utilize their intrinsic photoelectrical responses to store information in the form of analog conductance values. The conductance is regulated by the input incident light. For instance, the conductance of the device increases as either the duration of light (pulse width) or the intensity of light increases for a certain wavelength of light. [15,[23][24][25] The optoelectronic device responds differently to the different wavelengths of light depending on the properties of the light-responsive semiconductor used. [25][26][27] As such, the devices rely on capturing the light at particular wavelengths which results in the formation of electron-hole pairs. In certain materials, a phenomenon of persistent photoconductivity (PPC) can also be exploited for mimicking synaptic plasticity and implementing a neural network for processing of information for tasks such as pattern/image storage, recognition, and classification.
A CNN can be implemented when the current can be set to a certain real value using different pulse widths of light and the kernels of the neural network can be updated during the training phase by tuning the wavelength of light. [28][29][30][31] One of the key features of neural systems that is required to be mimicked by devices for learning and decision-making is longterm potentiation. A fundamental memory rule that results in long-term visual memory is the ability of the device to perform light-stimulated potentiation and conductance retention. This feature is essential for memorization and storage. The natural behavior of photoactive materials dictates that the conductance decays quickly once the input signal (light) is removed. Devices reported to date require a simultaneous use of frequent rehearsals (usually in the form of light or electrical pulses) and relatively high polarity-switching gate voltages to maintain longterm conductance retention. In the absence of these external stimuli, the conductance decays within a timeframe of a few milliseconds.
As such, a key challenge is the natural decay of the PPC. This results in the need for frequent refreshing, and thus poses a challenge in the practical applicability of these devices. As this decay in current directly affects the performance and accuracy of the neural network where conductance values represent the weight, the neural network is unable to retain its performance for extended durations which is a challenge for practical use and autonomous decision-making. [32,33] While these changes are currently managed through application of gate fields or alternating polarity voltage pulses, this significantly increases energy use per device and increases chances of neural network failure. Additionally, the decay of photocurrent is a natural behavior of optically active materials. As such, it is important to develop models and strategies to mitigate the effects of decay in optoelectronic devices for neuromorphic computing. Such a model is however not needed in cases where a particular conductance state does not naturally decay.
Optoelectronic devices have been proposed as a promising alternative to traditional artificial synapses in neural networks due to their ability to provide analog weight values that can be updated through iterative training cycles. These devices offer a unique opportunity to adjust synaptic weights in a parallel and energy-efficient manner, which is crucial for applications such as pattern recognition that require large-scale neural networks to process complex and varied data inputs. To optimize the performance of optoelectronic devices in neural networks, a compensation model is required to counter the natural decay of conductance in these devices, which poses a challenge for practical use and autonomous decision-making.
In this article, we address this gap through the design of an integrated compensation model for CNN applications. The compensation model is designed to counter the decay of conductance in optoelectronic devices and maintain a long-lived stable state thereby resulting in a long-term memorization capability without the application of an external field. The model enables full restoration of the conductance value that represents the weights from the decay conductance value by implementing what we call a compensation method. We show that our model is adaptable by applying it to two different photoresponsive material classes with markedly different photocurrent features, namely black phosphorus (BP) and doped indium oxide (In 2 O 3 ). We experimentally demonstrate that after applying our model, the memory retention ability is enhanced from few milliseconds to several minutes, while the neural network performance is maintained near the original level. We further show the ability of the model to retrieve lost information. This result opens a new pathway for practical deployment of such optoelectronic visual systems that are currently limited only to proof-of-concept demonstrations.
The model enables full restoration of the conductance value that represents the weights from the decay conductance value by implementing what we call a reversal method. We show that our model is adaptable by applying it to two different photoresponsive material classes with markedly different photocurrent features, namely BP and In 2 O 3 . We experimentally demonstrate that after applying our model, the memory retention ability is enhanced from few milliseconds to several minutes, while the neural network performance is maintained near the original level. We further show the ability of the model to retrieve lost information. This result opens a new pathway for practical deployment of such optoelectronic visual systems that are currently limited only to proof-of-concept demonstrations. In addition, the potential of translating this compensation model into a hardware system by using an integrated microchip/processor that receives input from the optoelectronic device is promising for future research. This proof of concept demonstrates that data from hardware can be easily adapted and implemented into a CNN, providing a foundation for further exploration of the hardware implementation of the proposed network.

Result and Discussion
Optoelectronic technologies have been shown to emulate the human visual system's ability to capture images and information (similar to retina) using photoactive materials, which subsequently harbor multiple analog conductance states to store memory and process information using a network of neurons similar to the visual cortex of the brain. [3,33,34] There are three key components for achieving this: 1) detection of light through photodetectors, 2) storage in an analog manner, and 3) processing using neural networks. [35,36] Among the various neural network architectures, CNNs are typically used given their inherent advantage in pattern and image recognition. [37] In this article, we look specifically at neural networks, where a typical layout of an artificial visual system along with a schematic layout of general CNN architecture is shown in Figure 1a,b.
The basic functionality of a CNN-enabled process is shown in Figure 1b. First, the pixel value of the image is held as neurons in the input layer. Then, the neurons in the input layer pass to the neurons in the hidden layer. In the hidden layer, there are multiple convolutional layers, activation functions, and pooling layers. The neurons in the hidden layer comprise weights and bias matrices. The neuron from one layer to another layer as a function of activation unit a i weights and bias values determines the output of the neuron from the input layer by updating weights and bias values during the training phase as shown in the following equation [37,38] In Equation (1), f is the nonlinear activation function, and w iÀ1 and b iÀ1 are the synaptic weight and bias of previous layer i À 1, respectively. For the layer i ¼ 2, a iÀ1 ¼ x i is the input data that comes from the previous neuron. Then, the pooling layer performs downsampling of the dimension of the given input from the convolutional layer to prevent overfitting during the training phase by reducing the number of parameters. [37,38] These values are subsequently converted to the output vector in the neurons of the output layer. This eventually provides the classification result. [38] The difference between the result and label value gives the accuracy which indicates the performance of the neural network. To achieve a higher performance, the network will keep training by updating the weights and bias values until high accuracy is attained. [39][40][41][42] In the case of optoelectronic devices, the weights and bias in the layers are stored to use as inbuilt analog memory array. However, this memory array in an optoelectronic device decays with time once the light stimulus is removed as shown in memory decay (red curve) in Figure 1c. If the resulting reduced weights are used, the neural network will not be able to classify the input image. Therefore, restoring the weights and bias to the original value (ideal memory (blue curve) in Figure 1c) without application of any external energy (e.g., gate fields) will significantly enhance utility and lifetime of the neural network.
To show the adaptability of the designed model across different materials, we implement it on two different photoactive materials for retaining image recognition capability for a longer period of time and also an ability to reconstruct the image from decayed weights. We fabricated two-terminal devices: one on a   [15,[43][44][45] A schematic representation of the device structure is shown in Figure 2a while the optical images of the respective devices are shown in Figure S1, Supporting Information. Both BP and doped In 2 O 3 have been reported to exhibit PPC behavior when illuminated with ultraviolet light. [15,[45][46][47] We chose these two different materials systems because of their strikingly different photocurrent decay rates upon stimuli removal. This is intended to demonstrate the adaptability of our CNN model for any material system. The photocurrent decay rate of BP is %10 times faster than that of the doped In 2 O 3 system (Figure 2b,c). First, we investigate the photoconductivity of the optoelectronic device under a monochromatic test light wavelength of 285nm. Two terminal devices, one for each of the photoresponsive materials (BP and doped In 2 O 3 ), are supplied with a small bias DC voltage of 50 mV. [48,49] A monochromatic 285 nm light source was used to illuminate the BP and doped In 2 O 3 with a power density of 3 mW cm À2 for 150s. Upon illumination with 3mW cm À2 , the current shows a steady increase. This can be termed as the "ON" period of the device or the visual input. www.advancedsciencenews.com www.advintellsyst.com Once the light is turned off, the current decreases exponentially until it reaches a value close to the baseline current. Depending on the pulse width of illumination, the magnitude of current rise can be controlled. Owing to the presence of defect and trap states, some residual current in the device may persist at levels slightly above the baseline current after stimulus removal. This is typically called a persistent photocurrent. [15,50] In comparison to BP, doped In 2 O 3 requires a longer duration for the photocurrent to decay (Note S2, Supporting Information), making the two materials an ideal test case for the performance of our CNN model. [51][52][53][54][55][56] The decay rate of the conductivity depends on the lifetime of the photogenerated charge carriers, and carriertrapping dynamics . [50,[57][58][59] In the human brain, there are multiple neurons, and these neurons communicate with each other via synapses. Synapses are connected with sensory organs and if the sensory organ is stimulated by heat, pressure, or other stimuli, it will activate the synapse. Similarly, in CNN, there is a kernel called synaptic weight in the artificial neuron and in the optoelectronic device, the conductance values represent the synaptic weights by applying 285 nm wavelength of light with 3 mW cm À2 in BP and doped In 2 O 3 .
For creating our CNN model, we focus on the decay characteristics of the photoresponsive materials. Herein, the photocurrent, which represents the conductance of the device, is measured by applying 50 mV across the two terminals of the device. The initial data is represented as the current value immediately after the light is off. The initial value can be controlled by the illumination pulse width. BP and doped In 2 O 3 are applied with 5, 15, 150 ms pulse widths of light. Figure 2d,e shows the current decay profiles after applying different pulse widths on BP and doped In 2 O 3 once the illumination is turned off. In synaptic mimicking optoelectronic devices, current values represent synaptic weights that form inputs of neural networks. Therefore, this decay implies loss of data and therefore immediately deteriorates the accuracy of neural networks for tasks such as classification, recognition, and identification. This also implies that to mimic the long-term memory characteristics, frequent rehearsals in the form of illumination cycles or polarity switching gate fields are applied.
To visualize this decay, we demonstrate data storage as current in an optoelectronic device (see Figure S2, Supporting Information). The image pixels are normalized within the range of 0 and 1, based on the peak current values (from 0.55 to 1.06 μA in BP and from 0.01 to 0.35 μA in doped In 2 O 3 ) by using a linear interpolation method which transfer the data from pixel domain to current domain of respective material (Note S3, Supporting information). Figure 2f,g shows how the conductance decay upon illumination removal manifests into the decay in a typical image that is stored using normalized current values derived from BP and doped In 2 O 3 devices, respectively. The image quality deteriorates with time. The figures also illustrate the effect of the varying photocurrent decay rates for the two material systems. For instance, in Figure 2f, for BP, at time 0 s, the image is very clear. When the current starts to decay, the image quality also decays. After 10s of decay time, the image brightness starts to decrease visually. This is because the current drops to 80% of the original value. After 60 s, the current drops to 40% of the original data which leads to the image being almost faded. And after 200 s, all data from the image can be perceived as lost and it goes totally black. Figure 2g shows the decay effect of doped In 2 O 3 . In doped In 2 O 3 , a similar trend is observed. However, the image faded slower in this case compared to BP. For instance, after 10 s, the image holds more information than BP. However, after 50s, the image of doped In 2 O 3 is still holding almost the same information as BP at 10s. After 200 s, the image is totally blacked out. This is due to the photocurrent decay rate differences between the two materials (doped In 2 O 3 decays slower compared to BP). As such, over time this leads to a poor performance of neural networks.
To demonstrate the decay effect on the performance of a neural network, an example CNN with 3 convolutional layers is built to classify the benchmark data obtained from the Modified National Institute of Standards and Technology (MNIST). [60] Figure 3a shows the architecture of the utilized neural network to classify the MNIST database. In the CNN, there are eight 3 Â 3 kernel weights for the first convolutional layer, sixteen 3 Â 3 Â 8 kernel weights for the second convolutional layer, and thirty-two 3 Â 3 Â 16 kernel weights for the third convolutional layer. Using this architecture of neural network, we trained the neural network with MNIST dataset and acquired 98.21% accuracy. This accuracy indicates the neural network's high-performance capabilities. We take this as the baseline accuracy for testing the performance of the neural network. To indicate the network is still performing as originally designed, we propose that the accuracy of the neural network should only drop at max by 10% of the baseline accuracy after applying the decay characteristic of the materials. Thus, the range of the accuracy of the neural network is between 98.21% and 88.39%.
The confusion chart of the neural network is shown in Figure 3b. Each convolutional layer has positive and negative weights. However, the conductance of the optoelectronic device is always positive. To overcome this issue, we separate the synaptic weight as a difference of two conductance value, that is, G ¼ G m,n À J, where J indicates the values of G max þG min 2 . If the G m,n is greater/lesser than J, it indicates a positive/negative weight, respectively. [15] Positive weights and negative weights are separately stored in the different arrays of the optoelectronic device by using the linear interpolation method to transfer from weight domain to current domain (Note S3, Supporting Information). Under simulation, the effect of decayed weights on the performance of the neural network is tested and the results are shown in Figure 3c. After the weights are reduced below 90% of the original value, the reduction of weight directly affects the performance of the neural network. When the weights drop to 85% of the original value, the neural network accuracy drops to 88.39% and when the weight drops to less than 65%, the accuracy sharply drops to 10%, which is the lowest possible accuracy (because we have ten classes). This shows that to preserve the high performance of the neural network, the value of the decayed weights needs to be restored as close as possible to the original value.
To overcome this core problem of optoelectronic vision technologies, herein, we propose a compensation model that can restore the decayed data close to its original value. The compensation model imitates the decay current curve of each pulse width, and the original weight value can be restored by applying an inverse method. To generate the model, we found that each individual decay curve for BP and doped In 2 O 3 can be fitted to the Kohlrausch-stretched exponential function as follows [55,61] where IðtÞ is the current value at time t, I 0 is the initial current value just after the illumination phase, α is the decay rate constant, β is the stretching exponent, and I ss is the steady-state current post decaying. [49] The decay parameters α, β, and I ss are dependent on the pulse width of light during the illumination phase, that is, they are dependent on the initial current I o . [55,62] Therefore, the model can be expanded to include the decay curves of different pulse width as follow In Equation (3), α, β, and I ss are hyperparameters. These hyperparameters are polynomial fits that depend on the initial current I o and are designed to prevent overfitting problems and make the model more robust to data errors, including white noise or temperature fluctuations (see Note S4, Supporting Information). By adapting to variations in these parameters, our method can account for changes in device initial current values that may occur for different devices under the same conditions. Additionally, we ensure that all devices are preconditioned to a baseline current before starting the measurement, further reducing the impact of device-to-device variation. Therefore, our compensation method focuses on accounting for device-to-device variation and ensuring that all devices start from the same known state before measurement, rather than relying on a specific value for the starting current. Figure 4a,b shows the comparison between the simulated data using the compensation model in Equation (3) and the real data under the different pulse widths of light for BP and doped In 2 O 3 , respectively. Figure 4c,d shows the root-mean-square error (RMSE) of the data to the initial current I 0 . The average value RMSE is about 1.7 and 0.7 nA for BP and doped In 2 O 3 , respectively. RMSE values are a thousand times smaller than the data value which indicates that the compensation model can mimic the real data with small variance. When performing the inverse modeling from Equation (3), we note that there are three main unknown variables which are the peak current I 0 , the decay current IðI 0 , tÞ, and the decay time t. Therefore, the peak current I 0 can be recalculated using the model, if the decay current value IðI 0 , tÞ and collecting time t are known. From the retrieved peak current, the original data values such as image pixels and weights are restored by using linear interpolation method. This also means that the data in current domain are converted back to the weight domain and used it on the same neural network to observe the performance of the neural network.
To understand the limitation of the compensation model, the decayed weights and compensated weights at different time points are fed into the previously built CNN. Figure 5 shows the performance of the neural network using decayed weights and compensated weights for BP and doped In 2 O 3 over time. It is clear that the compensation model can enable holding the initial high accuracy of the neural network for a much longer www.advancedsciencenews.com www.advintellsyst.com period. Figure 5a,b shows the performance of the CNN using decayed weights. We observe that in case of using the uncompensated decay weights, the accuracy of the neural network rapidly goes below our previously defined tolerance    Figure S5a,c, Supporting Information). Therefore, it is expected that the performance of the neural network will drop rapidly after that time. Similarly, in doped In 2 O 3 , the current drops to 85% around 0.01s, and the performance of the neural network is expected to drop rapidly after 0.01s as well (See Figure S5b,d, Supporting Information). Figure 5c shows the accuracy of the neural network using the compensation model to restore the original weight for BP. It can be observed that the holding time for the duration of the accuracy above 88.4% is 12.33 min (739.8 s). Similarly, Figure 5d shows the performance of the neural network using the compensation model in doped In 2 O 3 . The holding time is 150 min (9000 s) in this case. It is therefore clear that our compensation model can be adapted for different decay rates and significantly enhance the performance of the neural network for extended durations.
We would like to clarify that although our offline training mode is not intended for real-time applications, the conductance states of our optoelectronic devices can be easily refreshed by applying a gate voltage or a corresponding wavelength of light to the device. This allows us to initialize the synaptic weights while still benefiting from the advantages of neuromorphic and in-memory computing. The refresh process can be done in parallel for all devices in the network, making it highly efficient and scalable. While the hardware implementation details have yet to be explored, our work opens up new possibilities for future research in the field of neuromorphic computing.
Our optoelectronic devices serve as both synapses and neurons, making our in-memory computing approach highly efficient in terms of energy and computation. The process of writing the restored conductance states back to the optoelectronic devices can be accomplished using a circuitry that applies a voltage or a light pulse to the device. The focus of our current work is on developing the idea of restoring the data, and we believe there is great potential for future research in the field of neuromorphic computing.
To observe the efficacy of the compensation model visually, the decayed image pixel values from Figure 2f,g are restored using the model. Figure 6 shows the restoration of the image pixel value, which is obtained as a model simulation that relies on the electrical conductance of the BP and doped In 2 O 3 devices that are fabricated and tested experimentally. The quality of the images is retained for significantly longer. This has implications when such vision systems are deployed in practical applications allowing time for not only processing, but autonomous decisionmaking. However, if we observe the restoration of the image after holding time (12.33 Figure S6-S9, Supporting Information). This indicates the limitation of the compensation model which can be further designed for longer durations by extending the interpolation.
To investigate the limitation, from Figure 4a,b, we can see that as the time increases, the current graphs converge toward each other. In other words, the differences between each current graph shrink as the time prolongs. This also indicates that after

Conclusion
In summary, we have designed a compensation and restoration CNN model to enhance the retention of conductance-driven memory in optoelectronic vision devices. We successfully measured the decay characteristic of the conductance in the optoelectronic device and designed a CNN model that compensates for this decay. In storing the weights, the decay causes the performance of the neural network to deteriorate to very low values within a few seconds in both tested devices. It is worth mentioning that the model can be used in other optoelectronic devices by simply adapting the specific decay characteristic of the material used. In restoring the image pixel value, the compensation model can restore the image for the same duration as well. In conclusion, the model can enhance the memory retention ability from milliseconds to several minutes which also opens a new pathway for practical deployment in applications such as optoelectronic visual systems. With further study of the white noise of the data and the ambient environment effect on the device, it is possible to achieve a low-energy optoelectronic device with even longer memory retention that can practically be used in CNNs applications.

Experimental Section
Fabrication of Material: Black Phosphorus: Several layers of thin BP flakes were mechanically separated from bulk crystals (Smart Elements) and drytransferred to polycarbonate (PC)-coated glass slides with Gel-Pak thin films. SP flakes with a thickness of 5-25 nm were identified by optical and atomic force microscopy. The PC areas containing the selected BP flakes were then placed on a polydimethylsiloxane stamp and transferred to the desired locations on the pre-marked SiO 2 /Si substrate using the "HQ graphene" manipulation transfer system placed. The transfer stage was then heated at 200°C for 5 min to transfer the PC containing BP flakes to the substrate following a similar process described in ref. [63].
Fabrication of Material: Doped In 2 O 3 : The In 2 O 3-x Sb x nanosheets were synthesized by using squeeze printing method that has previously been reported. [45,63] In-Sb alloy was prepared by mixing metal precursors at 450°C for 2 h in a nitrogen glove box. Under ambient conditions, a small portion of the prepared alloy was placed on a preheated substrate of silicon (Si)/silicon dioxide (SiO 2 ). A second substrate was used to squeeze the placed alloy to obtain the nanosheets. Kapton tape was used to remove residual metals, followed by cleaning in isooctanol 180°C, rinsing in isopropanol, and compressing air-dry off.
Device Fabrication: Two terminal photoconductor devices were fabricated using positive photolithography, electron beam-assisted metal deposition and finally metal liftoff. The 10 nm Cr adhesion layer along with a 100 nm Au layer was used as lateral electrodes on the liquid metalsynthesized nanosheets. These fabricated two terminal photoconductors had an active area of 5 Â 25 μm 2 .
Testing Device: Photoresponse was performed using a Keysight 2910BL source measurement unit and Linkam stage at ambient conditions (in air at room temperature) with a drain-source bias of 50 mV. The devices were excited with light-emitting diodes (LEDs) (Thorlabs Inc.) with 285 nm wavelengths at a power intensity of 3 mW cm À2 (calibrated with a commercial photodetector, Newport Corporation) to measure its photodetection capabilities. All measurements were performed under dark conditions with exposure to only the target illumination wavelengths in ambient environment. The photoresponse of the samples was performed in the range of 5-150 s of illumination which was controlled by the microcontroller board. After each data was taken, the next data collection was restarted after the photoresponse reached to the baseline current (the current under dark). Field-effect transistor (FET) characterization of the device in contrast was carried out at gate voltages of À60 to 60 V, with the same drain-source bias of 50 mV.
CNN Simulation: The CNN architecture is shown in Figure 3a. The hidden layer of CNN was composed as follow: convolutional layers, each with eight channels of shape 3 Â 3; a 2 Â 2 max-pooling layer; 2 convolutional layers, each with 16 channels of shape 3 Â 3; a 2 Â 2 max-pooling layer; 2 convolutional layers, each with 32 channels of shape 3 Â 3; a 2 Â 2 max-pooling layer; and a fully connected layer with ten outputs.
Each convolutional layer was followed by activation layer called a rectified linear unit. MNIST dataset was used as input image with the size of 28 Â 28 Â 1.The MNIST dataset has 60 000 images in the training dataset and 10 000 images in the testing dataset. The CNN with aforementioned architecture was trained in MATLAB using training dataset. The specific settings in the training process are as follows: the batch size was 16, stochastic gradient descent with momentum optimizer, learning rate was 0.01, and total epoch was 30. The weights kernels in the convolutional layers were converted to decayed weight and compensated weights using linear interpolation and compensation model. Then, the performance of CNN with decay and compensated weights were tested using testing dataset.

Supporting Information
Supporting Information is available from the Wiley Online Library or from the author.