Neuron‐Inspired Time‐of‐Flight Sensing via Spike‐Timing‐Dependent Plasticity of Artificial Synapses

3D sensing is a primitive function that allows imaging with depth information generally achieved via the time‐of‐flight (ToF) principle. However, time‐to‐digital converters (TDCs) in conventional ToF sensors are usually bulky, complex, and exhibit large delay and power loss. To overcome these issues, a resistive time‐of‐flight (R‐ToF) sensor that can measure the depth information in an analog domain by mimicking the biological process of spike‐timing‐dependent plasticity (STDP) is proposed herein. The R‐ToF sensors based on integrated avalanche photodiodes (APDs) with memristive intelligent matters achieve a scan depth of up to 55 cm (≈89% accuracy and 2.93 cm standard deviation) and low power consumption (0.5 nJ/step) without TDCs. The in‐depth computing is realized via R‐ToF 3D imaging and memristive classification. This R‐ToF system opens a new pathway for miniaturized and energy‐efficient neuromorphic vision engineering that can be harnessed in light‐detection and ranging (LiDAR), automotive vehicles, biomedical in vivo imaging, and augmented/virtual reality.

Here, to alleviate the challenges, we report a simplified, lowpower, and neuron-like 3D-sensing platform by integrating avalanche photodiodes (APDs) with artificial synapses. We have demonstrated neuron-like integrated 3D-sensor by employing HfO 2 memristors and APDs. The HfO 2 memristors exhibit nonvolatile memory characteristics via their programmable conductance, which is dependent on the distribution of conductive oxygen vacancies. The conductance information can be harnessed as both analog memory and computing functionality via Ohm's law. By utilizing the neuron-like behavior of the memristor combined with the APD, the resistive time-of-flight (R-ToF) sensing approach provides the means for unprecedented miniaturization of ToF systems with a simplified structure and %50 times lower power consumption compared with conventional TDCs systems. Furthermore, to confirm the feasibility of our R-ToF system, we have performed 3D LiDAR imaging based on the proposed R-ToF principle. We have also explored the feasibility of using an R-ToF system for ANN-based classification by applying the STDP principle to depth-imaging and the neuromorphic MAC operation to HfO 2 crossbars. These multifaceted capabilities of our R-ToF system could be exploited further for use in full-3D visual sensing for advanced optoelectronic and machine vision applications.

R-ToF Principle and Device Characterization
In the biological visual system, the rod cell receives optical signals and converts them to electrical spikes toward the synapse that lies between the rod cell (presynaptic neuron) and the bipolar neuron (postsynaptic neuron). [28] If the timing is small enough, the synapse releases an acetylcholine neurotransmitter that transmits signals to the post neuron. Here, we have mimicked the biological synaptic responses in visual systems by implementing STDP in the memristors. The basic principle of STDP in the biological system is the transition of synaptic weights hinging on the time difference between presynaptic and postsynaptic spikes. Figure 1a shows the structural and Figure 1. Demonstration of R-ToF depth ranging. a) Schematic illustration of biological STDP behavior and R-ToF. The rod cell and the photodiode receive the optical signals and transmit to the synapse and the memristor, respectively. Inset: SEM image of memristor. Scalebar: 20 μm. b) Schematic illustration of R-ToF system including classification (implemented via Bechtold and Höfle [43] ). c) Principle of R-ToF system. The ToF is calculated by subtracting the receiving time (t r ) from the transmitting time (t t ). The ToF ðΔt 1 < Δt 2 < Δt 3 Þ corresponds to the target distances ðd 1 < d 2 < d 3 Þ. The shorter distance enables larger transitions of the conductance of the memristor, and the conductance is measured to range the distance.
www.advancedsciencenews.com www.advintellsyst.com functional analogy between the biological and artificial synaptic systems. The STDP learning rule applies to memristors because a conductive bridge by oxygen vacancies in the active medium is formed depending on the overlapped period between the presynaptic (to the top electrode of the memristor) and postsynaptic spikes (to the bottom electrode of the memristor). Therefore, the temporal difference between spikes can be obtained by reading the programmed conductance of the memristors. [29] The inset shows the scanning electron microscope (SEM) image of the fabricated memristors. We propose a new principle of R-ToF that converts memristor conductance to depth information. Figure 1b shows a schematic illustration of the R-ToF principle and classification. The ToF principle determines the distance by calculating the time difference between the transmitted and reflected flux as d ¼ 1 2 c · τ ToF , where d is the distance between a ToF sensor and target objects, c is the speed of light, and τ ToF is the time difference between transmitted (Tx) and received (Rx) signals. As shown in Figure 1c, the signals from the transmitter and receiver exhibit time-difference-based responses similar to the biological timevariant sandwiching spikes. Therefore, the polarities of the presynaptic and postsynaptic spikes are opposite. Following the ToF principle, the presynaptic spike in the R-ToF sensor is generated by an electrically transmitted signal, which also operates a picosecond laser. The postsynaptic spike in the system is transmitted by the current pulse of a received signal via APDs. The APDs adopted in the ToF sensing systems [30,31] as an optical receiver provide high internal multiplication gain, low dark current, and low excess noise. [32] The larger spike time differences between the transmitted and received signals ðΔt 1 < Δt 2 < Δt 3 Þ lead to longer programming times and higher peak amplitudes in the memristor by the superposition of the semisynchronous positive presynaptic peak and the negative postsynaptic peak. The longer and higher peak signals impart larger transitions of conductance (ΔG 1 > ΔG 2 > ΔG 3 Þ to the memristor. As a result, the conductance transitions of the memristor are converted to time-difference information following the precalibrated exponential-fitted relationship between the conductance and the programming time, which leads to the final depth information To confirm the functionality of R-ToF sensors, we fabricated and integrated the HfO 2 memristors with APDs. Figure 2a,b shows schematic illustrations of the R-ToF device and cross sections of the fabricated HfO 2 memristors and AlInAsSb APDs, respectively. The postsynaptic spike is originated from the photogenerated current of the APD usually in the range of 1 μA to 0.1 mA and is converted to the amplified voltage signal via a transimpedance amplifier (TIA) %1À2 V to program the memristors. A full description of the fabrication process is detailed in Experimental Section. Current-voltage (I-V ) characteristics of the fabricated HfO 2 memristor are shown in Figure 2c. The set and reset voltage are %2 and À1.5 V, respectively. The maximum and minimum resistance at the À0.1 V read voltage is %90 MΩ and 1 kΩ, respectively. Under the positive bias less than 2 V, the resistance of the HfO 2 memristor exhibits gradual transitions used for the R-ToF ranging in the following section. The gradual transitions are interpreted as the biological long-term plasticity (LTP), and the incremental voltage pulse below 2 V with a read pulse train ( Figure S1, Supporting Information) enables quantized analog conductance behaviors, as shown in Figure 2d. The cycling endurance performance of the memristor is shown in Figure 2e. The consistent switching response between the high resistance state (HRS) and the low resistance state (LRS) ensures high endurance in measuring depth information via the R-ToF system.
To convert the conductance switching of the memristors into time domain information, the time difference between spikes is calibrated by measuring the biomimetic resistive STDP of the memristor in nanosecond scale, as shown in Figure 2f. The LTP behavior of the artificial STDP rule decays when the time difference increases, which has also been observed in the electrosensory lobe of electric fish. [33] Similarly, the bioinspired memristive STDP learning rule shows an exponential relationship between the conductance transition and timing of spikes; therefore, the exponential fitting parameters are harnessed for the R-ToF conversion. This exponential relationship can be modulated by varying the spike decay, rise, and duration times. In the R-ToF sensor, the memristor enables optical programming through photogenerated current from APDs, as shown in Figure 2g,h. The reflected light intensity of the laser only slightly varies from 800 to 830 lux at the different object positions (10À55 cm) used in the following R-ToF ranging experiment, which is insufficient to influence the conductance switching of the memristors. Therefore, the conductance switching of the memristors is highly dependent on the resistive STDP behaviors produced via temporal variation of ToF ranging.

Neuromorphic R-ToF Ranging
The R-ToF system acquires depth information via a combination of optical programming and the STDP principle. The schematic R-ToF experimental setup is shown in Figure 3a. The presynaptic spike is generated and applied to both the memristor and the picosecond pulse laser as a trigger. Due to the parasitic inductance and capacitance from the external interconnection to the fabricated APDs, here we have used a commercial APD (Hamamatsu S9073) to confirm the feasibility. By applying a reverse bias up to À155 V to the APD, we were able to control the gain from 1 to 162 ( Figure S2, Supporting Information). As shown in Figure 3b, we performed a cyclic depth acquisition based on the R-ToF principle. The single cycle includes two read pulses, STDP spikes, and a reset pulse. The period of the single cycle is 3.3 μs corresponding to 330 kHz pulse repetition frequency (PRF) of the used picosecond pulse laser (see Experimental Section). The waveform is recorded for 100 μs including the absence or existence of a target object. To prevent the collision between the rippled reset pulse and the signals of the next cycle, we used an additional 1 μs delay time to mimic the biological refractory period in the transmission of nerve impulses. The postsynaptic spikes are only generated when there is a target object located in line-of-sight and the conductance of the memristor is stationary without the postsynaptic spikes. The R-ToF waveforms depending on the object locations are shown in Figure 3c. The peak of the superposed programming signal is low (high) when the time difference between the presynaptic spike and the postsynaptic spike is large (small), which leads to weak (strong) coupling. The analog transitions of www.advancedsciencenews.com www.advintellsyst.com the memristor conductance follow the analog programming via the aforementioned pulse train response generated by overlapped pre-and postsynaptic spikes, which verifies the depth memory functionality of the R-ToF system. Here, we have demonstrated R-ToF ranging from 15 up to 55 cm (Figure 3d). Prior to this depth measurement of the moving object, the nanosecond-scale transitions of the R-ToF sensor are verified by an experiment with a fixed position and a varied temporal difference of the presynaptic and postsynaptic spikes ( Figure S3a, Supporting Information). The detailed R-ToF ranging is demonstrated provided in the Experimental Section. Based on the R-ToF principle, the measured distances are obtained and compared with the actual distance, as shown in Figure 3e. Further ranging over 55 cm (so-called a long-range mode, Figure S3b, Supporting Information) is viable by applying a longer pulse delay with increased amplitude to the electrical presynaptic pulse and/or by incorporating multiple pulse-width laser optics. There is a tolerable difference between the measured and actual distance with %89% accuracy (concordance between the actual and measured distances) and 2.39 cm standard www.advancedsciencenews.com www.advintellsyst.com deviation (including both short-and long-range mode) that can be potentially harnessed as a macroscopic LiDAR system ( Figure S4 and Text 1, Supporting Information).

R-ToF-Based 3D Imaging
Using the R-ToF system, we demonstrated an R-ToF 3D LiDAR scan. As shown in Figure 4a, we emulated a street scene that includes a car, building, and wall miniature. The distances to each object are 15, 25, and 45 cm, respectively. By using series of single-point ranging, we observed the R-ToF-based imaging capability, as shown in Figure 4b, with an error map ( Figure S5, Supporting Information). For more details, see Experimental Section. Based on this feasibility, we also suggest the expanded R-ToF system to an array scale to realize 3D imaging and classification. Figure 4c shows the schematic of the expanded system that utilizes the synaptic STDP, LTP, and longterm depression (LTD) of HfO 2 memristors. A target object reflects the optical flux emitted by the laser source, and the  www.advancedsciencenews.com www.advintellsyst.com reflected optical signal is received by the APD array. The converted voltage signal via the TIA components generates STDP as a postsynaptic spike (the presynaptic spike is from reference electrical signal) to the one-transistor one-resistor (1T1R) array restoring rasterized depth information. The reconstructed image is then utilized as an input to pass through the feed-forward ANN (see Experimental Section). The transistor array enables access to the individual memristor cell for the programming, reading, and verification process. The Monte Carlo simulation shows the enhanced 3D reconstruction enabled by the averaged weight array over epochs that leads to the increased peak signal-to-noise ratio (PSNR), as shown in Figure 4d,e. The PSNR increases up to 35 dB, indicating promising reconstruction results concerning the ideal depth image. The spike bundles exhibit temporal responses depending on the pixel-wise depth information corresponding to the pixel locations of the APD. The normalized weight matrix is then converted to the depth information via parameterized function between the weight and depth. The entire 100 test and reconstructed images are shown in Figure S6, Supporting Information. We also present the simulated HfO 2 crossbars to enable the classification of the depth images (see Experimental Section). Figure 4f shows some examples of raw and R-ToF-reconstructed images. Although the reconstructed images contain some deformation and noise, the HfO 2 -based feed-forward ANN shows only a 4% accuracy decrease compared with the full-software classification (raw images with a software ANN), as shown in Figure 4g. The accuracy results of other combinations of the image types and the computer systems are in the range of these results ( Figure S7, Supporting Information). The robustness of the expanded R-ToF system can be further improved when both epoch and acquisition time increase, as shown in Figure 4h. As the scanning speed of the R-ToF system is inversely proportional to the acquisition time, the classification accuracy decreases if the scanning speed increases. At all acquisition times, the accuracy tends to increase at high training epochs. The expanded R-ToF system enables edge-computing and fully-hardware 3D classifier stemming from the depth acquisition with the ANN activation.

Discussion
The analog-domain depth ranging and storing capabilities of the R-ToF exhibit comparable power consumption and small footprint compared with the conventional TDC architectures ( Figure S8 and Text 2, Supporting Information), and enable two high degrees of freedom for further designing a 3D sensing system. First, the system can be electrically and optically modulated by tuning the spike schemes and gains from the APD and the TIA. Both memristive STDP and R-ToF behaviors are dependent on amplitude, decay, and rise times of presynaptic and postsynaptic spikes ( Figure S9, Supporting Information). By controlling the gain from an APD, up to 55.1 m ranging can be obtained (see Supplementary Text 3). In addition, the gain modulation relaxes a target object's reflectivity-dependent amplitude transition. In a practical LiDAR application, the intensity of the received optical signals depends on the distance from the objects as well as the reflectivity of the objects. The detection of low reflectivity objects can be improved through an optimized voltage window via the gain and spike modulation (see Supplementary Text 4). Furthermore, multiple echoes from multiple objects within a single ranging shot are separable when the distance between the individual objects are far enough. In this case, only one postsynaptic spike (from the closer object) is coupled to the presynaptic spike. Second, a high geometrical degree of freedom can be achieved including small footprint and complementary metal-oxide-semiconductor (CMOS) compatibility. Previous memristor studies have demonstrated CMOS-compatible architectures, small-footprint fabrication, and low power consumption because the memristor medium is also widely used for high-k dielectric layers in standard silicon CMOS process. [34,35] Heterogeneous integration methods are also applicable for our fabricated AlInAsSb APDs and memristors, and further system-level R-ToF implementation would enhance the vertical depth resolution as well ( Figure S10 and Note 4, Supporting Information). Furthermore, subnanoseconds switching speed of memristors has been reported, [36,37] indicating further improvement in the resolution of the R-ToF sensor (15 cm in this work) toward a millimeter ranging system. Ferroelectric materials such as Ag/BaTiO 3 /Nb: SrTiO 3 [38] and LiTaO, [39] and scalable magnetoelectric spin-orbit devices [40] are promising candidates due to their faster switching speed (more than one order) compared with the resistive HfO 2 layer.

Conclusion
In conclusion, we have demonstrated a neuron-like R-ToF method for 3D sensing. A memristor has been used to store the ranging information programmed by superposed pre-and postsynaptic spikes. The R-ToF system attained up to 55 cm ranging in a single mode, and further ranging was achieved by controlling the spiking scheme combined with the gain modulation of an APD and a TIA. To prove the feasibility of R-ToF sensing, we have demonstrated the expanded R-ToF system that includes two promising applications of 3D sensing technologies: 3D imaging and classification. 3D image reconstruction and classification using in-memory computing architecture were demonstrated by using multiple acquisitions and training epochs via an R-ToF array. The R-ToF system exhibits high biofidelity, a small footprint, simplified architecture, and low power consumption that provide a basis for realizing a fully-hardware neuromorphic 3D sensing technology.

Experimental Section
Device Fabrication: The HfO 2 memristors were fabricated on a 25 μm polyimide substrate. To prevent the expansion of the polyimide during the post process, a preannealing of the polyimide substrate was performed at 200 C for 30 min on a hot plate, followed by deposition of a 100 nm Al 2 O 3 buffer layer via atomic layer deposition (ALD) at 200 C. A 3/25 nm-thick Ti/Pt bottom contact was deposited and patterned by using e-beam evaporation and a photolithography process, respectively. A 5 nm HfO 2 switching layer was deposited using ALD at 200 C. A 50 nm Ta top metal contact was deposited via DC magnetron sputtering, followed by photolithography patterning. The surface area of the fabricated HfO 2 memristor was 5 Â 5 μm 2 . The Al 0.7 InAsSb APDs were grown on a n-type GaSb substrate (1À9 Â 10 17 cm À3 ) with the following epitaxial structure: a 300 nm n-type www.advancedsciencenews.com www.advintellsyst.com (2 Â 10 18 cm À3 ) GaSb buffer layer, a 300 nm n-type (2 Â 10 18 cm À3 ) Al 0.7 InAsSb contact layer, a 1500 nm Al 0.7 InAsSb unintentionally doped (UID) layer, and a 100 nm p-type (2 Â 10 18 cm À3 ) Al 0.7 InAsSb contact layer. The epitaxial growth was capped with a 50 nm p-type (1 Â 10 19 cm À3 ) GaSb layer. Be and Te were used as the p-and n-type dopants, respectively. The top-illuminated PIN APDs were defined using standard photolithography techniques. The mesas were etched into the n-contact layer with a citric/phosphoric acid solution prior to a 12 nm Ti and 85 nm Au contact deposition via electron beam evaporation. Electrical and Optical Characterization: Electrical performance of the memristors was characterized using a KEYSIGHT B1500A Semiconductor Device Analyzer equipped with a waveform generator/fast measurement unit and a pulse generator (KEYSIGHT 33600A Series). To measure resistive STDP and APD current, an oscilloscope (KEYSIGHT DSO-X 3024T), and two current amplifiers (Edmund 59-179 and EOC DLPCA-200) were used. For the optical programming via the current amplifiers, low-noise and 10 Hz mode with 10 6 gain settings were used. A continuous measurement mode was adopted for the characterization of I-V performance with 1 mA compliance current. The voltage range and width used for the analog conductance switching of the memristors experiment was from 0.5 to 1 V and 10 μs, respectively. The conductance was calculated using Ohm's law as I read ¼ G Â V read , and the transitions of the conductance were calculated as ΔG ¼ ðG final ÀG init Þ G init Â 100ð%Þ at 0.1 V read voltage. The initial resistance of memristors in the R-ToF system was %100 kΩ. A FOSTEC A20500 lamp was adopted as a light source. A Hamamatsu S9073 APD and a picosecond laser (Alphalas PLDD-50M) were employed for R-ToF ranging measurement with 100 V reverse bias applied to the APD and low-noise 10 3 gain of the TIA. The physical channel lengths of the presynaptic and postsynaptic spikes were matched to minimize the parasitic time delay. The peak programming voltage was limited to 2 V to protect the device from exceeding the transition ranging of the resistance. The remainder time delay generated by parasitic capacitance and inductance of the external cables and wire connections was compensated empirically by modulating the pulse timing via MATLAB. For the 1D R-ToF measurement, a commercially retro-reflective (metal coated, 0.9 reflectivity) object was adopted. A photograph of the experiment setup is shown in Figure 15, Supporting Information. In the dark room condition, 150 V reverse bias was applied to the APD to maximize the multiplicatively reflected optical signals. For the 1D R-ToF measurement, the picosecond laser was set to 203 and 750 mW peak power for short and long range, respectively. Amplitude (0.5 V) and pulse width (20 ns) are used for the presynaptic spikes. Each position was measured 5 times.
R-ToF 3D Imaging: We measured distances of a miniaturized truck at 15 cm, a scaled building (EatingBiting DIY 1:160 Outland Models) at 25 cm, and a wall at 45 cm. The target scene is shifted for the imaging to enable a point-by-point LiDAR scanning method (16 Â 8 pixels), and a scattering film is attached to the ranging spot. Due to the scattering effect that represents the realistic 3D ranging environment, different bias voltages are applied to the APD (15 cm for À140 V, 25 cm for À150 V, and 45 cm for À155 V).
3D Image Reconstruction Simulation: We used the Monte Carlo simulation using Brian2 spiking neural network (SNN) simulator in the Python language to demonstrate the STDP-based array-scale depth reconstruction. [41] Based on the R-ToF principle, normalized transition of weights (converted from resistance of memristors) corresponds to intensity of each pixel. The pixel dimension of the input image was 48 Â 48, and the intensity of each pixel can be converted to the depth information (from 15 to 80 cm). An optical acquisition variation (σ acq ¼ 0.03414) was extracted from the result of Figure 3e and was applied to the converted depth matrix; 2304 synapses with 4608 neurons were imposed to represent STDP of the pixelwise depth information. The resistance window (4À30 kΩ) of the fabricated memristors was normalized to represent weight values used in this simulation. The optical acquisition deviation including horizontal and vertical dimensions was approximated in the experimental data with three-sigma deviation. Additional device-to-device variation (uniformly distributed random numbers in the normalized initial resistance window) and update-to-update variation (σ U ¼ 0.04123Þ with respect to the memristors were included in the simulation. The update-to-update variation parameter was extracted from the result of Figure 2f with the normalized resistance window. STDP in the simulator was parameterized by fitting the exponential curve of the experimental data ( Figure 2f ) with three-sigma deviation by the equation where W is a weight matrix, and A pre and τ pre are exponential fitting parameters. The obtained parameters were used in differential equations to represent neural spiking responses by equations where a pre and a post are tracing variables for presynaptic and postsynaptic activity parameters, respectively. As the depth information of the time difference between the presynaptic spike and the postsynaptic spike is always positive, the presynaptic trace is updated as follows a pre ! a pre þ A pre (4) w ! w þ a post þ Gaussianð0, σ U Þ The Gaussianð0, σ U Þ denotes the 1D Gaussian distribution with zero mean and update-to-update deviation σ U ¼ 0.1. MSE is calculated by the equation where m and n are pixel dimension (m ¼ n ¼ 48), k is the current epoch, and W ideal is the ideal weight matrix. The final PSNR is calculated as follows PSNR ðdBÞ ¼ 20log 10 Each epoch was interpreted as 4 μs acquisition time (corresponding to 250 kHz) referred by the 0.7 μs excitatory and inhibitory times of the R-ToF system including an assumed 0.7 μs delay time from peripheral circuits.
Reconstructed Image Classifier Simulation: We trained a fully-connectedlayer classifier with four hidden layers (784-256-128-100-10) on the Fashion-MNIST dataset (F-MNIST, Modified National Institute of Standards and Technology database). [42] The depth acquisition was assumed to be rasterized and interpreted as intensities of pixels (between 0 and 1 intensity). Each F-MNIST image contained 28 Â 28 pixels, the initial time-decaying learning rate was 0.03, the decay was 0.0015, and rectifier (a softmax for the last output) nonlinearity activations were used. We trained and tested 10 000 (with 100 batch size) and 100 images, respectively. The intensity of the reconstructed test images was inverted, normalized, and rescaled. To compare the classification between the hardware-and software-based models, we exploited the physical parameters of our fabricated HfO 2 memristors on the ANN, as well as used the 100 reconstructed test images from the R-ToF image reconstruction simulation. Specifically, the conversion from conductance to weight values in the ANN was based on the interpolation of the results of Figure 2d, and the cycle-to-cycle weight variation (σ C ¼ 0.00048Þ was achieved from Figure 2e. We assumed that the HfO 2 -based ANN incorporates a crossbar structure with perfect access to each memristor cell and adopted 8-bit analog-to-digital converters (ADCs) to compute the quantized activation value of each column of the crossbar array.