Artificial Visual Electronics for Closed‐Loop Sensation/Action Systems

Artificial visual electronics that mimic the structure and function of human eyes can be a powerful tool to provide visual feedback in the closed‐loop sensation/action systems, which can be beneficial to achieve sophisticated functions in a precise and efficient way. Herein, how artificial visual electronics work in the closed‐loop sensation/action systems, mimicking the human eyes for human behaviors, followed by how artificial visual electronics are utilized in various fields are introduced. To fully mimic the human eyes, how to achieve the structural similarity of artificial visual electronics with eyeballs is highlighted, and focused on the key component, i.e., retina‐like 3D light‐detecting imagers. When combined with the machine‐learning method, such retina‐like 3D imagers are expected to significantly benefit the closed‐loop sensation/action systems.


Introduction
Sophisticated human behaviors can proceed naturally and smoothly because the motor systems work in coordination with the sensory systems, where closed-loop feedback continuously and efficiently happens among these systems in real time. [1] Among all the sensory systems through which brains obtain information from the environment, the eye is the predominant channel for most animals on this planet, including humans who obtain %90% of information via eyes. [2] In addition, the eye is a special organ because the retina is sensitive to light, meaning that it can deliver electrical potential to nerves under the light stimulus, whereas no other organs or nerves without modified genes have the light-detecting capability as far as we know. [3,4] Mimicking the light-detecting capability of eyes in electronics enabled the invention of artificial visual electronics. Such systems have been utilized in various commercial products, specifically referring to the image-detecting systems for smartphones, cameras, video recorders, autodriving vehicles, and security systems, all of which are indispensable to reshape the lifestyle and society in the Third Industrial Revolution. In the next industrial revolution for the new age of digitization and "smarter" networking of production systems, the closed-loop sensation/action systems will play an important role. [5,6] Considering the importance of eyes for humans, we believe that artificial visual electronics should be one of the priority subjects to be investigated in the closed-loop sensation/action systems.
In this perspective, we first discuss the importance of visual feedback in the closed-loop sensation/action systems for controlling human behaviors. Then, we summarize how artificial visual electronics have benefited society in the closed-loop sensation/ action systems. Furthermore, we introduce artificial visual electronics that fully mimic the structure and function of human eyes. Since the light-detecting elements of the retina locate at a 3D eyeball, we mainly focus on reviewing the fabrication techniques that can be utilized to build the retina-like 3D imagers. Finally, we discuss other promising techniques for artificial visual electronics, and the importance of combining the machine-learning method and artificial visual electronics in the future.

Closed-Loop Sensation/Action Systems with Visual Feedback
Humans rely on the cooperation of the sensory and motor system to conduct physical activities in a closed-loop configuration ( Figure 1a). [7,8] For instance, when a human tries to shake hands with another, the receptors on the hand first detect signals once he/she touches the hand of another human through both tactile perception and temperature perception. Then, the detected signals are delivered to the central nervous system via the afferent nerve, and a decision is made and sent to the motor system, which leads to the proceeding of the handshaking. To make sure that the handshaking happens correctly, visual feedback always happens in this process. The human instinctively checks the hand that he/she is holding via eyes, and sends visual feedback to the brain to trigger an internal decision whether to shake the DOI: 10.1002/aisy.202100071 Artificial visual electronics that mimic the structure and function of human eyes can be a powerful tool to provide visual feedback in the closed-loop sensation/ action systems, which can be beneficial to achieve sophisticated functions in a precise and efficient way. Herein, how artificial visual electronics work in the closed-loop sensation/action systems, mimicking the human eyes for human behaviors, followed by how artificial visual electronics are utilized in various fields are introduced. To fully mimic the human eyes, how to achieve the structural similarity of artificial visual electronics with eyeballs is highlighted, and focused on the key component, i.e., retina-like 3D light-detecting imagers. When combined with the machine-learning method, such retina-like 3D imagers are expected to significantly benefit the closed-loop sensation/action systems.
hand. This visual feedback is extremely important to form the closed-loop configuration for precisely controlling human behaviors.
Mimicking the natural closed-loop sensation/action systems is a highly promising strategy for artificial systems to precisely and efficiently achieve complicated functions. Such closed-loop concept has been widely utilized in various fields, such as drug delivery, [9] neural electronics, [10][11][12] human-machine interface, [13][14][15] optogenetics, [16] and so on. Adding artificial visual electronics into the closed-loop sensation/action systems will be even more powerful since it is the most direct way to gather information from the surrounding environment. In fact, there are already some reports that utilized artificial visual electronics in the closed-loop sensation/action systems. The ARGUS II device (Figure 1b), an implantable artificial visual electronic device to replace the human retina, was developed to help patients suffered from hereditary retinal diseases to fight blindness. [17] With the ARGUS II device, patients have showed an enhanced performance in the orientation and mobility tasks, including object localization, motion discrimination, and discrimination  [17] Copyright 2013, American Association for the Advancement of Science), c) as photodetectors to give feedbacks to control the light intensity of the LEDs, to study animals' behavior when light stimulates gene-modified nerves (Reproduced with permission. [18] Copyright 2013, American Association for the Advancement of Science), d) as an artificial vision receptor in the bimodal artificial sensory neuron that can control the movement of a robot hand more precisely than a unimodal one. (Reproduced with permission. [19] Copyright 2020, Springer Nature), and e) as biosignal sensors to continuously monitor photoplethysmogram (PPG) signals from human body in real time (Reproduced with permission. [21] Copyright 2018, National Academy of Sciences).
www.advancedsciencenews.com www.advintellsyst.com of oriented gratings. The restored vision, together with other sensations such as audition and tactility, improved the outdoor mobility of patients, enabling them walk along pedestrian courses and identification of doors and posts. Another interesting application is the implantable optoelectronic system, which was developed to accelerate basic scientific discoveries in the field of optogenetics and their translation into clinical technologies ( Figure 1c). [18] In this typical system, the temperature sensor and inorganic photodiode (PD, as the artificial visual electronic device) can provide information on the heat generation and light intensity, which are treated as two different signals. These two signals were directly related to the irradiation intensity of the micro light-emitting diode (LED), which was utilized to stimulate the gene-modified nerves of the mouse to control its leg movement. Altering light stimulation based on feedback from the temperature sensor and the PD has great potential for a closed-loop operation and precise control of the stimulation without any side effects caused by the temperature increase. Closed-loop sensation/action systems utilizing artificial visual system also benefit other applications, such as precisely decoding human gestures to control a robot hand (Figure 1d), [19,20] and long-term human health monitoring systems ( Figure 1e). [21] 3

. Artificial Visual Electronics
The human eye is the most delicate natural optical instrument which endows us the capability to perceive the wonderful world of light. Inspired by the superior performance of the human eye, the biomimetic design emerges as an attractive strategy for developing artificial visual electronics. To exactly mimic human eyes, we need to first know the anatomic structure and understand how they detect signals. As shown in Figure 2a, [22] the eye has a spherical shape, where the lens locates at the front hemisphere to focus light that passes through and the retina locates at the back hemisphere to detect the information that light brings. [23] The detected light signals are transformed into electrical potentials, which then are delivered to the brain by the nerves. As for the uniqueness of the structure of human eyes, the hemisphere where the photoreceptors on the retina locate can benefit to significantly reduce the complexity of optical systems. With the 3D shape, human eyes can directly compensate for the aberration from the curved focal plane, whereas no commercial 2D imagers can do. [24] In addition, the entire retina of an adult human contains 5-7 million cones on an area of about 2.7 cm 2 (pixel resolution of 38.6-54.0 μm 2 ), which are responsive to light with a wavelength ranging from 380 to 760 nm. [25,26] Due to the novel imaging system, human eyes can achieve exceptional characteristics, including a wide field of view (FOV) of 150-160 , a high resolution of 1 arcmin per line pair at the fovea, dynamic and fast imaging (30-60 Hz), and capability to distinguish different colors. Human eye-inspired artificial visual electronics on the plane substrates have already been developed and commercialized. However, artificial visual electronics on a 3D layout configuration is still in the infant stage. Recently, Gu et al. reported an advanced 3D artificial visual electronic device, fully mimicking the structure of the human eyes. [22] In Figure 2b,c, the perovskite nanowires locating on a hemisphere serve as the light-detecting working electrodes, mimicking the retina, and the tungsten (W) film on an aluminum (Al) hemispherical shell worked as the counter electrode. The ionic liquid was used to fill in the cavity between the above two hemisphere shells, serving as the electrolyte that mimics the vitreous humor in the human eye. The flexible eutectic gallium indium liquid-metal wires in soft rubber tubes were used for signal transmission, mimicking the nerves connected to the brain. Due to the novel design, compared with the conventional 2D image systems, this artificial visual electronic device with a 3D hemispherical configuration ensured a more consistent distance between pixels and lens, and achieved a wider FOV and better focusing onto each pixel (Figure 2d,e). Even though the pixel resolution was about 200 μm 2 by utilizing microneedle contacts with 2 mm distance between microneedles, this artificial visual electronic device has a promising potential to achieve high imaging resolution (0.22 μm 2 ) when individual nanowires are electrically addressed. However, it remains challenging to achieve an individual connection between the nanoscale structure and the microscale wire on a 3D hemisphere shell without decreasing the resolution. More works in the future need to be done to develop such artificial visual electronics with high resolution, fully mimicking the structure and function of the eyes.

Retina-like Imagers
In the sophisticated structure of the eyes, the most critical part is the light-detecting retina that locates on the hemisphere. Therefore, to mimic the function and structure of the human eyes, an easier way is to fabricate the retina-like imagers, which is the most critical part to build the advanced 3D artificial visual electronics. Conventional inorganic imagers are constructed on the 2D rigid substrates, which cannot deform to adjust themselves into a retina-like 3D shape due to the brittle nature of these rigid materials. Because the fabrication for the inorganic imagers is very complicated, it is also extremely difficult to directly fabricate the devices on a retina-like 3D hemispherical shell. Until now, only several groups reported the fabrication of the retinalike 3D imagers using structure engineering, a commonly utilized approach to minimize the strain on the devices under mechanical deformations. As for the fabrication procedures, imagers are first constructed on the 2D planar substrates, followed by deforming them into a retina-like 3D shape. The important and challenging task is to minimize the strain on the rigid devices during the deforming process. Ko et al. reported the first retina-like 3D imager by connecting each light-detecting pixel in the arrays of devices with compressive metal connections (Figure 3a). [27] After fabricating the imager on the 2D substrate, they transferred the imager onto a retina-like 3D hemispherical shell. During the transfer process, the mechanical strains were released by forming a buckled structure on the metal connections whereas the light-detecting devices in the imager did not deform and thus kept functionality. As-fabricated retinalike 3D imagers were consisted of 16-by-16 light-detecting pixels with a resolution of 860 Â 860 μm 2 and a curvature radius of %10 mm. Song et al. [28] and Kim et al. [29] further improved the maximum strain tolerance on the metal connection by utilizing a serpentine structure (Figure 3b,c). With a higher strain tolerance, it increased the density (270 Â 190 μm 2 for the study by Other methods were then developed to significantly minimize the tensile/compressive strains that happened to the metal connections. Inspired by the traditional origami art, Zhang et al. assembled five pieces of flexible light-detecting imagers onto a retina-like 3D substrate (Figure 3d). [30] Because all the lightdetecting pixels in the imagers just experienced a bending deformation, there were very small strains exerted on both the connections and light-detecting devices. Utilizing this strategy, it is possible to greatly improve the density and number of the light-detecting pixels in the retina-like 3D imager since no additional design is needed on the metal connections. However, the accurate resolution and number of pixels were not reported.
Another method to attach the flexible 2D light-detecting imager onto a retina-like 3D substrate was achieved by significantly reducing the thickness of the device. Wu et al. reported an ultrathin and conformable perovskite-based light-detecting imager with a total thickness of 2.4 μm (Figure 3e). [31] Combined with the vacuum-assisted drop-casting patterning process, they were able to reduce the size of the light-detecting pixel as small as 50 Â 50 μm 2 . Due to the ultrathin thickness, the ultraflexible imager can be conformally wrapped around a walnut that had a similar shape to the eyeball. However, in the arrays of the Figure 2. Artificial visual electronics, mimicking human eyes. a) Schematic showing the structure of the human eye, which is the organ for human to detect light signals and then transfer the potential to the nerves and the brain. b) The detailed structure of the hemispherical biomimetic electrochemical eye (EC-EYE). c) The optical image of the of the EC-EYE. d) The schematic FOV of the planar and hemispherical image-sensing systems. e) The reconstructed image (letter "A") of EC-EYE and its projection on a flat plane. a-e) Reproduced with permission. [22] Copyright 2020, Springer Nature.
www.advancedsciencenews.com www.advintellsyst.com device, cross talk could happen because there was no switching device in the circuit. In addition, for both of the aforementioned two approaches, since the imagers were not stretched or compressed on the 3D surface, there was inevitably space between the retina-like substrate and part of the devices in the imagers. [32] However, how this space issue will influence the image quality recorded by these retina-like imagers has never been discussed.

Future Perspectives
For the fabrication of the retina-like 3D imager, compared with the structure engineering, fabricating an imager with intrinsic stretchability can be another approach. Although the intrinsically stretchable organic field-effect transistors (OFETs) and circuits have been achieved recently, [33] there has not been any group reporting that such circuits can be combined with light-detecting devices, but it is possible when considering the fast development of intrinsically stretchable OFETs. Another way is to directly fabricate the light-detecting imagers on the retina-like 3D substrates by 3D/4D printing techniques. [34][35][36] In addition, compared with the human retina, all reported retina-like 3D imagers showed a lower resolution and fewer pixels and they did not show the capability to distinguish different colors, which means that improving these parameters will be important tasks in the future. In addition, there is a demand on how to use the signals detected by artificial visual electronics to guide the action of the closed-loop sensation/action systems. Recently, Wang et al. reported the fusion of multisensory and a machine-learning method for the closed-loop sensation/action systems in Figure 4. [20] For the fusion of multisensory (Figure 4a), they integrated visual data captured by a camera (Figure 4b) with somatosensory data from skin-like stretchable strain sensors. By utilizing the machine-learning method, they were able to recognize the human gestures with an accuracy of 100% even in nonideal conditions where images are noisy and under or overexposed (Figure 4c). The detected gestures were then utilized to guide a robot by assigning motor commands to different gestures. Due to the machine-learning method, they guided the robot pass through the labyrinth with zero error whereas there were six errors without using the machine-learning method (Figure 4d,e). In addition, different machine-learning methods imager, which is realized by an origami approach. e) Optical image (left) and structure illustration of a hemispherical imager, which can be seamlessly attached onto a walnut that has a similar shape with an eyeball. a) Reproduced with permission. [27] Copyright 2008, Springer Nature. b) Reproduced with permission. [29] Copyright 2020, Springer Nature. c) Reproduced with permission. [28] Copyright 2013, Springer Nature. d) Reproduced with permission. [30] Copyright 2017, Springer Nature. e) Reproduced with permission. [31] Copy right 2021, Wiley-VCH.
www.advancedsciencenews.com www.advintellsyst.com have been developed for the processing of visual data via various perception and reasoning algorithms, including support vector machine, [37] K-nearest-neighbor, [38] convolutional neural networks, [39] and artificial neural network. [20] Therefore, it can be expected that combining the machine-learning method with the artificial visual electronics can further benefit the closed-loop sensation/action systems in the future.

Conclusion
In this perspective, we introduced the significance of advanced artificial visual electronics for the closed-loop sensation/action systems. To fully mimic the human eyes, artificial visual electronics with a 3D retina-like configuration was highlighted.
Furthermore, we summarized various reported methods to construct 3D retina-like imagers, and proposed other promising fabrication techniques in the future for the application of such imagers in the closed-loop sensation/action systems. Finally, we proposed to combine the machine-learning method and artificial visual electronics in the closed-loop sensation/action systems. showing the closed-loop sensation/action system consisting of a data acquisition unit-somatosensory, a camera for capturing visual images, a computer, a wireless data transmission module, and a quadruped robot. c) Pictures of ten categories (I to X) of hand gestures that were assigned with specific motor commands for the movement guidance of the quadruped robot. FM, forward move; BM, back move. d) Scenarios of the robot walking through the labyrinth based on visual-based recognition and e) bioinspired somatosensory-visual-associated learning recognition. Reproduced with permission. [20] Copyright 2020, Springer Nature.
www.advancedsciencenews.com www.advintellsyst.com Xiaodong Chen is the president's chair professor of materials science and engineering, professor of chemistry and medicine (by courtesy) at the Nanyang Technological University (NTU), Singapore. He serves as director of Innovative Center for Flexible Devices (iFLEX) and director of Max Planck-NTU Joint Lab for Artificial Senses at NTU. His research interests include mechano-materials and devices, the integrated nano-bio interface, and cyber-human interfaces.