CMOS image sensor for wide dynamic range feature extraction in machine vision

This letter presents a wide dynamic range (WDR) feature extraction (FE) readout scheme for machine vision applications using CMOS image sensors (CISs). The proposed scheme with the proposed pixel struc- ture has two operating modes, the normal and WDR modes. In the normal operating mode, the proposed CIS captures a normal image with high sensitivity. In addition, as a unique function, a bi-level image is obtained for real-time FE even if a pixel is saturated in strong illumina- tion conditions. Thus, compared to typical CISs for machine vison, the proposed CIS can reveal object features that are blocked by light in real time. In the WDR operating mode, the proposed CIS produces a WDR image with its corresponding bi-level image. A prototype CIS was fab- ricated using a standard 0.35- μ m 2P4M CMOS process with a 320 × 240 format (QVGA) with 10- μ m pitch pixels. At 60 fps, the measured power consumption was 5.98 mW at 3.3 V for pixel readout and 2.8 V for readout circuitry. The dynamic range of 73.1 dB was achieved in the WDR operating mode.

Introduction: CMOS image sensors (CISs) have been used in a wide variety of applications, including mobile devices and computer visionbased intelligent systems [1]. The field of machine vision systems has rapidly grown in recent years. As a result, object detection and recognition using CISs have become important functions for extracting the features of a target object. In addition, consumer demand for real-time feature extraction (FE) is increasing for applications such as the Internet of Things and augmented reality.
FE from a target object is mainly based on obtaining a bi-level image from a human-friendly image. Therefore, the accuracy of the FE is closely related to CIS performance in terms of dynamic range, sensitivity, and frame rate. For example, when there is a weak contrast between an object and its surrounding environment, it becomes difficult to identify object features owing to the limited sensitivity of typical CISs. In addition, when an object is illuminated by strong light that is out of its dynamic range, the object is shown as covered by light. That is, light intensity information is lost because pixels are saturated. Thus, in machine vision, high sensitivity and a wide dynamic range (WDR) are essential for accurate FE. Furthermore, considering the operating latency in digital image processing for FE, it is desirable to simultaneously obtain a human-friendly image with its bi-level image for real-time FE.
Until now, various studies have achieved high sensitivity and WDR performance for mobile applications [2,3]. Based on these studies, we propose a WDR-FE readout scheme for a machine vision CIS. The proposed CIS has two operating modes, the normal and WDR modes, and bi-level images are obtained for real-time FE. In the normal operating mode, the proposed CIS simultaneously captures an image with high sensitivity and extracts its bi-level image. In addition, as a unique function, it reveals the features of a target object even when the image is saturated by strong light. In the WDR operating mode, the dynamic range of the proposed CIS is extended, and a WDR image is obtained along with its bi-level image.
Structure and operating principles: The simplified pixel layout and equivalent proposed pixel structure schematic are shown in Figure 1. The proposed pixel structure is designed based on a conventional 3T-APS structure [4], which is composed of a high-sensitivity photosensing region on the odd side (PD O ) and a low-sensitivity photosensing region on the even side (PD E ). The ratio of the area of each sub-region and the entire photosensing region is approximately 0.5. The sensitivity of PD E is reduced by shielding its photosensing region with metal layers. This reduces the amount of light that illuminates the PD E photosensing region. Note that the sensitivity (S) of a pixel is given by S = A pixel · FF · (λ · η)/ (1.24 · C PD ) [5], where A pixel is the pixel area, FF is the fill factor, λ is the wavelength, η is the quantum efficiency, and C PD is the integration capacitance of the photodiode. A photogate is located in the middle of PD O and PD E , and it can be electrically connected to C PDO and C PDE through the channel formed under the photogate with a positive bias voltage (V PG ). C PDO and C PDE represent the integration capacitance of PD O and PD E , respectively. Here, a negative bias voltage is used for electrically disconnecting C PDO and C PDE . Figure 2 shows the simplified schematic of the proposed singlecolumn single-slope (SS) ADC. It consists of a conventional SS ADC and an inverter-based bi-level detector. Every readout column shares a ramp generator based on a current digital-to-analogue converter (I-DAC). There are two outputs in the proposed pixel structure, that is, V PXO from PD O and V PXE from PD E . These outputs are selectively read out through an analogue multiplexer according to the operating mode. When pixel readout begins, the bi-level detector starts to observe whether the pixel output (V PX ) exceeds the predetermined reference (V REF − V th ) or not. Here, V th is a threshold voltage of M P in the bi-level detector.
The basic concept of the proposed WDR-FE readout scheme is to obtain the feature information of a target object even when an image is saturated. In other words, V PXO is used for normal imaging with high sensitivity, and V PXE is used for representing object features that are out of the dynamic range. This results in a WDR-FE. When pixel readout begins in the normal mode, the SS ADC first digitises V PXO and the bilevel detector simultaneously extracts bi-level information for V PXO . If When pixel readout begins in the WDR mode, a positive voltage V PG is applied to the photogate for electrically connecting C PDO and C PDE ; this gives C PD = C PDO + C PG + C PDE . Hence, the proposed pixel structure exhibits the dynamic characteristics of (Q PDO + Q PDE )/C PD , which can be expressed as V PXA = ((Q PDO + Q PDE )/2)/C PDO (C PDO ≈ C PDE C PG ). This is equivalent to averaging V PXO and V PXE (i.e., V PXA ). Hence, the noise performance and dynamic range of the pixel structure are improved and WDR imaging is achieved. Figure 3 shows the operation timing diagram and its waveform in two operating modes to provide an explicit explanation of the proposed readout scheme. In the normal mode (Figure 3(a)), when SX becomes high, the analogue correlated double sampling operation for V PXO is performed at node V IN of the first comparator. Note that V PXO is selected by default ( SO is high). When SEN is high, V FLAG changes to 'logic high' because V PXO exceeds V REF − V th . Its result (D E0 ) '1' is stored in the additional latch (2-bit latch) as the FE result with high sensitivity. Then, the SS A/D conversion is performed when CEN is high. If D OUT is equal to D FULL , then SE becomes high for obtaining additional information from the pixel represented as D FULL . The bi-level detector checks V PXE when SEN is high. If V PXE exceeds V REF − V th , V FLAG changes to 'logic high', and its result (D E1 ) is stored as the FE result with low sensitivity. If D E1 is '1', it implies that there is an object blocked by light. D E1 is utilised as the object feature information under the condition of pixel saturation, resulting in WDR-FE. Note that the bi-level image is synthesised by adding D E0 and D E1 in off-chip digital processing. Finally, AZ and RX become high for sampling the pixel reset value in the same manner as the conventional CIS with 3T-APS. In contrast, in the WDR mode (Figure 3(b)), PG is high during the pixel readout period. Then,

Fig 6 Measured linearity of the ADC. (a) Differential non-linearity, (b) integral non-linearity
the A/D operation for V PXA is performed in the same manner as the normal mode. Therefore, in the normal mode, WDR-FE is performed while maintaining high sensitivity. In addition, in the WDR mode, noise performance and the dynamic range are improved by the proposed pixel structure.
Experimental results: The prototype chip was implemented using a standard 0.35-μm CMOS process. The microphotograph of the chip is shown in Figure 4. The size of the chip is 4 × 4 mm. It consists of a 320 × 240 format (QVGA) with 10-μm pitch pixels and a 10-bit column-parallel SS ADC array with the ramp generator, including control circuitry and I/O pads. The fill factor of the proposed pixel structure is 35.8% on the odd side (PD O ) and 8.4% on the even side (PD E ). Additional test channels are added to the prototype chip to evaluate the performances of the proposed pixel structure and single-column SS ADC. The performance of the prototype CIS is verified in various test conditions by applying a control voltage (V REF ) through an external DAC mounted on the evaluation board. Note that V REF can be changed according to the desired FE condition. A negative voltage of 0.5 V and a positive voltage of 4 V are applied for V PG in the proposed pixel structure. Figure 5 shows the measured response of the proposed pixel structure according to light intensity. The linearity of the proposed pixel structure was verified by performing the measurement in two operating modes, that is, PD O and PD E in the normal mode and PD SUM in the WDR mode. The results demonstrate that the proposed CIS provides a dynamic range of 73.1 dB in the WDR mode while maintaining linearity for light intensity.
The measured maximum differential non-linearity and integral nonlinearity of the proposed SS ADC are 0.87 and 1.24 LSB, respectively, as shown in Figure 6. Table 1 summarises the performance of the prototype CIS. The power consumption was 5.98 mW at 60 fps. Off-chip digital offset adjustment was performed to reduce fixed-pattern noise as described in [6].
Sample images in the two operating modes were captured by the prototype CIS when the left side of the object was illuminated by strong light. The images are shown in Figure 7. In the normal mode with V REF at approximately 0.7 V (Figure 7(a)), the left side of the object is indistinguishable because the image is saturated. Real-time bi-level images are captured with and without the proposed WDR-FE readout scheme. With the proposed readout scheme, the bi-level image reveals perceptible object features even outside the dynamic range of the prototype CIS. In the WDR mode with V REF at approximately 0.55 V (Figure 7(b)), the left side of the object is clearly visible compared to the normal imaging mode because the dynamic range is extended. Table 2 presents a performance comparison of the proposed CIS with recently published works for obtaining WDR. Compared with other works, the proposed CIS can extract object features in a wide range of light conditions, which represents a unique function. Conclusion: In this letter, a WDR-FE readout scheme is introduced for machine vision CISs. The proposed CIS has two operating modes, that is, the normal and WDR modes, and bi-level images are extracted in real time. In the normal mode, the proposed CIS captures the normal image with high sensitivity. Additionally, as a unique function, the bilevel image is extracted even when the pixel is saturated. In the WDR mode, the proposed CIS obtains the WDR image with its bi-level image. From a commercial perspective, the proposed scheme could be utilised in machine vision applications to track the movement of a target object under strong light intensity.