Dominique Ginhac

Jérôme Dubois

Michel Paindavoine

Barthélémy Heyrman

A SIMD Programmable Vision Chip with High Speed Focal Plane Image Processing

Keywords: CMOS Image Sensor, Parallel architecture, SIMD, High-speed image processing, Analog arithmetic unit

A high speed analog VLSI image acquisition and low-level image processing system is presented. The architecture of the chip is based on a dynamically reconfigurable SIMD processor array. The chip features a massively parallel architecture enabling the computation of programmable mask-based image processing in each pixel. Extraction of spatial gradients and convolutions such as Sobel operators are implemented on the circuit. Each pixel include a photodiode, an amplifier, two storage capacitors, and an analog arithmetic unit based on a four-quadrant multiplier architecture. A 64 × 64 pixel proofof-concept chip was fabricated in a 0.35 µm standard CMOS process, with a pixel size of 35 µm × 35 µm. A dedicated embedded platform including FPGA and ADCs has also been designed to evaluate the vision chip. The chip can capture raw images up to 10 000 frames per second and runs low-level image processing at a framerate of 2 000 to 5 000 frames per second.

I. INTRODUCTION

T ODAY, digital cameras are rapidly becoming ubiquitous, due to reduced costs and increasing demands of multimedia applications. Improvements in the growing digital imaging world continue to be made with two main image sensor technologies: charge coupled devices (CCD) and CMOS sensors. Historically, CCDs have been the dominant image-sensor technology. However, the continuous advances in CMOS technology for processors and DRAMs have made CMOS sensor arrays a viable alternative to the popular CCD sensors. This led to the adoption of CMOS image sensors in several highvolume products, such as webcams, mobile phones, PDAs for example. Furthermore, new recent technologies provide the ability to integrate complete CMOS imaging systems at focal plane, with analog to digital conversion, memory and processing [START_REF] Fossum | Active pixel sensors: Are CCDs dinosaurs?[END_REF]- [START_REF] Litwiller | CCD vs. CMOS: Facts and Fiction[END_REF]. By exploiting these advantages, innovative CMOS sensors have been developed and have demonstrated fabrication cost reduction, low power consumption, and size reduction of the camera [START_REF] Aw | A 128128-pixel standard-cmos image sensor with electronic shutter[END_REF]- [START_REF] Smith | A singlechip 306x244-pixel CMOS NTSC video camera[END_REF].

The main advantage of CMOS image sensors is the flexibility to integrate processing down to the pixel level. As CMOS image sensors technologies scale to 0.18 µm processes and under, processing units can be realized at chip level (system-on-chip approach), at column level by dedicating processing elements to one or more columns, or ar pixellevel by integrating a specific unit in each pixel or local of neighboring pixels. Most of the researches deals with chip Manuscript submitted March 1, 2008;revised September 2008 The authors are with the LE2I Laboratory, Burgundy University, 21078 Dijon, France (email: dginhac@u-bourgogne.fr). and column-level [START_REF] Yadid-Pecht | In-Pixel Autoexposure CMOS APS[END_REF]- [START_REF] Sakakibara | A High-Sensitivity CMOS Image Sensor With Gain-Adaptative Column Amplifiers[END_REF]. Indeed, pixel-level processing is generally dismissed because pixel sizes are often too large to be of practical use. However, as CMOS scales, integrating a processing element at each pixel or group of neighboring pixels becomes more feasible since the area occupied by the pixel transistors decreases, leading to an acceptable small pixel size. A fundamental tradeoff must be made between three dependent and correlated variables: pixel size, processing element area, and fill-factor. This implies various points of view:

1) for a fixed fill-factor and a given processing element area, the pixel size is reduced with technology improvements. As a consequence, reducing pixel size increases spatial resolution for a fixed sensor die size. 2) for a fixed pixel size and a given processing element area, the photodiode area and the fill-factor increase as technology scales since the area occupied by the pixel transistors in each processing element decreases. It results in better sensibility, higher dynamic range and signal-to-noise ratio. 3) for a fixed pixel size and a given fill-factor, the processing element can integrate more functionalities since the transistors require less area as technology scales.

Consequently, the image processing capabilities of the sensor increase.

In summary, each new technology process offers 1) to integrate more processing functions in a given silicon area, or 2) to integrate the same functionalities in a smaller silicon area. This can benefit the quality of imaging in terms of resolution, noise for example by integrating specific processing functions such as correlated double sampling [START_REF] Nixon | 128 x 128 CMOS photodiode-type active pixel sensor with on-chip timing, control, and signal chain electronics[END_REF], anti blooming [START_REF] Wuu | A high performance active pixel sensor with 0.18um cmos color imager technology[END_REF], high dynamic range [START_REF] Decker | A 256256 cmos imaging array with wide dynamic range pixels and column-parallel digital output[END_REF], and even all basic camera functions (color processing functions, color correction, white balance adjustment, gamma correction) onto the same camera-on-chip [START_REF] Yoon | Single-chip cmos image sensor for mobile applications[END_REF]. Furthermore, employing a processing element per pixel offers the ability to exploit the high speed imaging capabilities of the CMOS technology by achieving massively parallel computations [START_REF] Krymsky | A 9-V/Lux 5000-Frames/s 512 x 512 CMOS Sensor[END_REF]- [START_REF] Sugiyama | A High-Speed CMOS Image With Profile Data Acquiring Function[END_REF].

In this paper, we discuss hardware implementation issues of a high speed CMOS imaging system embedding low-level image processing. For this purpose, we designed, fabricated, and tested a proof-of-concept 64 × 64 pixel CMOS analog sensor with per-pixel programmable processing element in a standard 0.35 µm double-poly quadruple-metal CMOS technology.

The rest of the paper is organized as follows. The Section II is dedicated to the description of the high speed algorithms embedded at pixel-level. The Section III is a general description Fig. 1. Overview of the image sensor with a processor-per-pixel array of the characteristics of the sensor. These characteristics are well detailed in the Section IV, which talks about the design of the circuit, with a full description of the main components such as the photodiode structure, the embedded analog memories, and the arithmetic unit are successively described. In the section V, we describe the test hardware platform and the chip characterization results, including an analysis of the fixed pattern noise. Finally, some experimental results of high speed image acquisition with pixel-level processing are provided in the last section of this paper.

II. HIGH SPEED FOCAL PLANE IMAGE-PROCESSING

CAPABILITIES

In an increasingly digital world, the most part of imaging systems has become almost entirely digital, using only an analog-to-digital (ADC) between the sensor and the processing operators. However, low-level image processing usually involves basic operations using local masks. These local operations are spatially dependent on other pixels around the processed pixel Since the same type of operations is applied to a very large data set, these low-level tasks are computationally intensive and require a high bandwidth between the image memory and the digital processor. In this case, an analog or a mixed-approach can offer superior performance leading to a smaller, faster, and lower power solution than a digital processor [START_REF] Martin | A mixed-signal array processor with early vision applications[END_REF]. Low-level image processing tasks are inherently pixel-parallel in nature. Integrating a processing element within each pixel based on a single instruction multiple data (SIMD) architecture is a natural candidate to cope with the processing constraints [START_REF] Cembrano | A 1000 FPS at 128 × 128 Vision Processor With 8-Bit Digitized I/O[END_REF]. This approach is quite interesting for several aspects. First, SIMD image-processing capabilities at focal plane have not been fully exploited because the silicon area available for the processing elements is very limited.

Nevertheless, this enables massively parallel computations allowing high framerates up to thousands of images per second. The parallel evaluation of the pixels by the SIMD operators leads to processing times, independent of the resolution of the sensor. In a classical system, in which low-level image processing is externally implemented after digitization, processing times are proportional to the resolution leading to lower framerates as resolution increases. Several papers have demonstrated the potentially outstanding performance of CMOS image sensors [START_REF] Krymski | A high speed, 500 frames/s, 10241024 cmos active pixel sensor[END_REF]- [START_REF] Kleinfelder | A 10 000 Frames/s CMOS Digital Pixel Sensor[END_REF]. Krymski et al. [START_REF] Krymski | A high speed, 500 frames/s, 10241024 cmos active pixel sensor[END_REF] describe a high speed (500 frames/s) large format 1024 × 1024 Active Pixel Sensor (APS) with 1024 ADCs. Stevanovic et al. [START_REF] Stevanovic | A cmos image sensor for high-speed imaging[END_REF] describe a 256 × 256 APS which achieves more than 1000 frames/s with variable integration times. Kleinfelder et al. [START_REF] Kleinfelder | A 10 000 Frames/s CMOS Digital Pixel Sensor[END_REF] describe a 352 × 288 Digital Pixel Sensor(DPS) with per pixel bit parallel ADC achieving 10,000 frames/s or 1 Giga-pixels/s. Secondly, the high speed imaging capability of CMOS image sensors can benefit the implementation of new complex applications at standard rates and improve the performance of existing video applications such as motion vector estimation [START_REF] Handoko | A CMOS image sensor for local-plane motion vector estimation[END_REF]- [START_REF] Liu | Photocurrent estimation from multiple nondestructive samples in a CMOS image sensor[END_REF], multiple capture with dynamic range [START_REF] Yang | A 640 x 512 CMOS Image Sensor with Ultra Wide Dynamix Range Floating-Point Pixel-Level ADC[END_REF]- [START_REF] Stoppa | Novel CMOS Image Sensor With a 132-dB Dynamic Range[END_REF], motion capture [START_REF] Liu | Simultaneous image formation and motion blur restoration via multiple capture[END_REF], and pattern recognition [START_REF] Wu | A Low-Photocurrent CMOS Retinal Focal-Plane Sensor With a Pseudo-BJT Smoothing Network and an Adaptative Current Schmitt Trigger for Scanner Applications[END_REF]. Indeed, standard digital systems are unable to operate at high framerates, because of the high output data rate requirements for the sensor, the memory, and the processing elements. Integrating the memory and processing with the sensor on the same chip removes the classical input output bottleneck between the sensor and the external processors in charge of processing the pixel values. Indeed, the bandwidth of the communication between the sensor and the external processors is known as a crucial aspect, especially with high resolution sensors. In such cases, the sensor output data flow can be very high, and needs a lot of hardware ressources to convert, process and transmit a lot of information. So, integrating image processing at the sensor level can alleviate the high data rate problem because the pixel values are pre-processed on-chip by the SIMD operators before sending them to the external world via the communication channel. This will result in data reduction, which allows sending the data at lower data-rates, and reduces the effect of the computational-load bottleneck.

Thirdly, one of the main drawbacks to design specific circuits integrating sensing and processing on the same chip is that these vision chips are often built as special-purpose devices, performing specific and dedicated tasks, and not reusable in another context [START_REF] Dudek | An analogue simd focal-plane processor array[END_REF]. So, it can be widely beneficial to integrate a versatile device, whose functionality can be easily modified. Moreover, except the basic operations such as convolutions with small masks, the majority of computer vision algorithms requires the sequential execution of different successive low level image processing on the same data. So, each processing element must be built around a programmable execution unit, communication channels, and local memories dedicated to intermediate results. Because of the very limited silicon area, the processing units are necessarily very simple, providing the best compromise between various factors such as versatility, complexity, parallelism, processing speeds and resolution.

To sum up, the flexibility to integrate processing down to the pixel level allows us to rearchitect the entire imaging system to achieve much higher performances [START_REF] Gamal | Cmos image sensors[END_REF]. The key idea is 1) to capture images at a very high framerate, 2) to process the data on each pixel with a SIMD programmable architecture exploiting the high on-chip bandwidth between the sensor, the memory and the elementary processors and 3) to provide results at the best framerate depending on the complexity of the image processing. In this paper, we present our approach to the design of a massively parallel, SIMD vision chip based implementing low level image processing based on local masks. Our analog processing operators are fully programmable devices by dynamic reconfiguration, and can be viewed as a software-programmable image processor dedicated to low-level image processing. The main objectives of our design are: 1) to evaluate the potential for high speed snap-shot imaging and, in particular, to reach a 10 000 frames/s rate, 2) to demonstrate a versatile and reconfigurable processing unit at pixel-level, and 3) to provide an original platform for experimenting with low-level image processing algorithms that exploit high-speed imaging.

III. DESCRIPTION OF THE ARCHITECTURE

The proof-of-concept chip presented in this paper is depicted in Fig. 1. The core includes a two-dimensional array of 64 × 64 identical processing element (PE). It follows the single instruction multiple data (SIMD) computing paradigm. Each of the PE is able to convolve the pixel value issued from the photodiode by applying a set of mask coefficients to the image pixel values located in a small neighborhood. The key idea is that a global control unit can dynamically reconfigure the convolution kernel masks and then implements the most part of low-level image processing algorithms. This confers the functionality of programmable processing devices to the PEs embedded in the circuit. Each individual PE includes the following elements:

• a photodiode dedicated to the optical acquisition of the visual information and the light-to-voltage transduction, • two Analog Memory, Amplifier and Multiplexer structures called [AM] 2 , which serve as intelligent pixel memories and are able to dissociate the acquisition of the current frame in the first memory and the processing of the previous frames in the second memory, • an Analog Arithmetic Unit named A 2 U based on four analog multipliers, which performs the linear combination of the four adjacent pixels using a 2 × 2 convolution kernel. In brief, each PE includes 38 transistors integrating all the analog circuitry dedicated to the image processing algorithms. The global size of the PE is 35 µm × 35 µm (1225 µm 2 ). The active area of the photodiode is 300 µm 2 , giving a fillfactor of 25 %. The chip has been realized in a standard 0.35 µm double-poly quadruple-metal CMOS technology and contains about 160 000 transistors on a 3.67 mm × 3.77 mm die (13.83 mm 2 ). The chip also contains test structures on the bottom left of the chip. These structures are used for detailed characterization of the photodiodes and processing units.

IV. CIRCUIT DESIGN

A. Pixel Structure

Each pixel in the CMOS image sensor array consists of a photodiode and a processing unit dedicated to low-level image processing based on neighborhoods. In our chip, the type of photodiodes is one of the simplest photo element in CMOS image sensor technology. It consists of N-type photodiodes based on an n + -type diffusion in a p-type silicon substrate. The depletion region is formed in the neighborhood of the photodiode cathode. Optically generated photocarriers diffuse to neighboring junctions [START_REF] Lee | CMOS Photodiodes with Substrate Openings for Higher Conversion Gain in Active Pixel Sensor[END_REF]. In order to achieve good performances, the photodiodes should be designed and optimized carefully, in order to minimize critical parameters such as the dark current and the spectral response [START_REF] Wu | Design, optimization, and performance analysis of new photodiode structures for CMOS active-pixel-sensor (APS) imager applications[END_REF]. The shape of photodiode layout, the structure of the photodiode, and the layout have significant influences on the performance of the whole imager [START_REF] Shcherback | Empirical dark current modeling for complementary metal oxide semiconductor active pixel sensor[END_REF], [START_REF] Shcherback | Photoresponse analysis and pixel shape optimization for CMOS active pixel sensors[END_REF]. The active area of the photodiode absorbs the illumination energy and turns that energy into charge carriers. This active area must be large as possible in order to absorb a maximum of photons. In the mean time, the control circuitry required for the readout of the collected charges and the inter-element isolation area must be as small as possible in order to obtain the best fill factor. We have theoretically analyzed, designed and benchmarked different photodiodes shapes [START_REF] Dubois | A 10 000 fps cmos sensor with massively parallel image processing[END_REF], and finally, an octagonal shape based on 45 • structures was chosen (see Fig. 1). More details about the photodiodes design can be found in the aforementioned paper. Here, only the basic concept behind the photodiodes design has been briefly overviewed.

The second part of the pixel is the analog processing unit. Existing works on analog pixel-level image processing can be classified into two main categories. The first one is intrapixel, in which processing is performed on the individual pixels in order to improve image quality, such as the classical Active Pixel Sensor or APS [START_REF] Yadid-Pecht | In-Pixel Autoexposure CMOS APS[END_REF], [START_REF] Yadid-Pecht | CMOS Active Pixel Sensor Star Tracker with Regional Electronic Shutter[END_REF]. The second category is interpixel, where the processing is dedicated to groups of pixels in order to perform some early vision processing and not merely to capture images. Our work clearly takes place in this category because our main objective is the implementation of various in-situ image processing using local neighborhoods. Based on this design concept, this forces a re-thinking of the spatial distribution of the processing resources, so that each computational unit can easily use a programmable neighborhood of pixels. For this purpose, the pixels are mirrored about the horizontal and the vertical axis in order to share the different Analog Arithmetic Units (A 2 U). As example, a block of 2 × 2 pixels is depicted in Fig. 1. The main feature of its innovative distribution is to optimize the compactness of the metal interconnections with pixels, to contribute to a better fill factor, and to provide generality of high speed processing based on neighborhood of pixels.

B. Analog Memory, Amplifier and Multiplexer : [AM] 2

In order to increase the algorithmic possibilities of the architecture, the key point is the separation of the acquisition of the light inside the photodiode and the readout of the stored value at pixel-level [START_REF] Chapinal | A 128 × 128 CMOS Image Sensor With Analog Memory for Synchronous Image Capture[END_REF]. Thus, the storage element should keep the output voltage of the previous frames whereas the sensor integrates photocurrent for a new frame. So, we have designed and implemented dedicated pixels including a light sensitive structure and two specific circuits called Analog Memory, Amplifier, and Multiplexer ([AM] 2 ), as shown in Fig. 2.

The system has five successive operation modes: reset, integration, storage, amplification, and readout. All these phases are externally controlled by global signals common to the full array of pixels. They all occur in parallel over the sensor (snapshot mode) in order to avoid any distortion due to a row-byrow reset. In each pixel, the photosensor is a N-type photodiode associated with a PMOS transistor reset. This switch resets the integrating node to the fixed voltage V dd . The pixel array is held in the reset mode until the init signal raises, turning the PMOS transistor off. Then, the photodiode discharges for a fixed period, according to the incidental luminous flow. The first NMOS transistor acts as a transconductance, producing the voltage V ph , directly proportional to the incident light intensity. The integrated voltage is polarized around V dd /2 by the second NMOS transistor. The calibration of the structure is ensured by the positive reference bias voltage (V bias = 1.35V ).

Following the acquisition stage, two identical subcircuits [AM] 2 i (with i = 1, 2) take place to realize the storage phase of V ph . Each [AM] 2 includes three pairs of NMOS and PMOS transistors and a capacitor which acts as an analog memory. The subcircuit [AM] 2 i is selected when the st i signal is turned on. Then, the associated analog switch is open allowing the integration of the photogenerated current in the corresponding C i capacitor. Consequently, the capacitors are able to store the pixel value during the frame capture from one of the two switches. The capacitors are implemented with double-polysilicium. The size of the capacitors is as large as possible in order to respect the fill-factor and the pixel size requirements. The capacitors values are about 40 fF. They are able to store the pixel value for 20 ms with an error lower than 4 %. Behind the storage subcircuit, a basic CMOS inverter is integrated. This inverter serves as a linear high-gain amplifier since the pixel signal is polarized around V dd /2. Finally, the last phase consists in the readout of the stored values in the capacitors C i . The integrated voltage across the capacitor C i can be readout on the output out i through one of the two switches, controlled by the r i signals.

Fig. 3 describes the experimental results of successive acquisitions in an individual pixel. The acquisitions occur when one of the two signals st 1 or st 2 goes high. The two combinaisons of acquisitions are presented in this example. After the first reset, st 2 is followed by st 1 whereas the inverted sequence of acquisitions is realized after the second reset. The signal V ph gives the voltage corresponding to the incidental illumination on the pixel and the the two outputs (out 1 and out 2 give the voltage stored in each of the capacitors when the associated readout signal raises. 2) The image acquisition is time-decorrelated from image processing, implying that the architecture performance is always the highest, and the processing framerate is maximum, 3) A new image is always available without spending any integration time.

C. Analog Arithmetic

Unit: A 2 U
When designing a pixel-level processing unit, you should consider adopting efficient strategies to minimize the silicon area occupied by the processor. To that end, we designed an analog arithmetic unit (A 2 U) which is able to perform convolution of the pixels with a 2x2 dynamic kernel. This unit is based on four-quadrant analog multipliers [START_REF] Ryan | Applications of a four-quadrant multiplier[END_REF], [START_REF] Liu | CMOS Squarer and Four-Quadrant Multiplier[END_REF] named M1, M2, M3, and M4, as illustrated in Fig. 5. Each multiplier requires 5 transistors. So, the transistor count of the complete unit is only 22 transistors. The last two transistors not depicted on the Fig. 5 serve as an output switch, driven by the column signal. It features relative small area, simplicity, and high speed. These characteristics make it an interesting choice in low-level image processing embedded at focal plane. Each multiplier M i (with i = 1, . . . , 4) takes two analog signals V i1 and V i2 and produces an output V iS which is their product. The outputs of multipliers are all interconnected with a diodeconnected transistor employed as load. Consequently, the global operation result at the V S point is a linear combination of the four products V iS . Image processing operations such as spatial convolution can be easily performed by connecting the inputs V i1 to the kernel coefficients and the inputs V i2 to the corresponding pixel values.

In order to keep the analysis simple, it is assumed in this section that the contribution of parasitic capacitances is negligible. Considering the MOS transistors operating in subthreshold region, the output node V iS of a multiplier can be

k r (V T hN -V i1 ) (V i1 -V i2 -V T hN ) = (V i1 -V iS -V T hN ) (V i2 -V iS -V T hN -V T hP ) (1)
with k r represents the transconductance factor, V T hN and V T hP are the threshold voltage for the NMOS and PMOS transistors. Around the operating point (V dd /2), the variations of the output node mainly depend on the product V i1 V i2 . So, the equation 1 can be simplified and finally, the output node V iS can be expressed as a simple first-order of the two input voltages V i1 and V i2 .

V iS = M V i1 V i2 with, M = k r -1 2V T hN + V T hP ≈ 8.07/V
(2) The important value of the coefficient M gives to the structure a good robustness by limiting the impact of the second-order intermodulation products. The first consequence is a better linearity of our multiplier design integrating only 5 transistors.

The theoretical analysis has been validated by an experimental study realized on the test structures embedded on the chip. Fig. 6 shows the experimental measures obtained with two cosine signals as inputs of the multiplier structure.

V i1 = Acos(2πf 1 ) with f 1 = 2.5kHz (3) V i2 = Bcos(2πf 2 ) with f 2 = 20kHz (4)
In an ideal case, the voltage V S at the output of the multiplier can be expressed as:

V S = AB 2 [cos(2π(f 2 -f 1 )) + cos(2π(f 2 + f 1 ))] (5)
The frequency spectrum, represented in Fig. 6(b) contains two main frequencies (17.5 kHz and 22.5 kHz) around the carrier frequency. The residues which appear in the spectrum are known as intermodulations products. Intermodulation is caused by non-linear behaviour of the of the structure (around 10 kHz and 30 kHz) and the insulation defects of input pads (at 40 kHz). Nevertheless, the amplitude of these intermodulation products is significantly lower than the two main frequencies. Indeed, at 40 kHz, the level of intermodulation is 9dB under the level of the main frequencies. Therefore, the contribution of the insulation defect is eight times smaller than the main signals. Furthermore, the experimental measures highlighted that the linearity of the multiplier is maximum for input signals with amplitude range between 0.6 V and 2.6 V. This corresponds to the typical range of values coming from the pixel since the [AM] 2 structures provide values around V dd /2=1.65V.

V. CHIP CHARACTERIZATION

An experimental 64×64 pixel image sensor has been developed in a 0.35 µm, 3.3 V, standard CMOS process with polypoly capacitors. Its functional testing and its characterization were performed using a specific hardware platform. The hardware part of the imaging system contains a one million Gates Spartan-3 FPGA board with 32MB SDRAM embedded. This FPGA board is the XSA-3S1000 from XESS Corporation. An interface acquisition circuit includes three ADC from Analog Device (AD9048), high speed LM6171 amplifiers and others elements such as the motor lens. Fig. 7 shows the schematic and some pictures of the experimental platform.

A. Electrical characterization

The sensor was quantitatively tested for conversion gain, sensitivity, fixed pattern noise, thermal reset noise, output levels disparities, voltage gain of the amplifier stage, linear flux, and dynamic range. Table I summarizes the main chip properties and the characterization results. To determine these values, the sensor included specific test pixels in which some internal node voltages can be directly read. The test equipment hardware is based on a light generator with wavelength of 400 nm to 1100 nm. The sensor conversion gain was evaluated to 54 µV/e -RMS with a sensitivity of 0.15 V/lux.s, thanks to the octagonal shape of the photodiode and the fill factor of 25%. At 10 000 frames/s, measured nonlinearity is 0.12% over a 2 V range. These performances are similar to the sensor described in [START_REF] Kleinfelder | A 10 000 Frames/s CMOS Digital Pixel Sensor[END_REF]. According to the experimental results, the voltage gain of the amplifier stage of the two [AM] 2 is Av = 12 and the disparities on the output levels are about 4.3 %.

B. Fixed pattern noise

Image sensors always suffer from technology related nonidealities that can limit the performances of the vision system. Among them, fixed pattern noise (FPN) is the variation in output pixel values, under uniform illumination, due to device and interconnect mismatches across the image sensor. Two main types of FPN occur in CMOS sensors. First, offset FPN which takes place into the pixel is due to fluctuations in the threshold voltage of the transistors. Second, the most important source of FPN is introduced by the column amplifiers used in standard APS systems. In our approach, the layout is symmetrically built in order to reduce the offset FPN among each block of four pixels and to ensure uniform spatial sampling, as already depicted in the layout of a 2x2 pixel block in Fig. 1. Furthermore, our chip does not include any column amplifier since the amplification of the pixel values takes place into the pixel by means of an inverter. So, the gain FPN is very limited and only depends on the mismatch of the two transistors. FPN can be reduced by correlated double sampling (CDS). To implement CDS, each pixel output needs to be read twice, once after reset and a second time at the end of integration. The correct pixel signal is obtained by substracting the two values. A CDS can be easily implemented in our chip. For this purpose, the first analog memory stores the pixel value just after the reset signal and the second memory stores the value at the end of integration. Then, at the end of the image acquisition, the two values can be transfered to the FPGA, responsible for producing the difference. In Fig. 8, the two images show fixed pattern noise with and without CDS using a 1-ms integration time. On the left image, the FPN is mainly due to the random variations in the offset voltages of the pixel-level analog structures. The experimental benchmarks of our chip reveal a FPN value of 225 µV RMS. The right picture shows the same image after analog CDS, performed as described above. The final FPN has been reduced by a factor of 34 to 6.6 µV. In the rest of the results, CDS has not been implemented since FPN has low values. Only, an entire dark image is substracted from the output images on the FPGA. Focus has been made on the development of low-level image processing using the two analog memories and the associated processing unit.

VI. HIGH-SPEED IMAGE PROCESSING APPLICATIONS

In this section, we provide experimental results of image processing implemented on our high-speed vision chip. First, we demonstrate the possibility of acquisition of raw images at different framerates, up to 10 000 frames/s. Secondly, we present an implementation of edge detection, based on the well-known Sobel operator. Fig. 9. Various raw images acquisition at 1 000, 5 000 and 10 000 frames/s

A. Sample Images

The prototype chip was used for acquisition of raw images. First, sample raw images of stationary scenes were captured at different framerates, as shown in Fig. 9. In the three views, no image processing is performed on the video stream, except for amplification of the photodiodes signal. From left to right, we can see a human face obtained at 1 000 frames/s, a static electric fan at 5 000 frames/s, and a electronic chip at 10 000 frames/s.

B. Sobel operator

The characteristics of our sensor, especially the analog processing unit, make it extremely useful for low-level image processing based on convolution masks. In this paragraph, we report an edge detector, the Sobel detector. The Sobel operator estimates the gradient of a 2D image. This algorithm is used for edge detection in the preprocessing stage of computer vision systems. The classical algorithm uses a pair of 3x3 convolution kernels (see Eq. 6), one to detect changes along the vertical axis (h 1 ) and another to detect horizontal contrast (h 2 ). For this purpose, the algorithm performs a convolution between the image and the sliding convolution mask over the image. It manipulates 9 pixels for each value to produce. The value corresponds to an approximation of the gradient centered on the processed image area.

h 1 =   -1 0 1 -2 0 2 -1 0 1   h 2 =   -1 -2 -1 0 0 0 1 2 1   (6)
The structure of our architecture is well-adapted to the evaluation of the Sobel algorithm. It leads to the result directly centered on the photo-sensor and directed along the natural axes of the image. The gradient is computed in each pixel of the image by performing successive linear combinations of the 4 adjacent pixels. For this purpose, each 3x3 kernel mask is decomposed into two 2x2 masks that successively operate on the whole image. For the kernel h 1 , the corresponding 2x2 masks are:

m 1 = -1 0 -1 0 m 2 = 0 1 0 1 (7)
The Fig 11 represents the 3x3 mask centered on the pixel ph 5 . Each octagonal photodiode ph i (i = 1, . . . , 9) is associated with a processing element P E i , represented with a circle on the Figure . Each P E i is positioned on the bottom right of its photodiode, as in the real layout of the circuit (see Fig 1). The first mask m 1 contributes to evaluate the following series of operations for the four P E i s:

V 11 = -(V ph1 + V ph4 ) V 12 = -(V ph2 + V ph5 ) V 14 = -(V ph4 + V ph7 ) V 15 = -(V ph5 + V ph8 ) (8)
and the second mask m 2 computes:

V 21 = +(V ph2 + V ph5 ) V 22 = +(V ph3 + V ph6 ) V 24 = +(V ph5 + V ph8 ) V 25 = +(V ph6 + V ph9 ) (9)
with V ij corresponding to the result provided by the processing element P E j (j = 1, 2, . . . , 9) with the mask m i (i = 1, 2), and V ph k (k = 1, 2, . . . , 9), the voltages representing the incidental illumination on each photodiode ph k . Then, the evaluation of the gradient at the center of the mask can be computed by summing the different values on the external FPGA. Note that V 12 = -V 21 and V 15 = -V 24 . So, the final sum can be simplified and written as V h1 = V 11 + V 22 + V 25 + V 14 . If we define a retina cycle as the time spent for the configuration of the coefficients kernel and the preprocessing of the image, the evaluation of the gradient on the vertical direction only spends a frame acquisition and two retina cycles. By generalization, the estimation of the complete gradient along the two axis spend 4 cycles because it involves 4 dynamic configurations. In short, the dynamic assignment of coefficient values from the external processor gives the system some interesting dynamic properties. The system can be easily reconfigured by changing the internal coefficients for the masks between two successive computations. First, this allows the possibility to dynamically change the image processing algorithms embedded in the sensor. Secondly, this enables the evaluation of some complex pixel-level algorithms, implying different successive convolutions. The images can be captured at higher framerates than the standard framerate, processed by exploiting the the analog memories and the reconfigurable processing elements and output at a lower framerate depending of the number of the dynamic reconfigurations. Moreover, the analog arithmetic units implementing these pixel-level convolutions drastically decrease the number of single operations such as additions and multiplications executed by an external processor (a FPGA in our case) as shown in Fig. 7. Indeed, in the case of our experimental 64 × 64 pixel sensor, the peak performance is equivalent to 4 parallel signed multiplications by pixel at 10 000 frames/s, i.e. more than 160 million multiplications per second. With a VGA resolution (640 × 480), the performance level would increase to a factor of 75, leading to about 12 billion multiplications per second. Processing this data flow by external processors will imply important hardware resources in order to cope with the temporal constraints.

As an illustration of the Sobel algorithm, Fig. 12 is an example sequence of 16 images of a moving object, namely, an electric fan. Two white specific markers are placed on the fan, i.e a small circle near the rotor and a painted blade. The speed rotation of the fan is 3750 rpm. In order to capture such a rapidly moving object, a short integration time (100 µs) was

Chip

This work SCAMP-3 [START_REF] Dudek | Implementation of simd vision chip with 128×128 array of analogue processing elements[END_REF], [START_REF] Dudek | General-purpose 128×128 SIMD processor array with integrated image sensor[END_REF] MIMD IP Chip [START_REF] Etienne-Cummings | A programmable focal-plane mimd image processor chip[END_REF] ACE16k [START_REF] Cembrano | A 1000 FPS at 128 × 128 Vision Processor With 8-Bit Digitized I/O[END_REF], [START_REF] Rodriguez-Vasquez | ACE16k: the third generation of mixed-signal SIMD-CNN ACE chips toward VSoCs[END_REF] VCS-IV [START_REF] Komuro | A digital vision chip specialized for high-speed target tracking[END_REF] used for the frames acquisition. The Sobel algorithm allow to distinguish clearly the two white markers even with a high framerate. In the preceding sections, we focused on 2x2 and 3x3 convolution masks. In the case of a 2x2 mask, the coefficients are fixed once before the beginning of the acquisition frame. In the case of a 3x3 mask, two possibilities can occur. First, the 3x3 mask presents some symmetrical properties (such as the Sobel kernel) and then the coefficients values can be fixed as in a 2x2 mask. Second, if the mask is not symmetric, it is necessary to dynamically reconfigure the coefficients during the acquisition frame. For masks which size is greater than 3x3 and more generally in the case of an NxN mask, a dynamic reconfiguration of coefficients is necessary during the acquisition frame in order to evaluate the successive values of the linear combinations of pixels.

VII. COMPARISON WITH OTHER SIMD VISION CHIPS

Table II shows an overview of some representative SIMD vision chips based on different alternatives for implementing vision processing at focal plane. The first column corresponds to the chip described in this paper. On the second column, we can find the main characteristics of SCAMP-3, a general purpose processor array implementing a variety of low-level image processing tasks at a high framerate. The third colum describes a Multiple Instruction Multiple Data Image Processing Chip (MIMD IP Chip) performing spatial convolutions using kernels from 3×3 to 11×11. The fourth column presents a vision chip integrating an imager and an array of mixedsignal SIMD processing elements that can process grayscale images with Cellular Neural Network Universal Machines (CNN-UM) mode of operation. The fifth vision chip in the table is a programmable SIMD vision chip that can perform various early visual processing such as edge detection, smoothing or filtering by chaining processing elements and reconfiguring the hardware dynamically. Finally, the last one is a programmable artificial retina in which each pixel contains a tiny digital processing element capable of grey-level image processing from low to mid-level vision (motion detection, segmentation, pattern recognition).

Compared to this state-of-the-art of high-speed CMOS image sensors, one easily sees that the chip reported in this paper lead to some major improvements. With a pixel size of 35 µm by 35 µm, the pixel pitch is almost 1.5 more compact than the smallest pixel described in [START_REF] Dudek | Implementation of simd vision chip with 128×128 array of analogue processing elements[END_REF], [START_REF] Dudek | General-purpose 128×128 SIMD processor array with integrated image sensor[END_REF]. This contributes either to a better resolution of the sensor for a given chip area or a lower cost for a given resolution. At the same time, as compared with the other processing elements, our solution relies on a compact analog arithmetic unit based on a four-quadrant multiplier architecture using only 38 minimal size transistors. This leads to a fill factor of about 25 %. Consequently, the active area of our photodiode is bigger compared to the major part of the other chips, providing a best sensibility to the sensor at high framerates of thousands of images per second.

From the performance point of view, all the chips on the table implement low-level image processing by programming or dynamically configuring the processing elements. Our solution is able to capture image, run user-defined 3x3 convolution masks, and provide the results on the sensor output bus in less than 200 µs, giving a framerate of 5000 images per second.

These temporal performances are compatible and even slightly higher than the other sensors since each of them is only able to provide low-level image processing results at 1000 frames per second, except the MIMD IP Chip.

From the programmability point of view, our chip can suffer from less versatility compared to other programmable sensors. Indeed, the implementation of complex low-level image processing require successive reconfigurations of the internal masks coefficients of the processing elements. This task may be more difficult in comparison with algorithms easily written in a machine-level language for the programmable sensors, such as the SCAMP-3 sensor.

VIII. CONCLUSION AND PERSPECTIVES

An experimental pixel sensor implemented in a standard digital CMOS 0.35 µm process has been described in this paper. Each 35 µm × 35 µm pixel contains 38 transistors implementing a circuit with photo-current integration, two [AM] 2 (Analog Memory, Amplifier, and Multiplexer), and an A 2 U (Analog Arithmetic Unit).

Experimental chip results reveal that raw image acquisition at 10 000 frames per second can be easily achieved using the parallel A 2 U implemented at pixel-level. With basic image processing, the maximal framerate slows down to about 5 000 fps. The potential for dynamic reconfiguration of the sensor was also demonstrated in the case of the Sobel operator.

The next step in our research will be the design of a similar circuit in a modern 130nm CMOS technology. The main objective will be to design a pixel of less than 10 µm × 10 µm with a fill factor of 20%. A second possibility would be the design of a sensor with emergent technologies using amorphous silicon in which 3D pixels can be built. With a photosensitive layer placed just behind the optical part, the pixel fill factor can reach 100% since the processing elements can be packed in the empty space below the photodiode.

Thus, with the increasing scaling of the transistors in such technologies, we could consider the implementation of more sophisticated image processing operators dedicated to face localization and recognition. Previous works of our team [START_REF] Yang | Implementation of an RBF neural network on embedded systems: real-time face tracking and identity verification[END_REF] have demonstrated the needs of dedicated CMOS sensors embedding low-level image processing such as features extraction. Moreover, actual works [START_REF] Farrugia | A Parallel Face Detection System Implemented on FPGA[END_REF] focus on a recent face detector named Convolutional Face Finder (CFF) [START_REF] Garcia | Convolutional face finder: a neural architecture for fast and robust face detection[END_REF]. CFF is based on a multi-layer convolutional neural architecture. The CFF consists of six successive neural layers. The first four layers extract characteristic features, and the last two perform the classification. Our objective would be to implement at pixel-level the first layers based on convolutions by different masks from 2 × 2 to 5 × 5.

In order to evaluate this future chip in some realistic conditions, we would like to design a CIF sensor (352 × 288 pixels), which leads to a 3.2 mm × 2.4 mm in a 130 nm technology. The exploitation of high FPS capability with this sensor could be achieved by two complementary ways. The first one is to integrate a dedicated Input/Output module, which is able to cope with a gigapixel per second bandwidth. Such modules have been already designed in other high-speed CMOS sensors. As an example, we can cite the Digital Pixel Sensor [START_REF] Kleinfelder | A 10 000 Frames/s CMOS Digital Pixel Sensor[END_REF] with his 64-bit (8-pixel) wide bus operating at 167 MHz, able to output 1.33 GB/s. The second way is to build a large sensor by assembling 64 × 64 pixel modules with a dedicated output bus for each of them. For example, with a 384 × 256 pixel sensor, this solution only requires 6x4=24 dedicated outputs. In the same time, we will focus on the development of a fast analog to digital converter (ADC). The integration of this ADC on future chips will allow us to provide new and sophisticated vision systems on chip (ViSOC) dedicated to digital embedded image processing at thousands of frames per second.

Fig. 2 .

 2 Fig. 2. The [AM] 2 structure

Fig. 3 .

 3 Fig. 3. High speed sequence capture with basic image processing One of the main advantages of the two [AM] 2 structures is

Fig. 4 .

 4 Fig. 4. Parallelism between capture sequence and readout sequence

Fig. 5 .

 5 Fig. 5. The A 2 U structure

Fig. 6 .

 6 Fig. 6. Experimental results of the multiplication of two cosine signals by the four-quadrant multiplier

Fig. 7 .

 7 Fig. 7. Block diagram and pictures of the hardware platform including FPGA board and CMOS sensor

Fig. 8 .

 8 Fig. 8. Images of fixed pattern noise (a) without CDS and (b) with CDS for an integration time of 1 ms

Fig. 10 .

 10 Fig. 10. A 2 500 frames/s video sequence of a milk drop splashing

Fig. 11 .

 11 Fig. 11. 3x3 kernel used by the 4 processing elements

Fig. 12 .

 12 Fig. 12. Sequence of 16 images with Sobel operator

TABLE I CHIP

 I

	CHARACTERISTICS AND MEASUREMENTS
	Technology	0.35 µm 2-poly 4-metal CMOS
	Array size	64 × 64
	Chip area	13.8 mm 2
	Pixel size	35 µm × 35 µm
	Number of transistors	160 000
	Number of transistors / pixel	38
	Sensor Fill Factor	25 %
	Dynamic power consumption	110 mW
	Conversion gain	54 µV/e -RMS
	Sensitivity	0.15 V/lux.s
	Fixed Pattern Noise retina (FPN), dark	225 µV RMS
	Thermal reset noise	68 µV RMS
	Output levels disparities	4.3%
	Voltage gain of the amplifier stage	12
	Linear flux	98.5%
	Dynamic range	68 dB

TABLE II COMPARISON

 II WITH OTHER SIMD SENSORS