Young-Gu Ju
email: ygju@knu.ac.kr

Scalable optical convolutional neural network based on free-space optics using lens arrays and a spatial light modulator

Keywords: optical neural network, convolutional neural network, free-space optics, optical computer, smart pixels

A scalable optical convolutional neural network (SOCNN) based on free-space optics and Koehler illumination was proposed to address the limitations of the previous 4f correlator system. Unlike Abbe illumination, Koehler illumination provides more uniform illumination and reduces crosstalk. SOCNN allows for scaling up of the input array and the use of incoherent light sources. Hence, the problems associated with 4f correlator systems can be avoided. We analyzed the limitations in scaling the kernel size and parallel throughput and found that SOCNN can offer a multilayer convolutional neural network with massive optical parallelism.

Introduction

In recent times, the advent of artificial neural networks with deep learning algorithms has led to considerable advances in applications such as image and speech recognition and natural language processing [START_REF] Hinton | Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups[END_REF][START_REF] Lecun | Deep learning[END_REF]. The convolutional neural network (CNN) is a type of deep learning algorithm that is particularly effective in image and video analysis [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. CNNs are specifically designed to automatically detect and extract features such as edges, corners, and textures from images; these features can be used to classify the images into different categories. These applications involve processing an input image by applying convolution operations using kernels of different sizes. The results of these convolutions are then pooled, passed through a nonlinear activation function, and sent to the next layer of convolutional operations. Although CNNs are excellent at solving classification and recognition problems, they require a massive amount of computation, especially when dealing with large images and kernels. When an input image with n × n pixels is convolved with a kernel of size k × k, the amount of computation is proportional to (n 2 × k 2 ). The computational requirement grows further with an increasing number of layers, resulting in high latency and large power consumption in the case of forward inference in the pretrained network. Although the use of graphics processing units can alleviate the issue of latency, real-time inference may still remain a challenge [START_REF] Chetlur | cuDNN: efficient primitives for deep learning[END_REF].

Currently, researchers are exploring the use of free-space optics to implement CNNs in an optical form owing to the high parallelism and energy efficiency of these optics [START_REF] Colburn | Optical frontend for a convolutional neural network[END_REF][START_REF] Chang | Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification[END_REF][START_REF] Lin | All-optical machine learning using diffractive deep neural networks[END_REF][START_REF] Sui | A review of optical neural networks[END_REF]. Optical convolutional neural networks (OCNNs) based on free-space optics traditionally use the well-known 4f correlator system to exploit the Fourier transform property. Although these types of OCNNs have some advantages, they cause several inherent problems because of the use of Fourier optics. The first issue is the limitation in the scalability of the input image array; a lens is used for Fourier transformation, and the lens has a finite space-bandwidth product (SBP) owing to its geometric aberration. The second issue is the latency caused by the time taken to generate the input array. In a Fourier transform-based system, a laser and a spatial light modulator (SLM) are required to generate a coherent input. However, the currently available SLMs are mostly slow and serially addressable, thereby causing remarkable latency. This latency diminishes the advantages of the massive parallelism of the optical neural network. Additionally, this latency makes it challenge to build a cascaded system for a multilayer neural network. The third problem is the difficulty in reconfiguring the kernels. The kernel pattern on the 4f system is the Fourier transform of the kernel pattern; obtaining the Fourier transform requires computation and can lead to significant delays in renewal.

To address these issues, we propose a scalable optical convolutional neural network (SOCNN) based on free-space optics and Koehler illumination, which uses lens arrays and a SLM. The SOCNNs proposed herein are a variation of the previously reported optical neural network. They accommodate the CNN architecture in the context of a linear combination optical engine (LCOE) [START_REF] Ju | A scalable optical computer based on free-space optics using lens arrays and a spatial light modulator[END_REF]. The goal of the LCOE was full interconnection; in contrast, the goal of the SOCNNs was partial connection with an unlimited input array size.

Theory

In a typical 4f correlator system, a mask is located at the focal plane of two lenses, as shown in Fig. 1. Lens1 performs the Fourier transform, while Lens2 performs inverse Fourier transform. The mask represents the complex-valued Fourier transform of a kernel, which is multiplied by the Fourier transform of an input pattern. Thus, the output plane displays the convolution of the input array and kernel.

The SBP of the 4f imaging system is approximately

( 𝐷 2 𝜆𝑓 ) 2 ,
where D, λ, and f are the diameter of the lens, wavelength of the light source, and focal length of the lens, respectively. The SBP can be expressed as ( 𝐷 𝜆 𝑓/# ) 2 using the f-number (f/#) of the lens. If the fixed wavelength and f/# are used, the SBP can increase infinitely with D. However, this SBP arises from the diffraction limit of the lens used in the system. As D increases, it becomes difficult for the system to reach the diffraction limit. A larger system requires less geometric aberration and more elements and tighter alignment tolerance to reach the diffraction limit. The SBP of the system is about

( 𝐷 2 𝑓 𝛿 ) 2 = ( 1 2 𝑓/# 𝛿 ) 2
where is the angular aberration of the lens. When a triplet lens has a f-number of 2, the angular aberration is approximately 3 mrad and the SBP is about 83 × 83. Since a lens system can worsen alignment problems during assembly with an increase in the number of elements, the practical scaling limit of the 4f system can be about 250 × 250 if the angular aberration is 1 mrad. Fig. 1 Example of a 4f correlator system that uses Fourier transform to implement an existing optical convolutional neural network (OCNN). The mask represents the Fourier transform of the kernel used in the CNN.

Fig. 2 Example of a simple OCNN with corresponding mathematical formula; ai (l) represents the i-th input or output node in the l-th layer; wij indicates the weight connecting the j-th input node and the i-th output node; bi is the i-th bias; Nm is the number of weights connected to an input/output or the size of a kernel; σ is a sigmoid function

To understand the architecture of the proposed SOCNN, it is essential to grasp the concept of CNN. An example of a CNN is shown in Figure 2; it shows four input nodes, four output nodes, and their synaptic connections along with mathematical representations. The CNN operates by receiving input signals through the input nodes, which are then transmitted through synaptic connections to the output nodes for processing; thus, the output is obtained. The strengths of the synaptic connections are modeled using mathematical representations that assign weights to each connection. In contrast to the full connection optical neural network such as the LCOE, in the CNN, each input or output node has local or partial connections whose weights are called kernels.

The concept of the SOCNN proposed herein is illustrated in Fig. 3(a). The CNN shown in Fig. 2 was transformed into a hardware schematic, containing laser diodes (LDs), lenses, a liquid crystal display (LCD), detectors, and electronics. The input node was replaced by an LD that sent three rays to lens array 1. The LD used in this architecture can be a multimode laser diode or a light-emitting diode (LED) because, unlike the traditional 4f correlator system, this system accommodates incoherent light sources.

Lens array 1 collimates the rays and sends them to the LCD, where each pixel transmits the corresponding ray according to a pretrained kernel in the CNN. The rays from the LCD pass through lens array 2, which focuses the rays and generates different ray angles depending on the distance of the LCD pixel from the optical axis of the individual lenses in the array.

A detector collects or adds the optical power of the rays arriving at different angles from different neighboring LDs or inputs with preset weights. In this scheme, the summed light is mathematically a convolution of the inputs and the kernel specified by the weights. These SOCNN can perform the calculations in parallel and, most importantly, in one step with the speed of light if the weights and inputs are preset. This type of calculation is called "inference" in the neural network community. Although the SOCNNs based on an LCD are reconfigurable, they are more suitable for inference applications owing to the low switching speed.

Detailed examination of the optical system, as shown in Fig. 3-(a), indicates that lens2 and lens3 form a relay imaging system in which the LCD is typically positioned at the focal plane of lens2, while the detector is placed at the focal plane of lens3. This arrangement ensures that the LCD and detector planes are conjugated-in other words, each pixel on the LCD forms an image on the detector plane. By establishing this conjugate condition, the illumination area of each ray is more clearly defined, and the crosstalk between channels is reduced.

Additionally, if an LD is placed at the focal plane of lens1, lens1 and lens2 form a relay system, and an image of the LD is formed at lens3. Overall, the lens configuration shown in Fig. 3-(a) constitutes a Koehler illumination system [START_REF] Arecchi | Field Guide to Illumination[END_REF][START_REF] Greivenkamp | Field Guide to Geometrical Optics[END_REF]. Figure 3(a) shows dotted magenta lines that represent the chief rays from the perspective of the condenser system and marginal rays from the perspective of the projection system. The red dotted rectangular block located at lens3 represents the image of the light source. Lens2 and lens3 work together to form a projection lens in the Koehler illumination system. In contrast, the rays originating from the LD point of emission spread out over the detector plane, providing uniform illumination. Koehler illumination-based optical computers have several advantages over previously reported architectures based on Abbe illumination in terms of uniformity of the illumination and control of the beam divergence of light sources [START_REF] Ju | A scalable optical computer based on free-space optics using lens arrays and a spatial light modulator[END_REF].

The key difference between this SOCNN and the previous LCOE is that each input of the SOCNN has a relatively small number of connections to the output array, whereas the LCOE has a full interconnection. The feature of partial connection greatly relieves the constraint on the size of the input array. In fact, unlike the traditional 4f correlator system, this SCONN does not have any theoretical limit of input array size. Only the size of the kernel array is limited, since the SLM pixels used for the kernel are imaged through lenses that impose a constraint on SBP; this topic will be explained in detail in the discussion section.

In the example shown in Fig. 3, the number of LCD pixels belonging to each input node is equal to the number of output nodes to which the input is connected. The LCD pixels belonging to each input node can be called a subarray of the SLM. The size of the subarray is the same as that of the receptive field from the viewpoint of the output node. In the case shown in Fig. 3-(b), the subarray comprises a 3 × 3 array where the spacing between the pixels is d. Given that the spacing between the detectors is a, the magnification of the projection system should match the size of the SLM subarray. The magnification of the projection system in SOCNN is written as f3 / f2 using the notations shown in Fig. 3-(b).

If an 8 × 8 pixel area on the LCD is assigned to a single kernel, it can be connected to 64 output nodes. For instance, an LCD with 3840 × 2160 resolution can accommodate up to 480 × 270 input nodes, which translates to 129,600 inputs. Considering the parallelism of the SOCNN, its performance is equal to the number of pixels in the LCD. If the system has N × N inputs and an M × M kernel, it can perform N 2 × M 2 multiplications and N 2 × (M 2 -1) additions in a single step. If the SOCNN takes the full advantage of the LCD resolution, (N × M) 2 equals the total number of pixels in the LCD, and this number is immensely large in modern devices. This LCD can be replaced by other types of SLM arrays for achieving high speeds if a fast refresh rate of weights is required. Furthermore, since the transmission of SLM pixels in the SOCNN is proportional to the weight of the kernel, extra calculations for Fourier transform are not required, unlike in the 4f correlator system-this is another advantage of SOCNNs for use as reconfigurable OCNNs in the future. After the optical process, the detector converts the light into current, and the remaining steps such as signal amplification, bias addition, and application of nonlinear functions (e.g., sigmoid, rectified linear units, local response normalization, and max-pooling) are performed electronically. These nonlinear functions are better handled by electronics than by optics because of their inherent properties. However, when electronics are used, interconnections between farneighboring electronics should be minimized to avoid traffic congestion. As long as the electronics employed are local and distributed, the optical parallelism of the system remains unaffected. The electronic part, including the detectors, is similar to the concept of smart pixels [START_REF] Seitz | Smart Pixels[END_REF].

The proposed system is a cascading system, and it can be extended in the direction of beam propagation. The signal from the output node is directly connected to the corresponding input of the next layer, allowing a detector, its corresponding electronics, and an LD in the next layer to form a synaptic node in an artificial neural network. If the system has L layers, N 2 × M 2 × L calculations can be performed in parallel in a single step; this ability greatly increases the SOCNN throughput for continuous input flow.

In fact, the addition of incoherent light by a detector and an LCD cannot represent the negative weight of a kernel in a CNN. If coherent light and interference effects are used, the system can represent subtraction between inputs. However, the use of coherent light may complicate the system and increase noise. The previous OCNN based on a 4f correlator system used a coherent light source and an SLM to generate an input array. As mentioned in the introduction, this coherent source entails many problems such as latency, noncascadability, and noise. Handling negative weights with incoherent light sources in this study can be solved by using the "difference mode" as described in previous references [START_REF] Ju | A scalable optical computer based on free-space optics using lens arrays and a spatial light modulator[END_REF][START_REF] Glaser | Lenslet array processors[END_REF].

To implement the difference mode in SOCNN, two detectors are required for each output node, or lens3 with two separate channels indicated by a red dotted circle, as shown in Fig. 4, is required. The two optical channels separate the inputs with positive weights from those with negative weights. Each channel adds input values multiplied by their respective weights using optical means. Subsequently, subtraction between the two channels is performed electronically through the communication between neighboring electronics. Note that the weight in the negative channel should be zero when the corresponding positive weight is used and vice versa. For example, if 𝑤 20 and 𝑤 02 are positive, and 𝑤 11 is negative, as shown in Figure 3-(a), then Eq. 1 and Eq. 2 are used to represent the positive and negative weight calculations, respectively, as shown in Fig. 4.

𝑤 200 = 𝑤 20 , 𝑤 002 = 𝑤 02 , 𝑤 101 = 0 (Eq. 1) 𝑤 210 = 0 = 𝑤 012 , 𝑤 111 = -𝑤 [START_REF] Greivenkamp | Field Guide to Geometrical Optics[END_REF] (Eq. 2)

This subtraction scheme simplifies the structure but requires an additional channel.

To implement other functions such as multiple kernels for the same input array, more than two detectors can be used for a single lens3. The number of detectors corresponding to each lens is denoted as Np in Fig. 4. For easy reference, the detectors associated with each lens 3 can be referred to as a "page of detectors." Since Fig. 4 represents a one-dimensional configuration, Nm and Np can be generalized into M × M and P × P, respectively, in a two-dimensional configuration. In this case, the array size of the subarray corresponding to one input node becomes (M × P) 2 . Suppose that the spacing between pixels and that between the subarrays are denoted as d and a, respectively. The side length of the subarray is MPd. Since P detectors are arranged along length a, the detector spacing d2 is equal to Md. This means that the magnification of the projection system consisting of lens2 and lens3 should be M. Thus, a page of detectors can be implemented for difference mode or multiple kernels. Fig. 4 Difference mode configuration of the SOCNN; this mode can also be used for calculating multiple kernels for a single input array; a generalized mathematical formula is given, where Np represents the number of detectors corresponding to a single lens3.

Discussion

Although theoretically, the SOCNN has no limit on the input array size, scaling the size of the kernel is limited due to lens3. The limit on the scaling can be analyzed using the method described in a previous report [START_REF] Ju | A scalable optical computer based on free-space optics using lens arrays and a spatial light modulator[END_REF]. This analysis involves calculating the image spreading of the SLM pixel through the projection system in terms of geometric imaging, diffraction, and geometric aberration. The overlap between the images of neighboring pixels and the required alignment tolerance can be estimated from the calculated image size. The analysis begins by examining an example system with an SOCNN architecture, and this is followed by an exploration of the factors that limit the system scale-up and how diffraction and geometric aberration affect this scaling.

To simplify the analysis, we investigated the architecture shown in Fig. 3-(a) instead of that shown in Fig. 4. The scaling analysis can be easily generalized into a page detector scheme. The proposed SOCNN is a system with two-dimensional (2D) input and output and a four-dimensional (4D) kernel such that the numbers of pixels are N 2 , N 2 M 2 , and N 2 for the input array, SLM array, and output array, respectively, where N and M are the number of rows in the square input array and the kernel array, respectively.

If all the three array components have the same size, the densest part is the SLM array. Therefore, it is better to initially design an array of SLM or LCD pixels for an example system. For the example system, we assumed a 5 × 5 array for the kernel. We assumed that the SLM had 5 μm square pixels, which were placed with a period of 20 μm in a rectangular array. According to the notations of Fig. 3 (c), ε and d are 5 and 20 μm, respectively.

The 5 × 5 SLM subarray accepts light from a single light source through a single lens. The diameter of each lens in the lens array and the side length of the SLM subarray are both 100 μm, and it is denoted by a in Fig. 3-(b) and(c). The distance a is also the pitch of lens array 1, lens array 2, and the detector. Lens2 is supposed to have an f/# of 2. Because the SLM pixel was at the front focus of lens2 and the detector was at the back focus of lens3, the image of the SLM pixel was formed at the detector plane. Since detector pitch a = 5d, the magnification of the projection system should be 5. In general, if the subarray size of the kernel is M × M, a is equal to Md, and the magnification should be M.

The magnification of this projection relay system was a ratio of the focal length (f3) of lens3 to the focal length (f2) of lens2, i.e. f3/f2 = M. Therefore, the geometric image size of a pixel without aberrations and diffraction was Mε. Because the pitch of the SLM pixel array was magnified to the pitch of the detector array, the duty cycle of the image of the SLM pixel was ε/d or 25% of the detector pitch, which is the same as in the SLM pixel pitch. Therefore, the duty cycle of the geometric image in the detector pitch remains the constant regardless of M, which is the kernel size.

The real image of one SLM pixel was enlarged by diffraction and aberration in addition to the geometric image size. The beam diameter that determined the diffraction limit was the image size of the light source for the condenser system composed of lens1 and lens2 according to Koehler illumination concept. However, because the image size of the light source could be as large as the diameter of lens1 or lens2, the beam diameter of the relay system comprising these lenses should be assumed to be the diameter of lens2 (D = a). The spot diameter attributed to diffraction was 2 λ f3/D = 2 λ M f2/D = 2 λ M f2/#, which was approximately 10 μm for a wavelength of 0.5 μm. The duty cycle of the diffraction spread in the detector pitch can be obtained by dividing the size of the diffraction spot by the detector pitch Md. The duty cycle of the diffraction spread corresponds to (2 λ f2/#)/d or 10% in terms of either the SLM pixel pitch or the detector pitch. The general formula for the duty cycle of diffraction spread is independent of M. Hence, the duty cycle of the diffraction origin is kept constant for a fixed f2/# when scaling up.

The effect of geometric aberration on the image spread can be investigated by assuming that f2/# is fixed during scaling Since f3/# = M f2/#, f3/# increases with the scaling factor M. The spherical aberration, coma, astigmatism, and field curvature are proportional to the third power, second power, first power, and first power of 1/(f/#), respectively [START_REF] Greivenkamp | Field Guide to Geometrical Optics[END_REF][START_REF] Geary | Introduction to lens design: with practical ZEMAX examples[END_REF]. In other words, angular aberration 3 of lens3 decreases with scaling However, because f2/# remains constant, the angular aberration due to lens2 becomes dominant. If lens2 has an fnumber of 2 and comprises three elements, the angular aberration 2 is about 3 mrad. The image spread due to the geometric aberration is f3 2 = M f22 = M D2 (f22 = M 2 d (f22. Since the detector pitch is M d, the duty cycle of the image spread in the detector pitch due to geometric aberration is M (f22. This value increases with scaling When M and f2are 5 and 2, respectively, the geometric aberration of lens 2 with 3 elements accounts for 3% of the duty cycle. If the maximum duty cycle of the image spread is 40%, the maximum M is about 66. In this case, the alignment tolerance is a duty cycle of 25% because the geometric image size and the diffraction spread are 25% and 10%, respectively. The duty cycle of 25% corresponds to 5 m in SLM plane-this value is usually feasible to achieve in terms of optomechanics.

As more elements are used for lens3, the angular aberration can be reduced, and the maximum M can be increased. However, for a larger number of elements, tighter alignment tolerance and higher difficulty for the assembly of the lens unit are necessary. From the viewpoint of a full connection optical neural network such as the LCOE, an M value of 66 may be small. However, from the viewpoint of OCNN, which aims for partial connection, an M value of 66 is very large.

In addition, if M is a relatively small number, as is usually the case for kernels in practice, the burden of optics and their alignment can be drastically reduced. For instance, if M = 5, a simple planoconvex lens can be used for lens2. For a planoconvex lens with an f-number of 8, the angular aberration is only 3 mrad, which is the same as that of an f/2 lens with three elements. In this case, when using the abovementioned SLM pixels, lens2 and lens3 have focal lengths of 800 μm and 4.0 mm, respectively, with a diameter of 100 μm. Generally, a larger M value results in a substantial increase in the focal length of lens3 since f3 = M 2 (f2/#) d. However, for small M values, f3 is within a reasonable length and may simplify the optics.

Further, the tangent to the half-field angle of lens3 can be obtained by dividing a half-field size M a/2 by f3. Since f3 = M f2, the tangent of the half-field angle is a/(2 f2) = 1/(2 f2/#); thus, it is independent of scaling factor M. When f2/# = 2, the half-field angle is about 14°, which is within a reasonable range. The half-field angle decreases as f2/# increases, implying less aberration and less of a burden for optics related to the field angle.

The concepts used in the proposed SOCNN architecture are related to those of lenslet array processors (LAP) [START_REF] Glaser | Lenslet array processors[END_REF]. A pseudoconvolution LAP in direct configuration was reported in [START_REF] Glaser | Lenslet array processors[END_REF]. The primary distinction between LAP and SOCNN is that SOCNN uses three layers of lens arrays with more emphasis on distributed electronics and neural network applications, while LAP uses only a single-layer lens array. From the viewpoint of illumination, the SOCNN is based on Koehler illumination, whereas the LAP is based on Abbe illumination [START_REF] Arecchi | Field Guide to Illumination[END_REF][START_REF] Greivenkamp | Field Guide to Geometrical Optics[END_REF]. Koehler illumination provides better uniformity in the detector area than Abbe illumination; this uniformity is especially beneficial when dealing with nonuniform sources such as the LED and multimode LDs. Unfortunately, a detailed description or design of the illumination scheme is not provided in [START_REF] Glaser | Lenslet array processors[END_REF]; such a description or design is critical to the convolution performance. The divergence of the light sources and their control are not specified for the input array in the pseudoconvolution LAP scheme, though they determine the coverage of convolution or the size of the kernel.

The parallel throughput of SOCNN depends on the size of the SLM, as mentioned in the theory. The SLM array can be divided into subarrays, depending on the size of the kernel array. For a given size of the SLM array, a smaller kernel size leads to a larger input array size. Additionally, the SLM can be divided to accommodate multiple kernels by copying the input array into multiple sections. In fact, each section of the SLM can handle different kernels and perform convolution in parallel [START_REF] Colburn | Optical frontend for a convolutional neural network[END_REF]. Therefore, the number of calculations per instruction cycle is equal to the number of SLM pixels. If the SLM has a resolution of 3840 × 2160, the total number of connections is approximately 8.3 × 10 6 , which is also the number of multiply and accumulate (MAC) operations in one instruction cycle. If electronic processing is assumed to be the main source of delay, with a delay time of 10 ns, the proposed optical computer in this study can achieve a throughput of 8.3 × 10 14 MAC/s. This throughput can be further increased by using multiple layers. Although multiple layers may cause delay in data processing, all layers perform calculations simultaneously similar to the pipelining technique used in digital computers. As the number of layers increases, the total throughput can also increase. Therefore, the proposed optical computer can achieve a throughput of 8.3 × 10 15 MAC/s when 10 layers are used.

To achieve massive parallel throughput, it is crucial to input 2D data in parallel at every instruction cycle. If the 2D input is generated by serially reconfiguring individual pixels of the SLM, the parallelism of the optical computer is considerably reduced, similar to the case of the 4f correlator system. The 4f correlator system suffers from serialization between layers when used in a cascading configuration because the input of the next layer can be generated by a single coherent source and SLM array. However, in the SOCNN architecture, this serialization problem occurs only in the first layer, not between the layers, as it can use incoherent, independent light sources for input.

One way to solve the issue of serialization is to start the first layer with a detector array and generate the 2D input using a real-time optical image, as suggested in a previous study [START_REF] Ju | A scalable optical computer based on free-space optics using lens arrays and a spatial light modulator[END_REF]. For instance, imaging optics can form images of a moving object on the detector array. These images serve as the first layer of an optical computer. Thus, serialization or deserialization of input data is unnecessary throughout the entire system because all the inputs in the following layers are fed from local electronics. This approach is similar to that of the human eye and brain, where the eye forms an image of the object on the retina, which is the first layer of the neural network that is connected to the brain. Since the SOCNN is used for the front-end of the optical neural network, this approach appears very reasonable.

Conclusions

Traditionally, 4f correlator systems have been used in optical computing to perform convolution by performing Fourier transforms using two lenses. However, these systems have limitations with regard to the scaling up of the input array because of geometric aberrations, and a single coherent light source together with an SLM is required to generate the input. This adds complexity and latency to the implementation of the multilayer OCNN with cascading configurations. In addition, the Fourier-transformed kernel used in the mask between two lenses requires extra calculation time and can cause latency in future systems that require higher refresh rates.

To address these issues, the SOCNN architecture was proposed. This architecture takes advantage of the Koehler illumination and comprises three lens arrays that form images of SLM pixels on the detector plane. The Koehler illumination scheme offers advantages over the previous Abbe illumination-based LAP architecture by providing more uniform illumination and lower crosstalk between the detectors.

The key advantages of the SOCNN are the scalability of the input array and the use of an incoherent light source. These advantages help avoid many problems inherent to the use of coherent sources, as in case of the 4f system. As a partial connection version of LCOE, the SOCNN inherits many advantages of LCOE, which is also based on free-space optics and Koehler illumination. Compared with the LCOE, the SOCNN has a smaller coverage of connection to the output; in the SOCNN, the use of the last lens limits only the kernel size, not the size of the input array. This is the major advantage of SOCNN over the existing 4f systems in terms of scaling up and parallel throughput of the system. Another advantage of the SOCNN is that the weights of the kernel are directly set by the proportional transmission of SLM pixels, unlike the 4f system, which requires Fourier transform and causes latency in the future reconfigurable system.

Although the SOCNN has an extensively scalable input array, there is a limit to scaling the kernel size because the kernel information spreads out through a lens. The scaling limit of the kernel array was by observing the effect of changes in geometric image size, diffraction, and geometric aberration on the final image size. As M, i.e., the number of rows in the kernel array, increases, the duty cycle of the geometric image size and diffraction spread in the detector pitch remain constant for a fixed f-number of lens2. In contrast, the duty cycle of image spread due to geometric aberration is proportional to M. When M is about 66, the duty cycle due to geometric aberration is equal to 40%, and the alignment tolerance has a duty cycle of 20%, which corresponds to 5 μm in the SLM plane. Usually, convolution does not require such a large array size, and hence, M = 66 seems sufficiently large for practical applications.

To estimate the parallel throughput of the SOCNN architecture, an example system was considered. The number of calculations per instruction cycle is equal to the number of SLM pixels. Assuming that electronic processing requires 10 ns and the system has 10 convolution layers, SLMs with a resolution of 3840 × 2160 can achieve a parallel throughput of 8.3 × 10 15 MAC/s.

In summary, a SOCNN based on free-space optics and Koehler illumination was proposed to overcome the challenges in the previous 4f correlator system. The results reported herein imply that the SOCNN can offer a multilayer CNN with massive optical parallelism.

Fig. 3

 3 Fig. 3 Scalable optical convolutional neural network (SOCNN) based on Koehler illumination and free-space optics using lens arrays and a spatial light modulator: (a) Schematics and the corresponding mathematical formula; (b) three-dimensional (3D) view of an example of the system with 3 × 3 inputs and 3 × 3 outputs; (c) the structural parameters of the SLM pixels and its subarrays

Declarations

The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.

The authors have no relevant financial or non-financial interests to disclose.

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.