David J Moss

Ultra-high bandwidth optical neural networks with Kerr soliton crystal microcombs: Scaling the network to the PetaOp regime

Keywords: Optical neural networks, neuromorphic processor, microcomb, convolutional accelerator

. Optical neural networks (ONNs) in particular can significantly increase the computing speed of ANNs in order to overcome the intrinsic bandwidth bottleneck of electronics. Convolutional neural networks (CNNs), are inspired by biological systems such as the visual cortex, and are a powerful approach to greatly reduce the parametric network complexity in order to enhance the accuracy of the predictions of the system. In this paper, we demonstrate a universal optical convolutional accelerator that can be used in conjunction with both electronic and optical neural networks. It operates beyond 10 Tera-OPS (TOPS -operations per second) and produces convolutions of extremely large scale images of 250,000 pixels in size with a resolution of 8-bits. It generates 10 convolutions simultaneously in parallel, with 10 different kernels. This processing simultaneouslyenough for facial image recognition. After demonstrating this, we then use the exact hardware to form a convolutional neural network consisting of a convolutional front-end followed by a deep optical neural network fully connected layer, together forming a CNN with ten neurons at the output. We successfully perform the recognition of all 10 hand written digits, each consisting of 900 pixel handwritten digit images. We achieve an accuracy of 88% which is very close to the theoretical accuracy of 90%. We use an approach that exploits the simultaneous multiplexing, or interleaving, within the time, space and wavelength dimensions, using an optical frequency comb supplied by an integrated Kerr micro-comb source. We compare the performance of different optical neural networks, explicitly showing that our approach is intrinsically scalable in both size and speed, up to the PetaOPs per second (POPs) regime in speed and to well over 24,000 synapses in size. We perform theoretical evaluation of the scaled system performance and show that it is trainable to much more complex networks for real-world demanding applications including real-time video recognition and autonomous unmanned vehicle control.

INTRODUCTION

Artificial neural networks (ANNs) can perform highly complex operations for decision making in applications such as medical diagnosis, playing board games, speech translation, face recognition, and much more [START_REF] Lecun | Deep learning[END_REF][START_REF] Schalkoff | Pattern recognition[END_REF][START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF][START_REF] Silver | Mastering the game of Go without human knowledge[END_REF]. They consist of collections of nodes that have connections that are weighted according to proper feedback in order to adjust the network parameters, so the system can "learn" while it performs. Classical fully connected networks face significant processing challenges, particularly for advanced applications such as extremely high-dimensional data. In order to dramatically simplify the tasks, convolutional neural networks (CNNs) can distill the input data representations from their raw form, in order to predict their properties with greatly simplified parametric complexity as well as highly improved accuracy [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF]. CNNs are inspired by the behavior of biological systems such as the visual cortex, and have been widely successful for many applications such as pattern recognition, natural language processing, computer vision, and many other areas [START_REF] Yao | Fully hardware-implemented memristor convolutional neural network[END_REF][START_REF] Lawrence | Face recognition: A convolutional neural-network approach[END_REF].

The power of ANNs is determined by the computing capability of the neuromorphic component hardware. In this regard, optical neural networks (ONNs) [START_REF] Shen | Deep learning with coherent nanophotonic circuits[END_REF][START_REF] Larger | High-speed photonic reservoir computing using a time-delay-based architecture: Million words per second classification[END_REF][START_REF] Peng | Neuromorphic Photonic Integrated Circuits[END_REF][START_REF] Lin | All-optical machine learning using diffractive deep neural networks[END_REF][START_REF] Feldmann | All-optical spiking neurosynaptic networks with self-learning capabilities[END_REF][START_REF] Xu | Photonic perceptron based on a Kerr microcomb for scalable high speed optical neural networks[END_REF][START_REF] Xu | 11 TOPs photonic convolutional accelerator for optical neural networks[END_REF][START_REF] Feldmann | Parallel convolutional processing using an integrated photonic tensor core[END_REF][START_REF] Zhou | Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit[END_REF][START_REF] Shastri | Photonics for artificial intelligence and neuromorphic computing[END_REF][START_REF] Wu | Artificial intelligence accelerated by light[END_REF][START_REF] Wetzstein | Inference in artificial intelligence with deep optics and photonics[END_REF] in particular are highly promising approaches towards next-generation neuromorphic systems for computation, since they can overcome the intrinsic speed bottleneck of electronic ANNs [START_REF] Yao | Fully hardware-implemented memristor convolutional neural network[END_REF][START_REF] Ambrogio | Equivalent-accuracy accelerated neural-network training using analogue memory[END_REF][START_REF] Esser | Convolutional networks for fast, energy-efficient neuromorphic computing[END_REF][START_REF] Graves | Hybrid computing using a neural network with dynamic external memory[END_REF][START_REF] Miller | Attojoule Optoelectronics for Low-Energy Information Processing and Communications[END_REF] and so can realize ultra-high speed computing, because of the extremely wide optical bandwidths available, particularly the >10 THz wide optical telecommunications C and L bands [START_REF] Xu | Photonic perceptron based on a Kerr microcomb for scalable high speed optical neural networks[END_REF]. ONNs have attracted huge interest and have formed the basis of major breakthroughs that have been recently reviewed [START_REF] Xu | Photonic perceptron based on a Kerr microcomb for scalable high speed optical neural networks[END_REF][START_REF] Xu | 11 TOPs photonic convolutional accelerator for optical neural networks[END_REF][START_REF] Feldmann | Parallel convolutional processing using an integrated photonic tensor core[END_REF][START_REF] Zhou | Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit[END_REF][START_REF] Shastri | Photonics for artificial intelligence and neuromorphic computing[END_REF][START_REF] Wu | Artificial intelligence accelerated by light[END_REF][START_REF] Wetzstein | Inference in artificial intelligence with deep optics and photonics[END_REF]. Operating in analog mode, they largely circumvent the limitations inherent in the time and energy taken up during storing and reading data to memorya well known drawback of classic computer architectures which is known as the von Neumann bottleneck [START_REF] Ambrogio | Equivalent-accuracy accelerated neural-network training using analogue memory[END_REF]. Significant progress has been achieved in addressing this through the use of high speed, trainable, and highly parallel ONNs [START_REF] Shen | Deep learning with coherent nanophotonic circuits[END_REF][START_REF] Larger | High-speed photonic reservoir computing using a time-delay-based architecture: Million words per second classification[END_REF][START_REF] Peng | Neuromorphic Photonic Integrated Circuits[END_REF][START_REF] Lin | All-optical machine learning using diffractive deep neural networks[END_REF][START_REF] Feldmann | All-optical spiking neurosynaptic networks with self-learning capabilities[END_REF][START_REF] Xu | Photonic perceptron based on a Kerr microcomb for scalable high speed optical neural networks[END_REF][START_REF] Xu | 11 TOPs photonic convolutional accelerator for optical neural networks[END_REF][START_REF] Feldmann | Parallel convolutional processing using an integrated photonic tensor core[END_REF][START_REF] Zhou | Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit[END_REF][START_REF] Shastri | Photonics for artificial intelligence and neuromorphic computing[END_REF][START_REF] Wu | Artificial intelligence accelerated by light[END_REF][START_REF] Wetzstein | Inference in artificial intelligence with deep optics and photonics[END_REF][START_REF] Appeltant | Information processing using a single dynamical node as complex system[END_REF][START_REF] Chang | Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification[END_REF][START_REF] Vandoorne | Experimental demonstration of reservoir computing on a silicon photonics chip[END_REF][START_REF] Brunner | Parallel photonic information processing at gigabyte per second data rates using transient states[END_REF][START_REF] Tait | Demonstration of WDM weighted addition for principal component analysis[END_REF]. These are further enhanced by the potential for full integration on a single photonic integrated circuit [START_REF] Shen | Deep learning with coherent nanophotonic circuits[END_REF][START_REF] Feldmann | All-optical spiking neurosynaptic networks with self-learning capabilities[END_REF][START_REF] Xu | 11 TOPs photonic convolutional accelerator for optical neural networks[END_REF][START_REF] Feldmann | Parallel convolutional processing using an integrated photonic tensor core[END_REF] which would yield computational densities that are extremely high. Nonetheless, despite these breakthroughs there is still significant room for advancement for ONNs. Practical real-world computer vision functions require the ability to process data at extremely large scales, and this is still extremely difficult because these systems largely contain fully connected networks where the input scale is governed primarily by hardware parallelism. This yields fundamental tradeoffs between the network footprint and its scale. Furthermore, ONNs so far have largely failed to demonstrate the extreme computing speeds that analog photonics can achieve.

Here, we report full experimental results for an optical convolution accelerator that has the ability to process and compress extremely large-scale data and at ultra-high speeds. By interleaving time, wavelength, and spatial dimensions through the novel application of integrated Kerr frequency soliton crystal micro-combs [29 -78], we realize a vector computing speed of 11.322 Tera-Ops per second (TOPS) and then demonstrate the processing of images that are 250,000 pixels in size, achieving the simultaneous generation of 10 convolution kernels. The convolution accelerator is completely reconfigurable in a fully dynamic sensewithout requiring any change in hardware. It is also intrinsically highly scalable. We show that the exact same hardware can operate simultaneously as both a front-end convolutional accelerator generating many parallel kernels simultaneously, as well as performing as a deep optical fully connected convolutional neural network. The CNN is then used for recognition of the full handwritten image set consisting of all ten digits (0-9). We achieve 88% accuracy which is very close to the theoretical accuracy of 90%. Finally, we present detailed architectures that will enable scaling the network both in speed to the Peta-OP per second (POP) regime as well as in size and scale to > 24,000 synapses, which is enough for almost all conceivable applications. We achieve this design through the use of existing commercially available components and equipment based on the S,C,L telecommunications wavelength bands. We theoretically evaluate the scaled systems achieving highly promising results.

Our approach towards optical neural networks is a major breakthrough towards achieving fully monolithically integrated ONNs, and is fundamentally enabled by the integrated Kerr soliton crystal microcomb. Futher, our approach is universal and stand-aloneit is fully compatible with both optical and electrical systems and hence it is able to function as a universal front end for ultrahigh bandwidth data compressing for any type of neuromorphic hardwareeither optical or electronic. Our approach will make massive-scale data machine learning for ultrahigh speed data processing in real-time a reality.

PRINCIPLE OF OPERATION

Figure 1 illustrates the principle of operation of our optical convolutional accelerator (CA). It contains high-speed electrical input signals and data port outputs. Figure 2 illustrates the experimental configuration for the convolutional accelerator for vector operation. First, the input data vector (X) is encoded serially where the temporal symbol intensity is represented by an electric waveform having a symbol rate of 1/τ (baud), with τ being the symbol period. The convolution kernel is similarly represented by a weight vector W having a length of R, that encodes the optical microcomb power of the lines through the use of spectral shaping with a commercially available system (Waveshaper). Next, the replicas weighted by W are generated by multicasting the time dependent waveform X onto the kernel wavelengths with an electro-optic modulator. Following this, the optical signal is then propagated through a delay, achieved with standard single mode fiber with a dispersion such that the delay step between the adjacent channels wavelength equals the duration of the symbols of X, thus interleaving the wavelength and time dimensions. Finally, the weighted and delayed copies of the signal are detected via high speed photodetection, effectively summing all of the signals in a particular time-slot, with each slot containing the convolution between W and X for each convolution window. Hence, the convolution window then essentially slides along at the baud rate of X, or the modulation speed. The computing speed is given by 2R/τ OPS, where each symbol output is generate by R multiply-and-accumulate (MAC) operations over a processing time τ. Since the processing speed in our approach directly scales as the number of wavelengths times the symbol, or baud rate, the speed can be dramatically enhanced to achieve the TOP, and even the Peta-OP, regime through the use of massively parallel channels of wavelengths generated by a microcomb chip. Importantly, the length of X, the input data, is unlimited. Hence, the convolution accelerator can operate on data at arbitrarily large scales, only limited by the accompanying electronics. Likewise, the number of kernels and their length are also essentially unlimited, subject only to the number of channels or wavelengths. We perform a parallel convolution of many kernels by adding extra sub-bands with each kernel having R wavelengths. After multicasting followed by the dispersive delay, the kernels (sub-bands) are then demultiplexed and separately detected using very high speed photodetectors. This generates a unique and separate electronic waveform, one for every kernel.

Considering that the convolutional accelerator fundamentally operates on vectors. However, they can in fact perform matrix operations that are needed for processing images, for example, by first effectively flattening the matrix into a one dimensional vector. The exact method that this is performed governs the "stride" of the sliding convolutional window as well as the effective computing speed of the matrix operations. The method used to flatten the matrix determines the receptive field (convolution slot) to slide with a horizontal stride of unity, so that every input element of the matrix has a corresponding convolutional output, together with a vertical stride that scales with the convolutional kernel size. A large vertical stride essentially produces sub-sampling of the raw input matrix vertically, which is effectively a partial pooling function [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] on top of the convolution itself. This effectively reduces the matrix computing speed, which is equal to an equivalent overhead cost, that inversely scales with the kernel size. Hence, a 3x3 kernel, for example, yields a reduction in speed by a factor of 3, equivalent to an overhead of 1/3. While this reduction can be avoided with a number of approaches in order to perform convolutions with a symmetric stride, thus avoiding any reduction in speed, this in fact is actually not necessary for most applications.

We follow a common approach where the raw input matrix is first sliced horizontally into multiple sub-matrices, each with a height equal to that of the convolutional kernel. The sub-matrices were then flattened into vectors and connected head-to-tail to form the desired vector (Figure 3). This flattening method equivalently makes the receptive field slide with a horizontal stride of 1 and a vertical stride equal to the height of the convolutional kernel. We note that a small stride (such as a horizontal stride of 1) ensures that all features of the raw data are extracted, while a large stride (3 or 5) reduces the overlap between the sliding convolution windows and effectively subsamples the convolved feature maps, thus partially serving as a pooling function. The AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF][START_REF] Bishop | Neural networks for pattern recognition[END_REF] used a stride of 4.

We note that although the homogeneous strides are generally used more often in digitally implemented CNNs, inhomogeneous convolution strides (unequal horizontal and vertical strides) such as those used here are often used and in most cases, including our experiments, do not limit the performance. Our performance was verified by the high recognition success rate of the CNN for full 10 digit prediction. Further, if desired homogeneous convolutions can be achieved by duplicating the weight-and-delay paths (each including a modulator, a spool of dispersive fibre, a demultiplexer and multiple photo-detectors) of the accelerator.

Figure 4 shows the system configuration for matrix processing of images, which is similar to the vector processor (Figure 2) except that the two dimensional images first need to be flattened into vectors.

Finally, our method is extremely flexible as well as highly reconfigurable without the need to change any hardware. We show that the same hardware that is used for image processing with the convolutional accelerator can also be used as a deep learning optical CNN. We demonstrate the performance of this CNN system in separate experiments. The hardware for the convolutional accelerator thus forms the basis of both the input convolutional accelerator processor as well as the subsequent fully connected CNN neuron layer (see below). The network achieves matrix multiplications by sampling each slot of the output waveform, because the vector dot product is equal to the special convolution of two input vectors W and X having the same lengths.

A detailed photonic convolution accelerator that operates in two distinct modes is shown in Figure 5. The left panel illustrates the system when it is operating in convolutional accelerator mode, suitable for both stand-alone large scale convolutional image processing as well as serving as the front end convolutional input of the CNN. The right side of Figure 5 depicts the architecture of the network when it is performing matrix operations for the fully connected optical CNN layer. Since the experimentally demonstrated networks that we use are too complex to be clearly presented, in Figure 5 we show simplified versions of the weights and input data in order to illustrate the system operation principle. The lengths of X and W that we use in Figure 5 are L = 13 and R = 4 for convolutions, and R = L= 4 for matrix operations of the fully connected layer, respectively.

The photonic convolution accelerator schematic is shown on the left side of Figure 5. First, the input vector data of length L and the weight vector of length R are multiplexed in the time and wavelength dimensions, respectively. The input data vector is contained in the temporal symbol intensities of a consecutive electrical waveform X[n] (n represents the temporal positions of the discrete symbols, n ∈ [1, L+R-1]), where X[n] is the accelerator electrical input. The weight vector of the kernel is then imposed on the optical power of the shaped microcomb lines, represented by W[R-i+1], for the i th wavelength channel (i ∈ [1, R], with i increasing with wavelength). The input electrical waveform X[n] is first multicast across all of the the shaped microcomb lines using an electro-optic modulator. Hence, the weighted replica of the i th wavelength channel is given by W

[R-i+1]• X[n].
Next, the optical signals of all channels, or wavelengths, are sequentially shifted in time by using an optical time-of-flight buffer. This produces a wavelengthdependent delay which effectively introduces a dispersion or time-step delay between adjacent wavelengths of τ (the difference in delay between wavelengths), which is designed to be equal to the duration of each symbol (the reciprocal of the Baud rate) of X[n]. Hence, the shifted replica is given by W

[R-i+1]• X[n-i].
Finally, the replicas for all of the wavelengths are summed together by the photo-detector, and the result is given by

[ ] [ ] [ ] ( )[ ] 1 R i n n i Ri n         Y W X W X (1)
where each calculated symbol Y[n] within the range of [R+1, L+1] represents the dot product between the kernel W and a region of X given by the sliding receptive field, [n-R : n-1] or [n-R, n-R+1, n-R+2, …, n-1]). By reading the different time slots of the output waveform, a convolution is obtained between the input data and the weight vector. This generates the feature maps, or output matrix convolutions, of the input image. Note that any high order dispersion of the dispersive delay will tend to degrade performance. In our experiments we paid particular attention to this to ensure that it was not an issue.

The convolutional accelerator can also perform matrix multiplications, as shown on the right side of Figure 5. These operations can be viewed as a special case for convolutional operations where the two input vectors, given by the flattened and pooled feature maps, together with the flattened synaptic weights of the fully connected layer, are equal in length (R=L). Figure 5 depicts a case where R=L=4. For this case, we assume that the input data vector, given by XFC[n], and the weight vector given by WFC[R-i+1], both have the same length given by R (i

∈ [1, R], n ∈ [1, R]).
Therefore, from Eq. 1, the output waveform following photodetection is given by

1 [ ] [ +1] [ ] R FC FC FC i n R i n i       Y W X (2)
By sampling at the time slot denoted by n=R+1, the matrix multiplication of the two input vectors is then given by

1 1 [ 1] [ +1] [ 1 ] [ ] [ ] R R FC FC FC FC FC i i R R i R i i i            Y W X W X (3)

EXPERIMENT 3.1 Soliton crystal optical frequency micro-combs

Optical frequency combs, with arrays of equally spaced discrete frequencies lines, are extraordinarily useful for many applications -particularly optical frequency metrology [START_REF] Pasquazi | Micro-combs: a novel generation of optical sources[END_REF]. Micro-combs provide the full capability of mainframe optical frequency combs, but in a compact and integrated form having an extremely small footprint [START_REF] Pasquazi | Micro-combs: a novel generation of optical sources[END_REF][START_REF] Moss | New CMOS-compatible platforms based on silicon nitride and Hydex for nonlinear optics[END_REF][START_REF] Kippenberg | Dissipative Kerr solitons in optical microresonators[END_REF][START_REF] Savchenkov | Tunable optical frequency comb with a crystalline whispering gallery mode resonator[END_REF][START_REF] Spencer | An optical-frequency synthesizer using integrated photonics[END_REF][START_REF] Marin-Palomo | Microresonator-based solitons for massively parallel coherent optical communications[END_REF][START_REF] Corcoran | Ultra-dense optical data transmission over standard fiber with a single chip source[END_REF]. Microcombs have underpinned a great number of ground-breaking achievements in many areas, including optical frequency high-resolution synthesizers [START_REF] Spencer | An optical-frequency synthesizer using integrated photonics[END_REF], ultrahigh bit rate telecommunications [START_REF] Marin-Palomo | Microresonator-based solitons for massively parallel coherent optical communications[END_REF][START_REF] Corcoran | Ultra-dense optical data transmission over standard fiber with a single chip source[END_REF], high dimensional and complex quantum state generation [36 -44], high performance microwave and RF signal generation, detection, and processing [68 -88], and much more. Figure 6 shows an illustration of the micro-comb chip together with output spectra with the accompanying optical pumping curves. Our work is based on a type of micro-comb that has been called "soliton crystals" because they exhibit a profile that resembles a crystal structure with tightly packed self-localized pulses in the angular dimension in the ring [START_REF] Corcoran | Ultra-dense optical data transmission over standard fiber with a single chip source[END_REF][START_REF] Cole | Soliton crystals in Kerr resonators[END_REF][START_REF] Wang | Robust soliton crystals in a thermally controlled microresonator[END_REF]. They naturally form in micro-cavities that have particular dual mode crossings, and most importantly do not require any complicated pumping dynamics, nor do they require any external stabilization methods. They are governed by the Lugiato-Lefever equations [START_REF] Pasquazi | Micro-combs: a novel generation of optical sources[END_REF][START_REF] Lugiato | Nonlinear Optical Systems[END_REF] and exhibit unique optical spectra (Fig. 6f) profiles typically termed "palm shaped", resulting from the spectral interference between the tightly packed solitons that circulate around the micro-ring. Soliton crystals obey highly reliable and repeatable deterministic generation that results from interference induced by the mode crossing on the background wave and the very high optical power in the cavity (Fig. 6c). This allows a very simple and reliable initiation even with just simple and slow adiabatic pump wavelength tuning. [START_REF] Corcoran | Ultra-dense optical data transmission over standard fiber with a single chip source[END_REF] In fact, soliton crystals can even be generated by using simple manual tuning of the pump laser wavelength. Figure 6d shows the intracavity optical power inside the micro-ring during the pump sweeping process. The reason that it is possible to slowly or adiabatically sweep the pump is because the optical power inside the cavity is over thirty times higher than that of single dissipative Kerr solitons (DKS), to the point where it is very close to the power of the background spatio-temporal chaos states [START_REF] Pasquazi | Micro-combs: a novel generation of optical sources[END_REF][START_REF] Corcoran | Ultra-dense optical data transmission over standard fiber with a single chip source[END_REF] from which they emerge. Therefore, soliton crystals have negligible thermal detuning that, for the case of DKS states, arises from the 'soliton step' at threshold and which leads to instability. This makes resonant pumping of DKS states extremely difficult, requiring dynamic pump wavelength and power "kicking" often with reverse tuning. It is this combination of high conversion efficiency together with the ease of generation that results in soliton crystals being highly useful. Our coherent optical soliton crystal microcomb (Figure 6) was a result of optical parametric gain followed by oscillation in a single integrated MRR (Fig. 6a,6b). The device was fabricated in CMOS-compatible Hydex [START_REF] Miller | Attojoule Optoelectronics for Low-Energy Information Processing and Communications[END_REF][START_REF] Appeltant | Information processing using a single dynamical node as complex system[END_REF][START_REF] Corcoran | Ultra-dense optical data transmission over standard fiber with a single chip source[END_REF], and achieved a Q factor greater than 1.5 million, with a radius of 592 μm, and a very low free spectral range (FSR) of ~ 48.9 GHz. The pump laser power was amplified by an erbium doped optical fiber amplifier (Pritel PMFA-37) to produce soliton crystal microcombs with more than 90 channels across the telecommunications C-band (1540-1570 nm). It resulted in very low-noise frequency microcomb lines produced by a device having an extremely small size of less than one square millimeter, with a power consumption of less than 100 milliwatts using the technique in [START_REF] Corcoran | Ultra-dense optical data transmission over standard fiber with a single chip source[END_REF].

The power fluctuations of the microcomb were evaluated by measuring the power stability of the comb lines over 66 hours, with the optical spectrum captured every 15 minutes (Figure 7). The extracted relative standard deviation was around ̶ 14 dB over 66 hours, indicating that the microcomb source can well serve as a stable multi-wavelength source for our convolutional accelerator.

Matrix Convolution Accelerator

The experimental arrangement for the convolutional accelerator operating in full matrix mode in order to operate on classic 500×500 face image is shown in Figure 4. The system generates 10 parallel convolutions simultaneously with 3×3 kernels in order to produce the distinct image processing functions. The weight matrices for all ten kernels were flattened into a composite kernel vector represented by W that embodies all of the 90 weights (10 kernels with 3x3=9 weights each). These were all then encoded onto the optical power of all ninety microcomb lines through the use of an optical spectral shaper (Waveshaper). Each kernel occupied its own frequency band consisting of nine wavelengths. The wavelength channels were generated by a coherent soliton crystal microcomb source that operated via optical parametric oscillation within a single micro-ring resonator (MRR) (Fig. 6b) with a radius of 592 μm, a FSR spacing ~ 48.9 GHz and with an optical bandwidth of ~ 36 nm for 90 wavelengths in the C-band (1540-1570 nm) [START_REF] Corcoran | Ultra-dense optical data transmission over standard fiber with a single chip source[END_REF].

The experimental results of the image processing are shown in Figure 8, where Fig. 8a shows the kernel weights and the optical spectrum of the shaped microcomb, while at the same time the electrical input waveform of the image (grey lines are theoretical while the blue lines are the experimental waveforms) are in Figure 8b. Figure 8c depicts the convolved results of the fourth kernel that consists of a top image processing Sobel function (grey lines = theory, red = experimental). Finally, Figure 8d displays the matrices of the corresponding recovered images and weight kernels.

The raw input face image that is 500×500 pixels in size was first electronically flattened into a vector X and then encoded as the intensities of the 250,000 temporal symbols, each having a resolution of 8 bits/symbol (limited by the electronic arbitrary waveform generator (AWG)), in order to form the electrical input waveform through the use of a high-speed electrical digital-to-analog converter, at a data rate of 62.9 Giga Baud (time-slot τ =15.9 ps) (Fig. 8b). The duration of each waveform was 3.975µs for each image, yielding a processing rate for all ten kernels of > 1/3.975µs, equivalent to 0.25 million ultra-large-scale images a second.

The input waveform (X) was then multi-cast onto all ninety shaped microcomb lines using an electro-optic modulator. This yielded replicas that were each weighted by the kernel vector W. Following this, the waveform was propagated across about 2.2 km of standard single mode optical fibre that had a standard dispersion equal to 17ps/nm/km. This length of fibre was chosen carefully to yield a sequential time shift in the weighted replicas with a progressive delay step of 15.9 ps between the adjacent wavelengths. This precisely matches the length of each input data symbol τ, and so results in wavelength to time interleaving for all of the 10 kernels.

After this, the ninety wavelengths were de-multiplexed into ten sub-bands each consisting of 9 wavelengths, with every sub-band equating to each kernel, and with each separately detected by ten high speed photodetectors. Detecting each wavelength acts to sum the temporally aligned symbols of each replica, and the output electrical signal for each kernel is shown in Fig. 8c (for kernel 4). All 10 electrical waveforms for the kernels were then converted into digital waveforms using ADCs and then resampled so that each waveform time slot contained the vector dot product between a kernel convolutional matrix and the input image, all moving with a sliding receptive field, or window. The output signals thus contained the ten convolutional matrix outputs (feature maps) representing the hierarchical extracted features of the input image waveform (Figure 8d).

The convolutional accelerator was based on the combination of spatial wavelength and temporal multiplexing, with the convolution window essentially sliding across X, the input data signal, at 62.9 Giga Symbols/s, the modulation baudrate. Each output symbol is generated by nine MAC (multiply-and-accumulate) operations, which is the size of each of the kernels. Therefore, the throughput, or vector computing speed for each kernel is 2×9×62.9 = 1.13 TOPS, and since all 10 kernels were generated simultaneously in parallel, the net computing speed of the system is given by 1.13×10 = 11.3 TOPS, or 11.321×8=90.568 Tb/s. Note that the speed in terms of Tb/s is slightly reduced by the optical signal to noise ratio (OSNR), which acts to reduce the effective number of bits (ENOBs), but in our case this reduction was small. Note that this speed is over 500 times the fastest ONNs that had been reported when this work was first published, and only one system published since has been faster [START_REF] Zhou | Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit[END_REF].

For the experiments on matrix image processing reported here, the convolution window had a sliding vertical stride of 3 (because of the kernels being 3×3), and so the net matrix computing speed was reduced by 3 to 11.3/3=3.8 TOPs. The use of homogeneous strides with unity horizontal and vertical strides, that would result in full vector speed operation, can easily be accomplished simply by replicating the system in parallel with weight-and-delay paths [START_REF] Xu | 11 TOPs photonic convolutional accelerator for optical neural networks[END_REF]. As mentioned, however, we did not find this necessary in order to achieve our successful performance. For the experiments performed here, the input data had a length of 250,000 pixels, but in principle this is unlimited in sizethe only consideration is the practical limitation imposed by the capacity of the external electronics.

To implement the required weights for the kernels, the power of the soliton crystal microcomb lines were modulated through the use of 2 commercial spectral shapers (Finisar WaveShaper 4000S) that are based on liquid crystal on silicon (LCOS) technology. The first waveshaper flattened the microcomb lines while the second was positioned immediately before the photo-detectors, and accomplished the precise shaping of the come lines that was required by the kernel weights. We used a feedback loop to increase the comb shaping accuracy with an error signal obtained by measuring the system impulse response for an input pulse (Gaussian shaped) and comparison with the theoretical weights.

Both the theoretical and experimental facial image processing results produced by the matrix convolutional accelerator for all 10 kernels, are shown in Figure 9, for the 500×500 pixel (250,000) facial image, featuring both the recorded waveforms and the recovered images. The time dependent waveform at the operation baud rate was generated from the electrical input data through the use of an electronic arbitrary waveform generator (AWG) (Keysight M8195A). The electrical waveform was then converted to optical signals by multicasting it onto all wavelength channels simultaneously using an electro-optic intensity modulator (iXblue) having a bandwidth of 40GHz. We used sampling points at a speed of 62.9 Giga samples/s for the image processing to form the input symbols. All of the optical signals, propagating down the same single fibre, were then transmitted across 2.2 km of standard single mode fiber with a dispersion of 17ps/nm/km in the telecom band, that was chosen to generate a relative temporal delay between adjacent wavelengths 15.9 ps/channel, in order to exactly match the baud rate of the input data.

In order to realize the theoretically required kernel weights, the microcomb wavelengths were adjusted in power with a Finisar WaveShaper 4000S, based on liquid crystal on silicon (LCOS). Two WaveShapers were used -the first flattened the spectrum while the second device performed a fine tuning to adjust the comb power precisely to encode the kernel weights on to the microcomb lines. The second waveshaper was placed just in front of the photo-detector. We used a feedback loop to increase the accuracy of microcomb tap shapes, by generating an error signal by measuring the impulse response of the system to a Gaussian pulse input and then comparing it to the theoretical channel weights.

Performance Metrics

Since there are no conventions in this very young field for quantifying or classifying the processing computing speed or performance of ONNs, here we clearly outline the definitions for performance that we use to quantify the system performance. We adapt a method that is commonly used in the electronics world to evaluate the performance of microprocessors. Here, the convolution accelerator processing power, which is closely related to the operational bandwidth, is termed the data throughput, given by the number of operations performed within a given time period. Since in our system the weight vectors and input data follow different paths and are interleaved in temporal, wavelength, and spatial dimensions, we use the time sequence at the electrical output port to calculate the throughput in a clear manner.

At the electrical output port, the output waveform has a total of L+R-1 symbols (where L and R are the lengths of the data input vector and kernel vector weights, respectively), amongst which L-R+1 symbols consist of the results of the convolution. Each output symbol is the result of R MAC operations or 2R OPS, where the duration of the symbols τ is given by the waveform input symbol duration. Therefore, since L is typically much greater than R for practical neural networks, the term (L-R+1)/(L+R-1) does not affect the throughput, or vector computing speed, which is given by 2 1 2 1

R L R R LR     (OPS) (4)
Hence, the computing speed of the vector convolutional accelerator that we report here is given by 2×9×62.9×10 = 11.321 Tera-OPS, or TOPS, for 10 parallel convolutional kernels.

Many applications, such as speech or audio recognition, operate directly on vectors and in that case the computing speed of the system is given directly by the vector speed of 2R/ τ. On the other hand, when processing matrix data as is typical for images, one has to include the overhead that reduces the net computing speed as a result of the flattening process used to convert the matrix to a vector. This is directly related to the size of the convolutional kernel, with 3-by-3 kernels, for example, generating an overhead of 1/3 so that the effective computing speed is correspondingly reduced to ~ 1/3 * 2R/τ. In our case this still resulted in a matrix operation speed that was in the multiple TOP regime, a result of the high degree of leveraging and parallelism yielded by interleaving the spatial-temporal-wavelength dimensions.

The output waveform for each kernel of the convolutional accelerator (length = L-R+1=250,000-9+1=249,992) contains 166×498=82,668 useful symbols. These are then temporally sampled in order to obtain the feature maps, with the balance of the symbols being discarded. Hence, the net experimental speed of the matrix convolution process is slower than the corresponding vector speed by the overhead factor of 3, with a resulting net speed of 11.321 × 82,668/249,991 = 11.321 × 33.07% = 3.7437 TOPS.

For the deep learning CNN, the front end convolutional accelerator layer had a computing vector speed of 2×25×11.9×3 = 1.785 TOPS while the convolution matrix speed, where we use 5x5 kernels, was 1.785×6×26/(900-25+1) = 317.9 Giga-OPS. For the deep CNN fully connected layer, Eq. ( 4) shows that each neuron's waveform output has a length of 2R-1, with the relevant (ie., useful) symbol being the one located at R+1, also arising from 2R operations. Hence, the fully connected layer speed is 2R / (τ*(2R-1)) for each neuron, and so for R = 72 for the experiment, and with 10 neurons being calculated in parallel, the net matrix multiplication speed then becomes 2R / (τ*(2R-1)) × 10 = 2×72 / (84ps* (2×72-1)) = 119.83 Giga-OPS for our experiments.

Deep Learning Optical Convolutional Neural Network

Our convolutional accelerator is scalable as well as completely and dynamically reconfigurable without any changes to the hardware. Hence, we used the exact same hardware to achieve the stand alone accelerator as well as both components of the CNNnamely the front-end convolution accelerator followed by the fully connected layer, which both together comprised the deep optical CNN. Figure 10 shows the full deep (multiple) level optical CNN architecture. We used the system (optical CNN) to perform recognition of the full ten digit (0-9) handwritten images, a task that is rarely demonstrated with optical networks. The feature maps form outputs of the matrix convolutional accelerator front end while the fully connected layers embody the neural network component. The feature maps are input into the subsequent fully connected layers that embody the neural network of the system.

The optical CNN architecture (Figure 11) includes convolution, pooling, and fully connected layers. Figure 12 shows the detailed experimental schematic of the optical CNN. The left side shows the input front end convolutional accelerator on and the fully connected layer is depicted on the right -both together form a deep learning (more than one layer) optical CNN. The optical wavelengths were generated by the soliton crystal microcomb, that were used by both the convolution accelerator and fully connected layers. A digital electronic signal processing (DSP) module was used for the temporal sampling, with the pooling function being performed by an external computer, although this latter function can be readily done all-optically.

The convolutional layer (Figure 12, Left) generates the heaviest computing workload of the entire network, typically occupying anywhere from 55% to 90% of the total system processing power. The digit images consist of 30×30 matrices each, containing grey-scale quantities that are digitized with a resolution of 8-bits. These were then flattened into vectors and then converted into a temporal signal at a speed of 11.9 Giga Baud (time-slot τ =84 ps)about 5 times slower than the stand alone convolutional accelerator discussed previously. We employed three kernels, each of size 5×5, and so the system needed 75 wavelengths from the microcomb. This resulted in a vertical stride of 5. The required temporal dispersive delay was generated by 13 km of SMF in order to be commensurate with the input data baud-rate. The microcomb wavelengths were separated, or de-multiplexed, into the 3 kernels, each of which were then photodetected separately with high speed photodetectors, after which the electronic waveforms were sampled and finally scaled with a nonlinear transfer function using digital electronics to generate the output of the convolutional front-end which consisted of the extracted hierarchical feature maps of the input images. The feature maps were then pooled electronically and flattened into a vector (Eq. 2,3) X FC (72×1= 6×4×3) per image that was then sent to the fully connected layer.

The architecture for the CNN front end convolutional layer as well as the experimental results are shown in Figure 13. The left side illustrates the experimental configuration, and the right side reports the experimental results for just a single convolutional kernel. It shows the weighted optical spectrum of the microcomb together with the associated kernel weights. The blue lines represent positive synaptic weights while red lines are negative weights. The electrical waveform shown at the input is for the case of the digit "3". The centre panel illustrates the theory (grey lines) and experiment (orange lines), for the convolution results and the feature map outputs.

The fully connected layer featured 10 neurons, one each for each of 10 handwritten digits (0 to 9). The input consisted of synapses with the synaptic weights represented by a 72×10 weight matrix W FC (l) (ie., 10 column vectors each of size 72×1) for the l th neuron (l ∈ [1, 10])with the number of microcomb wavelengths [START_REF] Xu | Photonic RF phase-encoded signal generation with a microcomb source[END_REF] equal to the length of the flattened feature map vector X FC . The l th port had a shaped optical spectrum with the optical power distribution being determined by the weight vector W FC (l) , which therefore served as the effective optical input to the l th neuron. After multicasting onto all 72 wavelengths simultaneously with a single optical modulator, and then propagating through the single mode fiber which generated the sequential delays between wavelengths, the optical signals were then weighted and then spatially and spectrally demultiplexed into 10 spatial output ports using a single Waveshapereach corresponding to one neuron. Since this part of the network operated via linear processing, the kernel wavelength weighting can be performed at any stage of the network -either before EO modulation or later just in front of the photodetectors. The advantage of applying the weights at the end just before detection is that both the demultiplexing and weighting functions can be performed using only one Waveshaper rather than two. Finally, the different outputs of the nodes and neurons were extracted through sampling the 73 rd symbol of the convolved results. The final output of the optical CNN was contained in the intensities of the output neurons, with the neuron having the highest intensity yielding the predicted category of the tested image (Fig. 14). Figure 14 shows the architecture and experimental results for the fully connected layer. Here, the left side shows the experimental configuration which was very similar to the convolutional layer, while the right side reports the experimental results for a single output neuron, featuring the shaped comb spectrum (top); the pooled feature maps for the digit "3" and the corresponding electrical input waveform (theory is the grey lines and experimental results shown in red in the middle. The neuron output waveform and sampled intensities are shown on the bottom.

The external, or peripheral systems used in our experiments included signal sampling, the nonlinear transfer function and the pooling functions. Here, these functions were performed with electronic digital signal processing hardware. However, in principle most, if not all, of these functions could be achieved purely in the optical domain with the VCA.

The supervised network training, which only needs to be performed once for a given network task, was achieved beforehand using offline digital electronics.

We note that handwritten digit recognition, although widely employed as a benchmark test in digital hardware, is still (for full 10 digit (0 -9) recognition) beyond the capability of existing analog reconfigurable ONNs. For example, a hidden layer having 10 neurons would need 9000 physical paths, which represents a significant challenge for state-ofthe-art nanofabrication technologies. Our CNN is the first integrable and reconfigurable ONN that able to not only perform complex and high level operations, including the full 10 handwritten digit recognition, but also at Tera-OP per second (TOP) processing speeds. The CNN convolutional layer used 5 sample points at 59.421642 Giga Samples/s to generate each single symbol of the input waveform, which matched the progressive delay time (84 ps) generated by the 13km long dispersive fibre. The generated electronic waveforms for 500 testing images [START_REF] Xu | Photonic perceptron based on a Kerr microcomb for scalable high speed optical neural networks[END_REF] formed the electrical signal input of both the convolutional and fully connected layers.

For the convolutional layer of the CNN, we used 5 sample points at 59.421642 Giga Samples/s to form each single symbol of the input waveform, which also matched with the progressive time delay (84 ps) of the 13km dispersive fibre.

The generated electronic waveforms for 500 images are shown in Figures 15,16, which served as the electrical input signal for the convolutional (Fig. 15) and fully connected layers (Fig. 16).

The 2 nd Waveshaper performed two functions simultaneously. It weighted and shaped the different wavelengths according to the kernel weightings, but it also spatially de-multiplexed the different wavelength channels into separate spatial ports, dictated by the configuration of the different convolution kernels For both of the discrete CA operating on the 500×500 image processing, as well as for the convolutional layer component of the CNN experiments, the the second Waveshaper simultaneously performed the wavelength shaping as well as power splitting (versus de-multiplexing) for all of the 10 output neurons. The power-split or de-multiplexed spatial ports were detected sequentially and then measured. However, both of these two functions could easily be performed simultaneously in parallel using a 20-port optical spectral shaper that is now commercially available (WaveShaper 16000S, Finisar) combined with multiple photodetectors.

We achieved negative weights on the channels channel in two different ways. For the discrete convolutional accelerator (500×500 image processing) experiment and the convolutional component of the CNN, the wavelength channels for each kernel were divided into two parallel spatial output ports by the WaveShaper, depending on the polarity of the kernel weights, which were then detected using a Finisar XPDV2020 balanced photodetector. On the other hand, for the fully connected layer we achieved negative weights by encoding them onto the input electrical waveform symbols during the electrical digital processing stage. Both of these approaches used to achieve the negative weights were successful. Finally, the output electrical waveform was temporally sampled and then converted to digital form with a high-speed oscilloscope (80 Giga Symbols/s Keysight DSOZ504A) to yield the convolved output. The extracted outputs of the convolution accelerator part of the CNN were then digitally processed further. This included rescaling to account for the photonic link loss using a reference bit, which was then mapped onto a given range with a nonlinear tanh function. The functions of the pooling layer were also digitally implemented, using the algorithm introduced in the network model. The residual discrepancy between theory and experiment was very small, for both the recognition and convolution functions, and was mainly due to the slight deterioration in the waveform input induced by the limitations in performance of the (electrical) arbitrary waveform generator. Improving this will result in even higher accuracy and better agreement between experiment and theory.

Our input intensity resolution generated by the AWG was 8-bits, although this does not guarantee that the networks operate at 8-bits throughout the system. The intensity resolution (i.e., the bit-resolution for digital systems) for analog ONNs is subject to the electrical signal-to-noise ratio (ESNR), which in our case mainly arose from the signal-to-noise ratio (SNRin) of the input electrical signal generated by the electrical Arbitrary Waveform Generator (AWG), which we note is part of the measurement system rather than the network itself, as well as the noise figure (NF) of our system (ESNR=SNRin-NF). The ESNR measurement and noise figure were also affected by the noise floor of the electrical spectrum analyzer we used to analyze the output. We estimated the ESNR of the system by applying a digital high-pass filter to the output waveforms (measured by an oscilloscope) to obtain the noise power. We obtained an ESNR of 41.36dB, which is capable of supporting almost 7 bits. The ENOBs scale at one bit per 6dB of ESNR, and so to achieve 8-bit resolution the ESNR of the system would need to be 20•log10(28) = 48 dB, about 6.6 dB higher than what we had.

The ESNR can be increased by using a lower noise AWG or by increasing the optical single-to-noise ratio (OSNR) of the system -or both.

One further consideration is that the digital system intensity bit resolution in terms of effective number of bits (ENOBs) for analog ONNs is affected by the signal-to-noise ratio (SNR). For an ENOB of 8-bits, the system SNR must be greater than 20•log10(28) = 48 dB. Our system had an SNR greater than this and so our speed in Tb/s was very close to 8 x the speed in Ops, and not reduced noticeably by our OSNR. The OSNR is related to the ESNR (bit resolution) by 1/ESNR=A/OSNR+B, where A and B are empirical constants determined by the system. We measured the OSNR of the system at several points with a 150 MHz resolution optical spectrum analyser (Finisar WaveAnalyzer). We calibrated the WaveAnalyzer against a standard optical spectrum analyzer (OSA) with 0.1 nm resolution by noise loading a single laser line. Directly after the microcomb, the OSNR was > 33 dB; After the pre-flattening implemented by the first WaveShaper, the OSNR was 30 dB. To achieve full 8 bit resolution solely by increasing the OSNR, we would need to increase it by > 6.6 dB. This can be achieved in principle in a number of ways, including via a "comb distillation" approach through filtering the comb with an FSR matching passive MRR [START_REF] Corcoran | Overcoming low-power limitations on optical frequency combs using a micro-ring resonator[END_REF]. This would act to improve the intrinsic ENOBs of the system, versus using a lower noise AWG which would mainly affect the noise in measurement setup. We reiterate that the effect of noise and the slightly reduced ENOBs had little effect on the performance of the CNN since our accuracy was already within 2% of the theoretical maximum. Finally, the slightly reduced ENOBs would only affect the speed of our system in bits/s (about 90Tbs/s at full 8 bits) and not in OPs/s.

Network training and digital processing

For the deep learning optical CNN experiments we used the MNIST (Modified National Institute of Standards and Technology) datasets of handwritten digital images [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] that comprises 60000 images for the training set and 10000 images as the test set. Figure 10 shows the structure of the CNN which was determined empirically using trial-and-errora standard approach for neural networks. In our case designing the network was significantly helped by the fact that the number of synapses and neurons of the network could be dynamically reconfigured without the need to modify the hardware. The 28×28pixel images of the input data was first padded with zeros into 30×30 images and then sliced into 5×180 matrices which were then convolved with the 5×5 kernels. The slicing operation made the receptive field slide horizontally with a stride of 1 across the rows and a vertical stride of 5 over the columns of the 30×30 input data (= 900 input nodes). Following this, the 6×26×3 feature maps were then pooled (using an average pooling method) to create matrices with a smaller dimension of 6×4×3. Finally, the matrix was further flattened into a 72×1 vector that generated the input nodes to the fully connected layer. Finally, the fully connected layer calculated the predictions which were encoded into the output of the 10 neurons. We used a nonlinear tanh function following the convolutional layer, pooling function and fully connected layers. We chose this, rather than other nonlinear functions such as ReLU which are widely used, since it can be experimentally achieved using a saturating electrical amplifier.

The training required to determine the pre-trained weights and biases was achieved prior to the experiments, and performed offline using a digital computer. We used the well-known Back Propagation algorithm [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] to adjust the weights. To validate the hyper-parameters of the CNN, we performed a 10-fold cross validation with 60000 samples of the training dataset, separated into 10 subsets and each was then used to test the trained network (6000 samples) with the rest of the 9 subsets (54000 samples). The results are shown in Figure 15, showing the training curve (a) and the accuracy (b) of using the k th subset as the test set during the 10-fold cross validation for the full 10 digit image recognition by the deep CNN. The test sets [START_REF] Xu | 11 TOPs photonic convolutional accelerator for optical neural networks[END_REF] were processed by both the optical CNN for 500 images for each digit recognition, and an electronic computer (with 10000 images) for comparison, shown in Figure 16, illustrating the results of full 10 digit (0-9) recognition by the deep level optical CNN. (a) The offline training curve is shown in (a) and the generated predictions (confusion matrix) obtained by using an electronic computer to process the output of the CNN are shown in (b). The numbers represent the number of predicted labels versus the actual label (digit), hence the diagonal elements represent correct predictions.

CNN Experimental Results

Figure 17 shows the shaped impulse response of the convolutional layer that has 3 kernels and 75 wavelengths, or weights, in total. Figure 18 shows the shaped impulse responses for the ten neurons, each with 72 synapses, at the fully connected layer. Figures 19202122show the full experimental waveform results and the resulting confusion matrix (Figure 23) of the CNN for 50 tested images, Figure 19 (a) shows the more limited test set of fifty handwritten digits tested in the first round of experiments, with their corresponding encoded electrical waveforms shown in Fig. 19 (b), which served as the electrical input of the convolutional layer. The electronic waveform generated from the extracted feature maps is shown in Fig. 20, that served as the electrical input of the fully connected layer. Figure 21 shows the full experimental results of the convolutional layer of the CNN, including output recorded waveforms of convolved results and corresponding recovered/sampled and theoretically calculated feature maps. Figure 22 shows the experimental results of the ten output neurons in the fully connected layer. The left graphs show the output waveforms, while the right graphs show the corresponding sampled (red dot) and theoretically calculated (grey dot) intensities of the neurons. Figure 23 shows the resulting confusion matrix (theoretical and experimental) for the experiments with 50 images.

Figures [START_REF] Appeltant | Information processing using a single dynamical node as complex system[END_REF][START_REF] Chang | Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification[END_REF][START_REF] Vandoorne | Experimental demonstration of reservoir computing on a silicon photonics chip[END_REF][START_REF] Brunner | Parallel photonic information processing at gigabyte per second data rates using transient states[END_REF][START_REF] Tait | Demonstration of WDM weighted addition for principal component analysis[END_REF][START_REF] Pasquazi | Micro-combs: a novel generation of optical sources[END_REF][START_REF] Moss | New CMOS-compatible platforms based on silicon nitride and Hydex for nonlinear optics[END_REF][START_REF] Kippenberg | Dissipative Kerr solitons in optical microresonators[END_REF][START_REF] Savchenkov | Tunable optical frequency comb with a crystalline whispering gallery mode resonator[END_REF][START_REF] Spencer | An optical-frequency synthesizer using integrated photonics[END_REF][START_REF] Marin-Palomo | Microresonator-based solitons for massively parallel coherent optical communications[END_REF][START_REF] Corcoran | Ultra-dense optical data transmission over standard fiber with a single chip source[END_REF] show the results for the full experimental testing of the CNN with 500 testing images as well as the resulting confusion matrix (Figure 36). Figure 24 shows the experimentally recognized 500 handwritten digit images and corresponding generated waveform that served as the input of the convolutional layer. Figure 25 illustrates the experimental input waveform into the fully connected layer for the recognition of 500 handwritten digit images. Figure 26 shows the experimentally generated waveforms of the convolutional layer for the recognition of 500 handwritten digits. Figure 27 shows the experimentally recovered feature maps of the 500 handwritten digit images for batch 1-5, and Figure 28 batch 6-10, and Figures 29,[START_REF] Moss | New CMOS-compatible platforms based on silicon nitride and Hydex for nonlinear optics[END_REF] show the corresponding simulated feature maps of the 500 handwritten digit images (Figure 29 shows batch 1-5 while Figure 30 shows batch 6-10).

Figures [START_REF] Kippenberg | Dissipative Kerr solitons in optical microresonators[END_REF][START_REF] Savchenkov | Tunable optical frequency comb with a crystalline whispering gallery mode resonator[END_REF][START_REF] Spencer | An optical-frequency synthesizer using integrated photonics[END_REF][START_REF] Marin-Palomo | Microresonator-based solitons for massively parallel coherent optical communications[END_REF] show the experimental results of the ten output neurons in the fully connected layer. The upper graphs show the output waveforms, while the lower graphs show the corresponding sampled (red dots) and theoretically calculated (grey dots) intensities of the neurons. Figures 31,[START_REF] Savchenkov | Tunable optical frequency comb with a crystalline whispering gallery mode resonator[END_REF] show the output of neurons 1-5 and 6-10 for the first batch, respectively while Figures 33,[START_REF] Marin-Palomo | Microresonator-based solitons for massively parallel coherent optical communications[END_REF] show the corresponding results (neurons 1-5 for Fig. 33 and 6-10 for Fig. 34) for the second batch. Figure 35 shows the experimental (right) and calculated (left) results of the ten output neurons in the fully connected layer. Figure 36 shows the experimentally generated predictions of the 500 testing images contained in the confusion matrix. The upper figure shows the neuron outputs for 50 images amongst the 500 testing images. The lower figure shows the confusion matrix.

We first experimentally tested 50 images of the handwritten digit MNIST dataset, followed by more extensive testing on 500 images. For the first set of experiments based on testing 50 images, each with a resolution of 8-bits, each image being 30 × 30 pixels in size of the handwritten digit dataset with the deep optical CNN. The confusion matrix for 50 images (Figure 23) shows an accuracy of 88% for the generated predictions, in contrast to 90% for the numerical results calculated on an electrical digital computer.

The convolutional component of the CNN experiment had a computing speed given by 2×75×11.9 =1.785 TOPS, or 14.3 Terabits/s. To process matrix images using 5×5 kernels, the convolutional layer yielded an overhead due to matrix flattening 5, resulting in an image computing speed of 1.785/5= 357 Giga OPS/s. The fully connected layer had a computing speed of 119.8 Giga-OPS. Each image had a waveform duration of 30×30×84ps =75.6ns, and hence the convolutional layer processed images at a speed of 1/75.6ns = 13.2 million handwritten digital images a second.

The second set of experiments using the deep optical CNN tested 500 images, again, each having a resolution of 8-bits, with a pixel matrix size of 30 × 30 pixels (900 pixels) for each of the handwritten digit dataset. The computing speed of the system was identical to that of the first set of experiments testing 50 images. Figure 35 shows the resulting confusion matrix that indicates that our system has a prediction accuracy for 500 images that is essentially the same (but with one digit higher resolution in the accuracy) as for 50 images -89.6% for theory versus 87.6% for experiment. The fact that the CNN achieved close to the theoretical accuracy indicates that the impact of effects that could limit the network performance and reduce the effective number of bits (ENOBssee below), such as electrical and optical noise or optical distortion (due to high order dispersion), is small. The fact that these numbers agree with the first set of experiments means that 50 images is more than enough to adequately evaluate the system.

We note that full 10 digit (0 -9) handwritten digit recognition, although commonly performed by electronic digital networks, is very rare for optical ONNs. It requires a very large number of parallel physical paths for fully connected networks. For example, a hidden layer having 10 neurons would need 9000 physical paths, which represents a significant challenge for state-of-the-art nanofabrication technologies. Our CNN is the first integrable and reconfigurable ONN that able to not only perform complex and high-level operations, including the full 10 handwritten digit recognition, but also at Tera-OP per second (TOP) processing speeds. The CNN convolutional layer used 5 sample points at 59.421642 Giga Samples/s to generate each single symbol of the input waveform, which matched the progressive delay time (84 ps) generated by the 13km long dispersive fibre. The generated electronic waveforms for 500 testing images [START_REF] Xu | 11 TOPs photonic convolutional accelerator for optical neural networks[END_REF] formed the electrical signal input of both the convolutional and fully connected layers.

Table 1

Performance comparison of state-of-the-art optical neuromorphic hardware CW §: Indicating the approach used continuous-wave sources as the input data signal, high-speed updating of the input data is not demonstrated to achieve a high computing speed.

Performance comparison

We list the performance of recent optical neuromorphic processors in Table 1. This section is not intended to be complete but just highlights some key work that addresses the most crucial technical issues for optical processors. The dimension of the input data directly governs the processing task complexity. In real world applications, the dimension of the input data is typically quite large, with a human face image requiring > 60,000 pixels, for example. Therefore, to ultimately make optical computing hardware practical, the input data dimension needs to be at > 20,000 at a minimum.

Here, we process of images with 250,000 pixels, which is 224 times higher than previously reported.

The most important metric for computing hardware is arguably the computing speed and this is the main advantage of optical systems. Although to date there has not been a consensus on a definition of optical hardware computing speed, the key parameter is the number of data sets that can be processed within a given time frame -how many images per second can be processed. Therefore, although some systems [START_REF] Shen | Deep learning with coherent nanophotonic circuits[END_REF][START_REF] Lin | All-optical machine learning using diffractive deep neural networks[END_REF][START_REF] Feldmann | All-optical spiking neurosynaptic networks with self-learning capabilities[END_REF], have a low latency because of the short optical paths, the computing speed is nonetheless very low because of the lack of high-speed data interfaces -in other words, the input and output nodes are not updated at a high rate. Although some methods [START_REF] Larger | High-speed photonic reservoir computing using a time-delay-based architecture: Million words per second classification[END_REF][START_REF] Tait | Demonstration of WDM weighted addition for principal component analysis[END_REF] have high-speed data interfaces, the parallelism of their computing is not high and so the speed is very similar to the input data rate. In our work, [START_REF] Xu | 11 TOPs photonic convolutional accelerator for optical neural networks[END_REF] we achieve a record speed of 11.321 Tera-OPS, more than 500 x higher than any systems that had been reported when our results were published, by using high-speed data interfaces (62.9 Giga Baud) combined with time-wavelength interleaving,.

Finally, the reconfigurability and scalability of our method greatly enhances the versatility of the optical computing hardware. Approaches that are unable to reconfigure the synapses dynamically [START_REF] Lin | All-optical machine learning using diffractive deep neural networks[END_REF] (termed "Level 1" in the table) are hardly even trainable. While Level 2 systems [START_REF] Larger | High-speed photonic reservoir computing using a time-delay-based architecture: Million words per second classification[END_REF][START_REF] Feldmann | All-optical spiking neurosynaptic networks with self-learning capabilities[END_REF][START_REF] Tait | Demonstration of WDM weighted addition for principal component analysis[END_REF] support online training, they are only able to perform one specific task since the network structure is fixed and inflexible after the system is fabricated. Level 3 systems [START_REF] Tait | Demonstration of WDM weighted addition for principal component analysis[END_REF] can process different tasks although the function of each layer is fixed. This significantly limits the hardware in terms of being able to implement more complex tasks than matrix multiplication. Our work is the first that operates at Level 4 with full dynamic reconfigurability in all aspects of the system performance. With our approach the synaptic weights can be reconfigured simply by software reprogramming of a WaveShaper. Moreover, the number of synapses for each neuron can also be redesigned by re-assigning the wavelength channels using the de-multiplexer. The number of layers can be redesigned by simply varying the number of stacked devices. Finally, the computing function can be changed from convolution to matrix multiplication simply by changing the method of sampling. Finally, the physical level of integration of the system determines the computing density (processing capability per unit area). For systems that are not that amenable to integration [START_REF] Shen | Deep learning with coherent nanophotonic circuits[END_REF][START_REF] Lin | All-optical machine learning using diffractive deep neural networks[END_REF][START_REF] Tait | Demonstration of WDM weighted addition for principal component analysis[END_REF], the ultimate achievable computing density is very low. While other approaches have achieved a limited level of integration of the weights and summing circuits [START_REF] Shen | Deep learning with coherent nanophotonic circuits[END_REF][START_REF] Feldmann | All-optical spiking neurosynaptic networks with self-learning capabilities[END_REF] -probably the most challenging issueadvanced integrated light sourceshave yet to be achieved. The performance of the light source directly impacts the capacity of the overall system for both input data scale [START_REF] Shen | Deep learning with coherent nanophotonic circuits[END_REF] and number of synaptic connections per neuron [START_REF] Feldmann | All-optical spiking neurosynaptic networks with self-learning capabilities[END_REF]. The square millimeter sized microcomb that we use produces a great many precisely spaced wavelengths. This significantly enhances the overall parallelism and thus the computing density. Our approach is a significant step towards the full integration of optical computing hardware.

SCALING THE NETWORK

The architecture that we present here based on optical microcombs is highly amenable to scaling in network speed, size and performance to be able to process arbitrarily large input data size. The size of the input data is only limited in practical terms by the electrical digital-to-analog converter memory. Hence, it is possible in principle to process images with 4K-resolution (4096×2160). Furthermore, by integrating 100 photonic convolution accelerators layers together, which is still far fewer than the typical numbers of processors in electronic chips, such as the 65536 processors integrated in the Google TPU [START_REF] Graves | Hybrid computing using a neural network with dynamic external memory[END_REF] chip, the optical CNN would then be able to solve much more challenging image recognition functions, and with a vector computing speed of 100 × 11.3=1.130 Peta-OPS. Moreover, the network structure presented here is able to be trained online, since the optical Waveshaper used to weight the synapses can be reconfigured dynamically at speeds as fast as 500 milliseconds, or even potentially much faster with recently demonstrated optical integrated spectral shapers [START_REF] Metcalf | Integrated line-by-line optical pulse shaper for high-fidelity and rapidly reconfigurable RF-filtering[END_REF].

Even though our system had a reasonably large optical latency of over 100 nanosecond, mainly a result of the fibre spool, the latency does not affect the operational speed, which is a separate performance metric. Further, the latency of the system can be essentially eliminated to less than 200 picoseconds by using integrated highly dispersive elements such as customized chirped Bragg gratings or photonic crystals [START_REF] Sahin | Large, scalable dispersion engineering using cladding-modulated Bragg gratings on a silicon chip[END_REF] or etalon based tunable dispersion compensators [START_REF] Moss | Tunable dispersion and dispersion slope compensators for 10Gb/s using all-pass multicavity etalons[END_REF][START_REF] Lunardi | Tunable dispersion compensators based on multi-cavity all-pass etalons for 40Gb/s systems[END_REF].

Finally, existing nanofabrication technology can yield significantly higher levels of integration. The micro-comb source itself is already integrated and based on a CMOS compatible platform that is highly compatible with large-scale integration. Other components including the modulator, dispersive media, de-multiplexer optical spectral shaper, and photodetector have all been reported in integrated devices [START_REF] Metcalf | Integrated line-by-line optical pulse shaper for high-fidelity and rapidly reconfigurable RF-filtering[END_REF][START_REF] Sahin | Large, scalable dispersion engineering using cladding-modulated Bragg gratings on a silicon chip[END_REF][START_REF] Wang | Integrated lithium niobate electro-optic modulators operating at CMOS-compatible voltages[END_REF].

While optical neural networks are not yet at the level of performance as state-of-the-art electronic chips (>200 TOPs/s, scales with bit depth [START_REF] Xu | Photonic perceptron based on a Kerr microcomb for scalable high speed optical neural networks[END_REF][START_REF] Xu | 11 TOPs photonic convolutional accelerator for optical neural networks[END_REF][START_REF] Feldmann | Parallel convolutional processing using an integrated photonic tensor core[END_REF][START_REF] Marin-Palomo | Microresonator-based solitons for massively parallel coherent optical communications[END_REF]), our approach achieves operation speeds in the TeraOPs/s regime for the first time for optical networks. Further, there is enormous potential for scaling our systems through enhancing the spatial and wavelength dimensions and additional schemes such as using polarization. Both the convolutional accelerator and the CNN can be scaled in speed and processing power to enhance the parallelism using readily available off-the-shelf components. In the first instance, expanding the systems beyond the telecommunications C-band (1530-1570nm) to include the L-band (1570-1620nm) would yield a bandwidth of 90nm or 225 wavelengths (or channels) at a 50GHz spacing (0.4nm), versus the 90 wavelengths over 36nm in the C-band used here. These are both mainstream telecommunications bands for which there exists a tremendous amount of commercially available components and systems, including L-band EDFAs, Waveshapers, and many other components. Further, in the mainstream telecommunications bands (C+L) polarization sensitive components and devices are also available, meaning that taking advantage of polarization would yield an additional factor of 2x. Finally, spatial-division multiplexing, readily achievable using wavelength separation with either the Waveshaper or even just simple passive devices such as comb interleavers and passive filters, can offer, almost unlimited scalability, subject only to power/noise and scaling issues (cost, footprint, energy etc). Multiplying the system by a factor of at least 10, by using 10 parallel spatial paths, in principle is straightforward with existing components.

For the convolutional vector accelerator, operating with 3×3 kernels, and making use of polarization, the computational speed would be 2 × 2 × 9 × 62.9 = 2.26 TeraOPs/s per kernel. Making use of the C+L bands would produce 225 wavelengths at a 50GHz spacing, which would in turn allow 25 kernels, resulting in a processing speed of 25 × 2.26 = 56.6 TeraOPs/s. Using 10 spatial dimensions (through the Waveshaper) would enhance this to 0.57 PetaOPs/s.

The scale of the fully connected layer also has the potential to be significantly and readily increased with existing offthe-shelf technology. Since the number of neurons relies on spatial-division parallelism, or multiplexing, this number is, in principle, unlimited -subject only to tradeoffs in the OSNR (optical signal to noise ratio). By increasing the number of spatial paths (each with individual spectral shaping via more powerful WaveShapers and separate photo-detection), the number of neurons can be increased arbitrarily with existing instrumentation (subject to the SNR as mentioned). The number of synapses can also be significantly boosted though both wavelength and spatial division multiplexing. Making use of the full C+L band, supporting over 225 50GHz-spaced or 450 25GHz-spaced wavelength channels, and by exploiting dual polarization modes, the wavelength-division parallelism, and hence number of synapses per neuron could reach 225×2=450 (or even 900 at a 25GHz spacing, with tradeoffs in modulation rate). Further, even introducing a minimal number of additional spatial paths for each neuron (3 spatial paths, for example), the total number of synapses for 10 neurons @ 50GHz can reach 225 × 2 × 3 × 10 = 13,500 synapses in total.

Beyond this, a wider spectral region can readily be employed, although beyond the C+L bands, each has some challenges associated with it. Using the S+C+L telecommunications bands (1460-1620nm) would yield over 20THz in bandwidth.

The telecommunications S-band (1460-1530nm), although less widely used than the mainstream C and L bands, is still practical with wideband optical devices available including semiconductor and Raman amplifiers. This would yield a total wavelength range of 160nm, equating to 400 channels at 50GHz spacing. Figure 37 shows a fully connected layer using the full C+L+S bands along with polarization, 3 spatial dimensions and 10 neurons, yielding 405 (wavelengths) × 2 (polarizations) × 3 (spatial paths) × 10 neurons = 24,300 synapses. Figure 38 shows the vector convolutional accelerator, using the C+L+S bands (with 405 wavelengths) as well as 10 parallel spatial paths, and exploiting polarization. This would yield a speed of 62.9 Giga-Baud × 405 × 2 × 2 × 10 = 1.019 PetaOPs/s (or "POPs/s"). In this case the wavelengths would be distributed over 45 kernels each at 3×3 in size (so that 405=45×9).

Finally, in the long term, the full telecommunications bands including the O-band (1260 nm to 1360 nm) and even the Eband (water absorption band: 1360nm-1460nm) could be exploited, resulting in a total optical bandwidth of 1260nm-1620nm = 360nm, or 900 channels (Table 2). Using the same arguments as above, extending the network to 50 neurons (which is feasible since only spatial multiplexing is used for this) the CNN could be expanded to yield 900 (wavelengths) × 2 (polarizations) × 3 (spatial paths) × 50 neurons = 270,000 synapses.

Note that in terms of optical bandwidth, micro-combs themselves are not a limiting factorthey have demonstrated full octave-spanning spectraand morefrom a single device, including from the near and mid-infrared [START_REF] Della Torre | Mid-infrared supercontinuum generation in a low-loss germanium-on-silicon waveguide[END_REF][START_REF] Sinobad | Mid-infrared supercontinuum generation in silicon-germanium all-normal dispersion waveguides[END_REF][START_REF] Sinobad | High coherence at f and 2f of a mid-infrared supercontinuum in a silicon germanium waveguide[END_REF] In terms of the microcomb structure, the tradeoffs between comb FSR spacing and baud-rate are subject to the total available optical bandwidth, and are very similar to tradeoffs for ultrahigh bandwidth optical data communications. The computing speed of the accelerator is fundamentally determined by the available optical bandwidth. Within a certain optical band, the number of comb lines is inversely proportional to the FSR (i.e., the modulation rate). As long as the modulation rate matches the Nyquist bandwidth (half of the comb spacing), the network can be flexibly tailored to specific applications without sacrificing speed. In the case of the 49GHz microcombs studied here, as long as the optical band is sufficiently used, (i.e., the comb covers the full band and the modulation bandwidth (~24.5 GHz) matches with the FSR (~49GHz), the Nyquist bandwidth is ~ 24.5 GHz)), the computing speed does not vary dramatically with the number of comb lines or the FSR. So far, integrated microcombs feature FSRs ranging from 20 GHz to 1 THz, offering many options to choose from in terms of Baud-rate versus number of wavelengths. Having said this, we note that even for optical communications, this issue (the optimum channel spacing vs baud rate and modulation format) is still in fact an open question to a degree. Indeed, the exact optimum between the number of comb lines and the modulation rate is a function of the specific requirements for a given application. For applications that do not require a large number of kernel weights (wavelengths), a large FSR (modulation bandwidth) should be employed to make more extensive use of the optical band and achieve a high computing speed. While for those requiring a large number of kernel weights, a small FSR would be more favorable towards offering sufficient wavelengths.

Note that the preceding discussion does not address the issue of extending the CNN to much deeper levels. The electronic functions required for this have already been performed in this work, and include pooling, re-sampling, and retiming. Further, some if not all of these can be realized all-optically. The pooling function can be implemented via the convolution accelerator with an averaging kernel (with all kernel weights set to be equal), followed with down-sampling to reduce the data scale. The reduction in speed of the convolutional accelerator when used for matrix processing, brought about by the overhead associated with flattening the matrix into a vector, is outlined in detail in [START_REF] Xu | 11 TOPs photonic convolutional accelerator for optical neural networks[END_REF], along with an example of a system architecture designed to eliminate this overhead for the case of an accelerator operating with 3×3 kernels, and in the process generating a symmetric convolution. We note that this is almost never an issue, however, and that asymmetric convolutions are the norm.

The reduction in speed of the convolutional accelerator when used for matrix processing, brought about by the overhead associated with flattening the matrix into a vector, is outlined in detail in the supplementary pdf file. Figure 39 illustrates an example of a system architecture designed to eliminate this overhead for the case of an accelerator operating with 3×3 kernels, and in the process generating a symmetric convolution. We note that this is almost never an issue, however, and that asymmetric convolutions are the norm. The accompanying pdf file in the additional supplementary material graphically illustrates this.

Finally, we discuss the potential energy efficiency of the optical CNN. As the optical CNN in this work was demonstrated with discrete device-challenging to directly reflect the ultimate potentials of our approach, we estimate the energy efficiency of a fully integrated optical CNN (with the same network configurations as the demonstrated CNN), in line with the estimation protocols in [START_REF] Shen | Deep learning with coherent nanophotonic circuits[END_REF]. The power consumption of the comb source can reach as low as 98 mW [START_REF] Kippenberg | Dissipative Kerr solitons in optical microresonators[END_REF]. The power consumption of the optical layers (after been fully integrated) would be dominated by the optical power necessary to trigger an optical nonlinear function [START_REF] Shen | Deep learning with coherent nanophotonic circuits[END_REF]. Assuming the ultimate integrated version of our system employs the similar waveguide dimension and saturable absorber as in [START_REF] Shen | Deep learning with coherent nanophotonic circuits[END_REF], the total power consumption of the convolutional and fully connected layers can be estimated as 72+10=82 mW (1 mW for each nonlinear device). The overall computing speed of the CNN is given as the sum of the convolutional and fully connected layers' speeds, which is 357 +119.8=476.8 Giga OPs/s. As such, with an assumed power consumption of 200 mW for the electrical layer (could even be lower in practice), the energy per operation of the optical CNN can be roughly given as (98+82+200)mW / 476.8GOPs/s = 0.79 pJ/operation-comparable to state-of-art electronics (0.5pJ/operation [91]).

Theoretical evaluation of a scaled network

With the powerful feature extracting capability of the convolutional layer, a modest increase in the scale of the fully connected network would yield a significant increase in the recognition accuracy. Figure 40 shows the network structure of a CNN with two appropriately scaled fully connected layers. With a slightly larger scale for the input nodes ( 90) and an additional fully connected layer with 50 neurons, a prediction accuracy of 94.28% is achieved.

The scaled fully connected network can be achieved by the convolution accelerator through the methods outlined above. The hidden layer (i.e., the first fully-connected layer) would require 90 wavelengths in the C band and 50 spatially multiplexed neurons, while the output layer (i.e., the second fully-connected layer) would require 50 wavelengths in the C band and 10 spatially multiplexed neurons. In fact, this can be achieved in any manner of approaches that are compatible with the powerful convolutional layer in order to yield the best performance. The use of highly nonlinear materials may offer significant advantages for integrating novel functionality and higher levels of integration. [START_REF] Ghahramani | Full bandstructure calculation of 1st, 2nd and 3rd harmonic response functions of ZnSe, ZnTe, and CdTe[END_REF][START_REF] Wu | Graphene oxide: versatile films for flat optics to nonlinear photonic chips[END_REF][START_REF] Qu | Enhanced nonlinear four-wave mixing in silicon nitride waveguides integrated with 2D layered graphene oxide films[END_REF][START_REF] Jia | Large Third-Order Optical Kerr Nonlinearity in Nanometer-Thick PdSe2 2D Dichalcogenide Films: Implications for Nonlinear Photonic Devices[END_REF][START_REF] Zhang | Enhanced Kerr nonlinearity and nonlinear figure of merit in silicon nanowires integrated with 2D graphene oxide films[END_REF][START_REF] Zhang | Optimizing the Kerr nonlinear optical performance of silicon waveguides integrated with 2D graphene oxide films[END_REF][START_REF] Zhang | Design and optimization of four-wave mixing in microring resonators integrated with 2D graphene oxide films[END_REF] Table 2 Telecommunications wavelength bands. Main commercial bands (C, L) highlighted in yellow.

Band

CONCLUSION

We report an ultra-high speed universal optical convolutional accelerator that operates at speeds beyond 10 Tera-Operations per second (11 TOPS). It performs convolutions of ultra-large scale images at 250,000 pixels, with a digital resolution of 8-bits, and simultaneously for 10 kernels. This scale is more than adequate enough for facial image recognition tasks. We then employ the identical hardware to combine and form a sequential deep optical CNN with 10 output neurons. We successfully achieve the recognition of the full 10 handwritten digits, each image being 900 pixels in size, achieving an accuracy of 88% which is very close to the theoretically calculated accuracy of 90% for our system. Our approach is trainable and scalable to much more complex networks to perform much more demanding tasks for applications such as real-time video recognition for unmanned autonomous vehicles.

1

 1

Figure 1 |

 1 Figure 1 | Operation principle of the Tera-FLOPS photonic convolution accelerator. EOM: electro-optical Mach-Zehnder modulator. SMF: standard single mode fibre for telecommunications. PD: photodetector.

Figure 1 |

 1 Figure 1 | Principle of operation of the photonic convolution accelerator. EOM: electro-optical Mach-Zehnder modulator. SMF: standard single mode fibre for telecommunications. PD: photodetector.

Figure 2 |

 2 Figure 2 | Vector Convolutional Accelerator, for processing one dimensional data, consisting of the experimental setup (right panel), the optical and electronic control and signal flow (left panel). Pump: continueswave pump laser. EDFA: erbium doped fibre amplifier. MRR: micro-ring resonator. EOM: electro-optical Mach-Zehnder modulator. SMF: standard single mode fibre for telecommunications. PD: photodetector.

Figure 3 .

 3 Figure 3. Flattening method applied to the input data and kernel matrices. Also see the video presentation.

Figure 4 |

 4 Figure 4 | Image processing, including the experimental setup of the convolutional accelerator (right panel), the signal flow and the optical and electronic control (middle panel), and the processing flow of the raw input image (left panel). PC: polarization controller. MRR: micro-ring resonator. EOM: electro-optical Mach-Zehnder modulator. SMF: standard single mode fibre.

Figure 5 |

 5 Figure 5 | TOPS photonic convolution accelerator operating in two modes. Left: convolution operation mode with R = 4 and L = 13; Right: matrix operation mode with R = L = 4. Electro-optical modulator (EOM), optical buffer that has progressive wavelengthsensitive delay, and an optical-to-electrical conversion module (O/E).

Figure 6 |

 6 Figure 6 | a. Soliton crystal microcomb, generated by pumping an on-chip high-Q nonlinear micro-ring resonator with a CW laser. b. Image of the MRR and a SEM image of the MRR cross section. c. Measured dispersion of the MRR showing the mode crossing at ~1552 nm. d. Measured soliton crystal step of the intra-cavity power, and e. optical spectrum of the microcomb when sweeping the pump wavelength. f. Optical spectrum of the generated coherent microcomb at different pump detunings at fixed power.

Figure 7

 7 Figure 7 S26. (a) Measured comb power stability over 66 hours and extracted (b) standard deviation and (c) the relative standard deviation

Figure 8 |

 8 Figure 8 | Experimental results of the image processing. a. The kernel weights and the shaped microcomb's optical spectrum. b. The input electrical waveform of the image (grey lines are theory and blue experimental). c. The convolved results of the 4 th kernel that performs top Sobel image processing. d. The weight matrices of the kernels and corresponding recovered images.

Figure 9 .

 9 Figure 9. Experimental and simulated large scale facial image processing results performed by the convolutional accelerator with simultaneously processing ten convolutional kernels.

Figure 10 .

 10 Figure 10. Deep (multiple) level CNN structure. The feature maps are the convolutional matrix outputs while the fully connected layers embody the neural network component.

Figure 11 .

 11 Figure 11. The architecture of the optical CNN, including the convolutional, pooling, and fully connected layers.

Figure 12 .

 12 Figure 12. Experimental configuration of the optical CNN. Left side: input convolutional accelerator front end. Right side: fully connected layer, both of which comprise the deep learning optical CNN. The microcomb generates the wavelengths for both the convolution accelerator and fully connected layer. The electronic digital signal processing (DSP) module that performs the sampling and pooling functions is separate to this structure.

Figure 13 .

 13 Figure 13. Convolutional layer. Architecture and experimental results. Left panel: experimental setup. Right panel: experimental results of one convolutional kernel, showing shaped microcomb optical spectrum and corresponding kernel weights (blue lines are positive, red negative synaptic weights), the input electrical waveform for the digit "3" (middle: grey lines theory, yellow experiment), convolved results and feature maps.

Figure 14 .

 14 Figure 14. Fully connected layers. Architecture and experimental results. The left panel depicts the experimental setup, similar to the convolutional layer. The right panel shows top: the experimental results for one output neuron, including the shaped comb spectrum; middle: the pooled feature maps of the digit "3" and the corresponding input electrical waveform (the grey lines are theory and red experimental; bottom: output waveform of the neuron and sampled intensities.

Figure 15 .

 15 Figure 15. (a) Training curve and (b) the accuracy, of using the kth subset as the test set during the 10-fold cross validation for the full 10digit image recognition by the deep CNN.

Figure 16 .

 16 Figure 16. Results of full 10 digit (0-9) recognition by the deep level optical CNN. (a) The offline training curve and (b) generated predictions (confusion matrix) obtained by using an electronic computer to process the output of the CNN. The numbers represent the number of predicted labels versus the actual label (digit), hence the diagonal elements represent correct predictions.

Figure 17

 17 Figure 17 Convolutional layer shaped impulse response.

Figure 18 .

 18 Figure 18. The shaped impulse response of the ten neurons at the fully connected layer.

Figure 19 .

 19 Figure 19. (a) Experimentally recognized 50 handwritten digit images (b) generated waveform that served as the input of the convolutional layer.

Figure 20 .

 20 Figure 20. Generated (red) and simulated (grey) electrical waveform that served as the input for the 10 th neuron in the fully connected layer.

Figure 21 .

 21 Figure 21. Experimental results of convolutional layer, including output waveforms of convolved results and corresponding recovered and theoretically calculated feature maps.

Figure 22 .

 22 Figure 22. Experimental results of the ten output neurons in the fully connected layer. The left graphs show the output waveforms, while the right graphs show the corresponding sampled (red dot) and theoretically calculated (grey dot) intensities of the neurons.

Figure 23 .

 23 Figure 23. Experimental and theoretical results for image recognition. The upper figures show the sampled intensities of the ten output neurons of the fully connected layer, while the lower figures show the confusion matrices with the darker colours indicating a higher recognition score.

Figure 24 .

 24 Figure 24. The experimentally recognized 500 handwritten digit images and corresponding generated waveform that served as the input of the convolutional layer.

Figure 25 .

 25 Figure 25. The experimental input waveform into the fully connected layer for the recognition of 500 handwritten digit images.

Figure 26 .

 26 Figure 26. The experimentally generated waveforms of the convolutional layer for the recognition of 500 handwritten digit images.

Figure 27 .

 27 Figure 27. The experimentally recovered feature maps of the 500 handwritten digit images (batch 1-5).

Figure 28 .

 28 Figure 28. The experimentally recovered feature maps of the 500 handwritten digit images (batch 6-10).

Figure 29 .

 29 Figure 29. The simulated feature maps of the 500 handwritten digit images (batch 1-5).

Figure 30 .

 30 Figure 30. The simulated feature maps of the 500 handwritten digit images (batch 6-10).

Figure 31 .

 31 Figure 31. Experimental results of the ten output neurons in the fully connected layer. The upper graphs show the output waveforms, while the lower graphs show the corresponding sampled (red dot) and theoretically calculated (grey dot) intensities of the neurons. This figure shows the output of neuron 1-5 for the first batch.

Figure 32 .

 32 Figure 32. Experimental results of the ten output neurons in the fully connected layer. The upper graphs show the output waveforms, while the lower graphs show the corresponding sampled (red dot) and theoretically calculated (grey dot) intensities of the neurons. This figure shows the output of neuron 6-10 for the first batch.

Figure 33 .

 33 Figure 33. Experimental results of the ten output neurons in the fully connected layer. The upper graphs show the output waveforms, while the lower graphs show the corresponding sampled (red dot) and theoretically calculated (grey dot) intensities of the neurons. This figure shows the output of neuron 1-5 for the second batch.

Figure 34 .

 34 Figure 34. Experimental results of the ten output neurons in the fully connected layer. The upper graphs show the output waveforms, while the lower graphs show the corresponding sampled (red dot) and theoretically calculated (grey dot) intensities of the neurons. This figure shows the output of neuron 6-10 for the second batch.

Figure 35 .

 35 Figure 35. Experimental (right) and calculated (left) results of the ten output neurons in the fully connected layer.

Figure 36 .

 36 Figure 36. Experimental generated predictions of the 500 testing images. The upper figure shows the neuron outputs for 50 images among the 500 testing images. The lower figure shows the confusion matrix. See Fig 24 for the output for all 500 images.

Figure 37 .

 37 Figure 37. Designed expanded fully connected layer with 3 additional spatial dimensions and 10 neurons, making use of polarization multiplexing. Pump: continues-wave pump laser. EDFA: erbium doped fibre amplifier. MRR: micro-ring resonator. EOM: electro-optical Mach-Zehnder modulator. PBS: polarization beam splitter. PD: photodetector.

Figure 38 .

 38 Figure 38. Designed scaled convolutional accelerator over the C+L+S bands, with spatial and polarization multiplexing. The 405 available wavelengths (on a 50GHz grid) would be split into 45 kernels each 3×3 in size.

Figure 39

 39 Figure 39 Designed symmetric-stride convolutional accelerator. The number of introduced spatial paths is equal to the height of the convolutional kernel. Pump: continues-wave pump laser. EDFA: erbium doped fibre amplifier. MRR: micro-ring resonator. EOM: electro-optical Mach-Zehnder modulator. PD: photodetector.

Figure 40 .

 40 Figure 40. (a) CNN structure with 2 fully connected layers. (b) The offline training curve and (c) generated theoretical predictions (confusion matrix).

  down to the visible region. One of the more restrictive components is the optical amplifier. While both C and L band amplifiers are in widespread use in installed optical fibre networks, they do not operate in any other band. Raman amplifiers are extremely flexible and versatile in wavelength and so can potentially operate in any of the telecommunications bands. SOAs as well are quite versatile with devices available in the O and S bands. While the Waveshaper has commercially developed for the C and L bands, the fundamental technology behind it (liquid crystal on silicon -LCOS) is capable of supporting operation in any of the telecommunications bands. The same holds true for most of the other components such as modulators, detectors etc.while the commercially available components are generally designed for operation in the C and L bands, there is nothing fundamental in producing devices designed for the other bandsit is mostly a question of cost and scale.

Competing interests: The authors declare no competing interests.