Monsieur Damien Querlioz

Monsieur Giacomo Indiveri

Jerome Casas

Claudio Lazzari

Teresita Insausti

D R B Ly

Scalable Synaptic Eligibility Traces with Resistivity Drift of Phase

Keywords: Bayesian, machine learning, resistive memory, neural networks, electronic circuits, bioinspiration Bay ésien, apprentissage automatique, m émoire r ésistive, r éseaux de neurones, circuits électroniques, bio-inspiration

Artificial intelligence is a field that, historically, has benefited from the combination of ideas from across inter-disciplinary boundaries which have improved models of AI and the algorithms that operate on them. Biological nervous systems in particular have inspired various model topologies and algorithmic tricks that have led to leaps in performance. In contrast, as computing power and memory availability have increased relentlessly since the 1950's, models of artificial intelligence have largely failed to recognise the constraints imposed by, or incorporate the opportunities offered by, the underlying computing hardware. While this is not immediately apparent in the cloud computing setting; the mismatch between model, algorithm and hardware is the limiting factor that currently curtails the efficient application of locally-adaptive artificially intelligent systems at the edge. In this thesis, the interdisciplinary boundary between machine learning, emerging technologies and biological nervous systems will be explored with the objective of proposing a new, hardware-focussed, approach for the application of energy efficient and locally-adaptive edge neuromorphic computing systems. Resistive memories are a leading candidate as an enabling technology for AI to greatly reduce its energy requirements. This is largely owed to the efficient and parallelised implementation of the dot-product operation that pervades machine learning as well as its material-level compatibility with advanced CMOS processes. However, until now, the application of RRAM has been confined predominantly to implementations of gradient-based machine learning algorithms, namely backpropagation, to train RRAM-based multi-layer perceptron models. The fundamental properties of RRAM though, predominantly their conductance variability, are, on the contrary, not compatible with learning algorithms based on the descent of error gradients. This thesis recognises that, in contrast, the intrinsic properties of this technology can be harnessed through Bayesian approaches to machine learning where, like device conductance states, model parameters are described as random variables. RRAM-based implementations of Markov Chain Monte Carlo sampling algorithms are implemented and applied to the training of RRAM-based models. An RRAM-based computing hardware capable of supporting such models is also proposed. Inspired by the organisational principles of animal nervous systems, whereby memory and processing are distributed and arguably indistinguishable, this thesis proposes analogue circuit solutions for biological models of neurons and synapses and for a system-level architecture to interconnect such elements. Reflecting the role played by ion-channels embedded in biological neuronal membranes, these circuits co-localise memory and computation by incorporating resistive memory devices directly into the circuits themselves; determining model parameters and the interconnectivity between these elements locally. Relative to similar approaches, this obviates the need for volatile on-chip working memory and the use of analogue-to-digital conversion, both entailing significant energy demands. In recognition of the efficient solutions animals like insects have uncovered throughout the course of evolution, their nervous systems are used to guide the development of model architectures. Based on recent neurophysiological studies, models inspired by the cricket cercal system and the fruit fly motion detection system are proposed. To achieve an equivalent performance to the cercal system model, multi-layer perceptrons require between one and two orders of magnitude more memory elements; offering a means of scaling the proposed MCMC sampling algorithms to more complex tasks. It is also discussed how the 'small-worldness' of networks of neurons found in animal nervous systems can provide a solution to the spatial connectivity constraints inherent to the proposed RRAM-based computing fabric.

Book chapters

• Synaptic realizations based on memristive devices, Memristive Devices for Brain-Inspired Computing, Woodhead Publishing, 2020

Patents

• Odd-even memory array architecture for MCMC sampling 2021

• Temporal difference reinforcement learning algorithm using volatile conductance states 2020

• Phase change memory based spike-time-dependent plasticity algorithm, 2020

• Volatile memory based three-factor learning algorithm, 2020

Introduction

The human species, homo-sapiens, emerged an estimated 300,000 years ago. As modern humans we share, in large part, the same genes as our ancestors and our bodies and brains remain relatively unchanged. We have little common ground however in terms of how we live our lives and how we see the world. We can attribute this difference, despite our similarities, to our innate drive and ability to invent tools and ideas that have incrementally impacted how we are able to interact with the world over these millennia.

This encompasses pre-historic inventions such as stone daggers for hunting and techniques to start fires;

those from antiquity like religion, philosophy and bronze; all along the winding road of inventions through printing, economics and 5G telecommunications networks that have led us to this modern age. This thesis is concerned with another invention which, at the time of writing, is anticipated to give rise to a further incremental step on this path -artificially intelligent systems. Artificial intelligence (AI) is a scientific and engineering field concerned with the development of computer models that reproduce certain intelligent behaviours observed in animals such as reasoning, planning or learning. Since AI models and computers are fundamentally inter-twined, the term 'computer' is not only reserved for the computing machines with which we are today familiar, but importantly extends to any engineerable system that harnesses physical principles for computation. These computer models, other than developing theories of psychology and cognition, can also be applied in an engineering sense to solve practical problems. For example, the replacement or augmentation of human labour in monotonous or dangerous tasks, in complex tasks like medical diagnosis as well as applications which are otherwise unsuitable for humans such as the 'mining' of, or pattern extraction from, large and high-dimensional datasets.

The potential of the current generation of approaches in AI has been recognised both by governments and large technology corporations, which are jointly driving the rapid growth of its commercial application [1,[START_REF]Edge AI Chipsets: Technology Outlook and Use Cases[END_REF]. However, lurking in the shadow of numerous grand success stories, there are fundamental drawbacks and limitations which, to support the anticipated growth, must be confronted. Principally there are environmental issues, concerning the spiralling quantities of energy demanded by state of the art AI models [START_REF] Strubell | Energy and policy considerations for deep learning in nlp[END_REF][START_REF] Schwartz | Green ai[END_REF] for incremental performance gains [START_REF] Huang | Speed/accuracy trade-offs for modern convolutional object detectors[END_REF], the 'cloud-centric' application of AI instead of in distributed and resource constrained 'edge' environments [START_REF]Edge AI Chipsets: Technology Outlook and Use Cases[END_REF] and ethical issues regarding the lack of model interpretability [START_REF] Leilani H Gilpin | Explaining explanations: An overview of interpretability of machine learning[END_REF] and the absence of model uncertainty quantification in many cases [START_REF] Ghahramani | Probabilistic machine learning and artificial intelligence[END_REF][START_REF] Begoli | The need for uncertainty quantification in machine-assisted medical decision making[END_REF]. The progress in the field can be attributed predominantly to progress in underlying computing hardware, rather than a revolution from the algorithmic or modelling perspective, applied to 'connectionist', or 'neural network', AI models. This thesis explores how, in a similar spirit, an emerging technology called resistive memory can be married together with neural network models. It proposes a new, hardware-focused, approach that requires extremely low quantities of energy with respect to existing solutions. Additionally, in recognising the important role it has played in the development of the field via the tricks discovered by evolution, biological nervous systems will be used as a means of inspiration to guide the direction of the work.

Before introducing the content of this thesis in more detail, it will first be instructive to briefly recount the history of artificial intelligence, the computer hardware that has been used as a substrate in AI and elaborate upon some of the predominant problems faced by field which this work aims to address, in order to better understand its place and contribution.

A brief history of artificial intelligence

Although artificial intelligence would appear to be something new, being formally named in 1956 [START_REF] Mccarthy | A proposal for the dartmouth summer research project on artificial intelligence, august 31[END_REF], the seeds that would go on to become the roots of the field were gradually sewn in the centuries beforeduring an AI 'pre-history'.

Pre-history

The link between artificial intelligence and computer science is unshakeable -if intelligence were to be implemented artificially, on what other substrate besides a 'computer' could it be realised? The first system bearing the hallmarks of computer would appear in 1804 in France -the Jacquard loom [START_REF] Fernaeus | Revisiting the jacquard loom: threads of history and current patterns in hci[END_REF]. The loom was a fabric weaving machine, programmable via hole punched paper cards which determined routines to be executed with the apparatus (Fig. 1.1a). Some decades later, in 1832, Charles Babbage who was an admirer of the Jacquard loom, proposed a mechanical computer he called the Analytical engine (Fig. 1.1b), featuring an arithmetic logic unit, memory and a means of realising conditional flows and loops [START_REF] Michael | The mathematical work of Charles Babbage[END_REF] -although it would never be built. George Boole would go on to formalise the mathematics of the logical manipulation of symbols (which had been evolving from the time of the ancient Greek philosopher Aristotle) in 1854 [START_REF] Boole | An investigation of the laws of thought: on which are founded the mathematical theories of logic and probabilities[END_REF], before Alan Turing would take such concepts of logic and combine them with Babbage's ideas and propose the first digital computer system during the second world in 1936 [START_REF] Mathison | On computable numbers, with an application to the entscheidungsproblem[END_REF]. The ideas of Turing would then be taken forward by the likes of the Hungarian scientist John von Neumann [START_REF] Von | First draft of a report on the edvac[END_REF] who led the effort to manufacture the first binary stored-program Electronic Discrete Variable Automatic Computer (EDVAC) (Fig. 1.1c) architecture which remains widespread today. Largely it has been the manipulation of models, residing in the memory of the computer, by mathematical algorithms, which execute in the computers arithmetic centres, of these 'von Neumann' computers that has been the principal means of implementing and applying AI ever since. Parallel to the development of the ideas that lead to modern digital computers, several important mathematical theories would also be developed during these centuries that would go on to lay the foundations of AI. In 1805, one year after the invention of the Jacquard loom and also in France, the mathematician Adrien-Marie Legendre published a paper on the least-squares regression method [START_REF] Legendre | On the method of least squares. A Source Book in Mathemathics[END_REF] which remains, two centuries later, one of the central concepts of subsets of approaches to AI based on the minimisation of an error metric of a model given some data. Later, an iterative method for solving such problems would be proposed by Augustine-Louis Cauchy in 1847 called gradient-descent [START_REF] Lemar Échal | Cauchy and the gradient method[END_REF]. At the time of writing, gradient-descent is arguably the most important algorithm in modern day artificial intelligence and, in a broader sense, the field of mathematical optimisation.

Remarkably, some decades even before Adrien-Marie, an English reverend and mathematician named Thomas Bayes wrote down a novel interpretation on probability theory [START_REF] Bayes | an essay towards solving a problem in the doctrine of chances. by the late rev. mr. bayes, frs communicated by mr. price, in a letter to john canton, amfr s[END_REF]. What would come to be known as Bayes' rule described how, instead of calculating probabilities based on frequency of past events, a posterior probability could be updated through the product of a prior belief on this probability and the likelihood of a set of observations. Bayes actually did not publish this work during his lifetime, and his manuscripts were encountered by chance after his death in 1761 while his family were sorting through his possessions. Had they not been discovered, the French mathematician Laplace would also independently arrive at the same set of conclusions as Bayes half a century later in 1820 [START_REF] Simon | Th éorie analytique des probabilit és[END_REF]. These techniques set the foundations for a statistical approach to artificial intelligence called Bayesian inference whereby a posterior probability is iteratively updated as new evidence becomes available [START_REF] Bayes | an essay towards solving a problem in the doctrine of chances. by the late rev. mr. bayes, frs communicated by mr. price, in a letter to john canton, amfr s[END_REF]. A Russian mathematician Andrei Markov, also working in the field of probability theory one and a half centuries after Thomas Bayes, published several works describing what have become known as Markov chains [START_REF] Andreyevich | Extension of the limit theorems of probability theory to a sum of variables connected in a chain[END_REF].

Markov chains describe a random and 'memoryless' process whereby new states of an arbitrary system results from random permutations to its current state, offering a way to model systems with apparently stochastic characteristics. Markov chains and Bayes' theorem would eventually be married together in several works in the follow century, beginning with an article from Nicholas Metropolis and Stanislav Ulam on the Monte Carlo method [START_REF] Metropolis | The monte carlo method[END_REF][START_REF] Metropolis | Equation of state calculations by fast computing machines[END_REF], and later expanded upon by Wilfried Keith Hastings [START_REF] Keith | Monte carlo sampling methods using markov chains and their applications[END_REF], that set the groundwork for 'Markov chain Monte Carlo' (MCMC) sampling algorithms. Crucially, Markov chain Monte Carlo would allow for Bayesian inference to applied to problems where, in the majority of cases, conjugate likelihood and prior distribution pairs did exist and did not permit an analytical solution to the posterior [START_REF] Edward I George | Conjugate likelihood distributions[END_REF]. At the time of writing, MCMC is considered to be of the most important and influential computer algorithms ever proposed [START_REF] Barry | The best of the 20th century: Editors name top 10 algorithms[END_REF], with a marked impact upon countless scientific and commercial fieldsartificial intelligence among them.

In the early 20 th century, prompted by experimental results such as the measurement of action potentials propagating down the giant axon of the squid to trigger an escape response [START_REF] Zachary | The functioning of the giant nerve fibres of the squid[END_REF], a group of scientists who called themselves Cyberneticians, would begin to build 'bottom-up' models of animal nervous systems [START_REF] Wiener | Cybernetics or control and communication in the animal and the machine[END_REF]. An early contribution to the field was made by Americans Warren McCulloch and Walter Pitts in 1943 when they proposed a model of a neuron that was activated as a function of a weighted sum of its input [START_REF] Warren | A logical calculus of the ideas immanent in nervous activity[END_REF]. This would be followed with another model proposed by Alan Hodgkin and Andrew Huxley, based on their own experiments they conducted with the squid giant axon, that expanded on the ideas of the McCulloch-Pitts model by incorporating dynamical properties of neuronal membranes and the 'spiking' activation behaviour observed in the squid axon [START_REF] Hodgkin | A quantitative description of membrane current and its application to conduction and excitation in nerve[END_REF]. They would go onto receive the Nobel prize for their work in 1963. At the same time, a Canadian scientist named Donald Hebb proposed that neurons, described by such models, could organise themselves via correlative rules based on their co-activation as a means of adaptation and learning in the nervous system -an idea that would come to be called Hebbian learning [START_REF] Olding | The organization of behavior: A neuropsychological theory[END_REF]. Successive works on the visual systems of Drosophila [START_REF] Hassenstein | Systemtheoretische analyse der zeit-, reihenfolgenund vorzeichenauswertung bei der bewegungsperzeption des r üsselk äfers chlorophanus[END_REF] and cats [START_REF] David | Receptive fields, binocular interaction and functional architecture in the cat's visual cortex[END_REF], would go onto recognise that networks of these neuronal elements were were capable of computing certain functions defined by their topology -respectively motion and edge detection. An early Cybernetic system, brought to life by a hard-wired network of neuronal like transistors, was the 'beast' robot -assembled by scientists at John Hopkins University [START_REF] David | Autonomous systems[END_REF]. The beast, a big metal can on wheels (Fig. 1.1d), was equipped with ultrasonic sensors on it's exterior which were connected to a small network of analogue logic circuitsculminating in no more than some dozens of transistors. The outputs of these gates controlled the wheels of the robot and, it is said, it would wander aimlessly around the corridors of the University avoiding obstacles and turning corners. When the beast was low on battery it would find, and plug itself into, an electrical socket. Then, when fully charged and ready to wander anew, it would un-plug itself and once again be on its way. The behaviour of the system has since drawn comparisons with the feeding and survival behaviours of simple uni-cellular organisms such as Paramecium [START_REF] Naitoh | Ionic mechanisms controlling behavioral responses of paramecium to mechanical stimulation[END_REF] as well as multi-cellular organisms equipped with simple nervous systems such as the C. Elegans worm [START_REF] Li | A map of the interactome network of the metazoan c. elegans[END_REF].

Σ y X 0 X 1 X 2 w 0 w 1 w 2 (a) V ext V int C m R n (t,V) R l V n V l I p V m t (b)

1950's, 60's and 70's

The first general purpose model inspired by biological neural networks was the perceptron [START_REF] Minsky | Perceptrons: An introduction to computational geometry[END_REF], proposed in 1958 by Frank Rosenblatt. The perceptron is composed of layers (i.e. a multi-layer perceptron)

of McCulloch-Pitts neurons. Each layer is networked by a so-called fully-connected matrix of synaptic edges, weights or parameters. Perceptrons allowed neurons to be leveraged as single linear units, in that they define a linear hyper-plane through their input feature space, which, through their combination in a network permitted non-linear functions to be implemented. An intuitive example of the utility of such networks is the three neuron, six synapse network which implements the non-linear XOR logic function [START_REF] Minsky | Perceptrons: An introduction to computational geometry[END_REF] -something that cannot be solved with a single linear neuron alone. The parameters of early perceptrons were determined by optimisation techniques such as the Widrow-Hoff delta learning rule [START_REF] Widrow | Adaptive switching circuits[END_REF]. This was a first example of a supervised 'machine learning' algorithm whereby the model would learn, not in biologically-plausible fashion as Hebb had envisaged [START_REF] Olding | The organization of behavior: A neuropsychological theory[END_REF], but rather using iterative mathematical operations implemented in the arithmetic centres of von Neumann computers. In their rule, the difference between a 'teacher' signal and the actual output of the neuron units would be multiplied by the activation of the input neurons and a learning rate constant to provide the updates that were required to applied on the intermediate synaptic weight matrix. However, the delta rule could only be used to parameterise synapses in a two layer perceptron. A technique called automatic differentiation, proposed in 1970 by a Finnish master student called Seppo Linnainmaa [START_REF] Linnainmaa | Taylor expansion of the accumulated rounding error[END_REF], would facilitate, although not until one decade later, the 'training' of multi-layer perceptrons. In the meantime, another type machine learning approach, inspired by Darwinian theories of evolution, was developed and applied to the optimisation of models, such as the multi-layer perceptron, called genetic and evolutionary algorithms [START_REF] Henry | Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence[END_REF]. Such algorithms operate by mixing together properties of a previous 'generation' of candidate models, selecting the best according to some fitness metric and then 'breeding' together the parameters of the best, or the 'fittest', models to produce a new generation of models, and so on.

As an alternative to the 'black-box' optimisation of perceptrons, another prominent early direction of exploration was into models of symbolic reasoning. A first, extremely famous, example of such a model was the chess playing program developed by American electrical engineer Claude Shannon -the same Shannon who would later make even greater contributions to information theory -in the year 1950 [START_REF] Shannon | Xxii. programming a computer for playing chess[END_REF].

Shannon's program searched a 'tree' of possible moves from a current game state using a search algorithm called minimax. After searching the tree to a certain depth, or a number of moves in the future, the algorithm 'backs-up' possible eventual outcomes to the top of tree and permits a decision to be taken regarding the optimal move (Fig. 1.3b). Some years later Arthur Samuel would augment Shannon's approach with a technique called 'rote' learning (this time using the game of draughts) where the algorithm would back-up and assign to nodes a measure of value based on the time to reward [START_REF] Arthur | Some studies in machine learning using the game of checkers[END_REF]. This concept of assigning value based on a temporal difference would set a foundation for a great number of algorithms that would be proposed in later decades. Another tree-based model, the goal-tree, was developed by James Slagle in 1963 [START_REF] James R Slagle | A heuristic program that solves symbolic integration problems in freshman calculus[END_REF], which took an initially complex integration problem and proceeded to use a set of pre-defined heuristic transformations to expand, therein constructing a tree, and simplify the original problem into a final form, at the terminal tree leaf nodes, that could be easily solved with another set of simple mathematical rules. Goal-trees were later applied to a set of problems known as 'block worlds', whereby the computer model was tasked with arranging scattered blocks on a table into a specific, user defined, arrangement -a non-trivial task requiring the computer model to reason and plan. In this case, based on a limited set of pre-defined logical rules and block manipulation functions, the computer would proceed to heuristically build a goal-tree and define a sequence of block manipulations required to obtain the required arrangement [START_REF] Gerald | A computational model of skill acquisition[END_REF]. One feature of goal trees was that, after performing a set of actions, the computer model could be queried to explain it's reasoning by backing-up the tree it has just built -offering an interpretation of its actions. A related AI model, based on logical relations between 'symbols', were semantic networks, often called knowledge graphs, applied by Ross Quillan in 1966 [START_REF] Ross | Semantic memory[END_REF] as a knowledge inference engine. Semantic networks are composed of nodes representing entities, and edges between the nodes explaining their relation or interaction and set the basis for the 'knowledge graphs' that underpin search engines like Google. For example, a directed edge in a semantic network 'is a' could connect the node 'cat' to the node 'animal'. In 1970, Patrick Winston would propose 'arch', or 'one-shot', learning whereby, an algorithm constructs a semantic network by inter-connecting symbols through specific relations consistent with examples presented to the model -building a comprehensive description of an object based on very few observations, or very few shots. An arch, for example, can be characterised by a horizontal block that is related to two vertical blocks below in that they support it, and those two vertical blocks were related to each other in that they do not touch each other (Fig. 1.3c) [START_REF] Patrick H Winston | Learning structural descriptions from examples[END_REF]. Unlike the delta learning algorithm used to train perceptrons, which required many training examples to produce incremental changes to model parameters, arch learning is capable of constructing and augmenting a number of relations between nodes from single examples. Additionally, like goal-trees, the resulting semantic graph also benefits from a means of interpretability. The culmination of these, and many other related ideas from this epoch that have not been discussed (such as constraint propagation [START_REF] Guzm Án | Decomposition of a visual scene into three-dimensional bodies[END_REF][START_REF] Waltz | Understanding line drawings of scenes with shadows[END_REF]), would ultimately lead to what were marketed as expert systems [START_REF] Waterman | A guide to expert systems[END_REF] in the 1980's. These systems incorporated knowledge into a hand-crafted logical model using control flow statements like if, then and else. Despite their perceived lack of elegance from AI practitioners, expert systems would ultimately mark the first time that an AI system was massively commercialised and applied to real-world problems.

Many of these approaches to machine learning, such as the application of the Widrow-Hoff delta rule to the perceptron, can be labelled as 'supervised' algorithms. Modern day machine learning is often split into three such categories -supervised, unsupervised and reinforcement learning. Ideas pertaining to the latter two were also proposed during these decades. Unsupervised learning algorithms, those without a teaching signal or labelled data, can trace their roots back to a paper in 1967 by Thomas Cover proposing the nearest neighbour algorithm [START_REF] Cover | Nearest neighbor pattern classification[END_REF]. Unsupervised learning algorithms, instead of learning to produce a desired output, looked for underlying structure in a set of data and use mathematical constructs, like euclidean distance from a common centroid or a parametric probability distribution, to group together similar data-points. (Fig. 1.3d). A few years earlier, in 1964, a Scottish computer scientist called Donald Michie, described the first reinforcement learning approach with his 'tabular' approach to noughts and crosses (tic-tac-toe) [START_REF] Michie | Experiments on the mechanization of game-learning part i. characterization of the model and its parameters[END_REF]. Perhaps for want of a computer, Michie demonstrated his idea using a set of 304 drawers, each containing coloured beads, and required a human operator to correctly move the beads between the drawers based on a small set of rules. Each drawer corresponded to one of the 304 possible game states of noughts and crosses and the colour of beads in each drawer corresponded to the possible actions taken from that state. For each step of the game, a bead was randomly sampled from the drawer of the current state by the operator and a nought or cross was placed accordingly. If the actions that were taken during a round of the game resulted in a loss, Michie's model would be 'punished' whereby the all of the drawn beads that were played in that losing game were removed from their drawers.

In the event of winning a game, the model was 'rewarded' by placing three extra beads of the colours that were drawn in the respective drawers. This had the effect that, with an increasing number of iterations, the model would have a higher probability of choosing actions in game states it had already visited that would lead to an eventual victory. The perceptron model composed of an input layer of neurons, representing the input data points, and an output layer of neurons each of which corresponds to a prediction, normally a label, about the input. These outputs, y, are inferred using the dot product between the data X and the weight matrix W and then the weight vector can be updated, or trained, through the multiplication of the difference between a teacher signal t and the actual outputs y with the input data X and a learning rate α. (b) A search tree model whereby each green circle represents a node, a specific state in a game of chess for example, and each blue branch that extends downwards a possible decision taken from that node. Blue upward pointing arrows provide a notion of backing-up the search tree, such that information about future states can inform what decision is taken in the current state -drawn as the green node at the top of the tree. (c An example of arch learning, using the task of learning the properties of an arch -drawn as the blue rectangular blocks. Arch learning allows the construction of a semantic network with relations (blue directed arrows) between nodes (green circles) that characterise the object to be recognised. (d) An example of a nearest neighbours unsupervised learning model whereby the distance between the centroid of a data cluster (green and blue crosses) and a new data point (white circle) can be used to determine to which cluster it belongs.

1980's and 90s

The 1980's began with a proposal for what would become a hugely important neural network architecture, named the 'neocognitron' by Japanese computer scientist Kunihiko Fukushima [START_REF] Fukushima | Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position[END_REF]. The modelling approach known at the time of writing as convolutional neural networks would descend from this idea.

In his paper, Fukushima draws heavily upon research into the architectural mechanisms uncovered in the visual cortex of the cat several decades earlier [START_REF] David | Receptive fields, binocular interaction and functional architecture in the cat's visual cortex[END_REF]. This 'bio-inspired' neural network architecture employed a combination of fixed and modifiable synapses which, contrary to later approaches with convolutional neural networks, would be trained using an unsupervised machine learning algorithm. Later in 1986, the automatic differentiation technique originally proposed in 1970 [START_REF] Linnainmaa | Taylor expansion of the accumulated rounding error[END_REF], was famously applied to train a multi-layer perceptron model -solving the credit assignment problem between network layers that had previously limited perceptron models [START_REF] David E Rumelhart | Learning representations by backpropagating errors[END_REF]. This technique came to be known as backpropagation. Three years later, the 'universal approximation theorem' was presented by George Cybenko stating that a finite hidden layer of fully-connected neurons of such a multi-layer perceptron model was capable of approximating any continuous function [START_REF] Cybenko | Approximation by superpositions of a sigmoidal function[END_REF]. The combination of backpropagation with universal approximators gave rise to tremendous expectation about the potential of such a modelling approach, although it would take further decades of innovation, by in large at the hardware level, before that potential would finally be realised. Backpropagation was extended to the setting of recurrent neural networks simultaneously by several scientists towards the end of the 1980s using a technique called backpropagation through time [START_REF] Paul | Backpropagation through time: what it does and how to do it[END_REF], whereby recurrent connections between neurons were defined, not with cyclic loops that were forbidden by backpropgation, but by 'unrolling' the network into several timesteps. In 1997 an important recurrent neural network architecture was proposed called the long short term memory network [START_REF] Hochreiter | Long short-term memory[END_REF]. Somewhat like the neocognitron, long short term memory networks, incorporated specific connectivity patterns which permitted, for example, architectural mechanisms enabling functionality such as 'forgetting' which proved useful in the tasks related to time-series data.

In the 1990's Christopher Watkins would take forward the ideas of Donald Michie, thankfully this time with a digital computer, and Arthur Samuel and propose the reinforcement learning approach known as Q-learning [START_REF] Christopher | Q-learning[END_REF]. Instead of a set of drawers containing beads, Q-learning uses a state-action 'Q-table '.

Each cell of a Q-table contains a Q-value corresponding to the expected, temporally discounted, future reward based on taking optimal actions from that state. The Q-learning algorithm explores these states through interaction with an environment and uses the Bellman equation from dynamic programming [START_REF] Bellman | Dynamic programming[END_REF] to populate, or learn, the state-value pairs in each cell. Around the same time, Richard Sutton would formalise the work of Arthur Samuel into what is now called temporal difference learning [START_REF] Richard S Sutton | Learning to predict by the methods of temporal differences[END_REF]. Temporal difference learning algorithms are characterised by a temporally decaying parameter called an eligibility trace that tracks the time since, for example, a particular state has been visited. This approach was applied in the 1994 by engineers working at IBM whereby a temporal-difference model learned, from self-play, to play the game of backgammon, eventually obtaining a 'master level' of performance [START_REF] Tesauro | Td-gammon, a self-teaching backgammon program, achieves master-level play[END_REF]. IBM would also produce DeepBlue two years later that, despite being unsuccessful the first time around, would go on to beat the reigning world chess champion in 1997. DeepBlue employed a search-tree algorithm, largely similar to the minimax algorithm used by Shannon four decades before, called alpha-beta [START_REF] Campbell | Deep blue[END_REF].

The success therein was not a substantial modelling innovation but rather the development of a massively parallelised computer architecture that allowed faster and deeper search to be performed. With DeepBlue, IBM would go on to set two trends in artificial intelligence in the subsequent decades -that of applying 'brute force' computing to AI and also the use of public competitions as a marketing strategy of large corporations.

Similarly to how the neocognitron took architectural inspiration from biology, an American scientist called John Hopfield would, in 1982, take inspiration from the dynamical properties of biological neural networks and propose a new idea for how memories could be stored and accessed in a recurrently connected population of neurons [START_REF] John | Neural networks and physical systems with emergent collective computational abilities[END_REF]. The neurons of early 'Hopfield networks' were McCulloch-Pitts neurons which output either +1, if it's weighted input exceeded a threshold, or -1 otherwise. Under the condition of symmetrical connectivity, it was proven that, upon presentation of a static input pattern, the state of the network would tend towards a stable attractor of +1 and -1 activations -serving as an activation 'barcode' that represented a memory. New attractors could be formed in Hopfield networks through the Hebbian modification of synaptic parameters based on the co-activation of the pre-and post-synaptic neurons pairs. Two years later Hopfield networks would be implemented using neurons implementing a sigmoid activation function [START_REF] John | Neurons with graded response have collective computational properties like those of two-state neurons[END_REF] and later with dynamical, Hodgkin-Huxley, neuron models [START_REF] Maass | Networks of spiking neurons can emulate arbitrary hopfield nets in temporal coding[END_REF] culminating in a research field dedicated to investigating 'attractor networks'. In the 1990's American physicist Carver

Mead would draw analogues between the modulation of conductivity in neuronal ion channels in biological neuronal membranes and transistors [START_REF] Mead | Neuromorphic electronic systems[END_REF] giving rise to a new field called 'neuromorphic' computing. Neuromorphic computing would concern itself with the bottom-up emulation of biological neural networks and biological Hebbian learning mechanisms, attractor networks for example, using sub-threshold analogue transistor circuit models of neurons and synapses.

These two decades also saw the proposal and demonstration of many other important supervised and unsupervised learning models and algorithms such as the self-organising map proposed in 1990

by Finnish scientist Teuvo Kohonen [START_REF] Kohonen | The self-organizing map[END_REF]. Additionally, in 1995, Vladimir Vapnik published a paper that combined a linear classification approach he had been developing since the 1960's called support vector machines [START_REF] Vapnik | Generalized portrait method for pattern recognition[END_REF], with a set of mathematical tricks called kernel methods that were also originally proposed in the 1960's [START_REF] Mark A Aizerman | Theoretical foundations of the potential function method in pattern recognition learning[END_REF]. The incorporation of the kernel trick into the support vector machine permitted an implicit projection of data, which was perhaps not linearly separable in its original space, into new space where it becomes separable using linear support vector machine classifier [START_REF] Cortes | Support-vector networks[END_REF]. Another important idea, called boosting, was developed during the 1990's [START_REF] Freund | A desicion-theoretic generalization of on-line learning and an application to boosting[END_REF]. Boosting recognised the predictive power of a large ensemble of individually weak predictive models which, while alone were perhaps only slightly better than random guessing, together allowed particular subsets of models to specialise in the detection of specific features of a complex dataset. Boosting was combined with decision trees by Tim Kam Ho in 1995 to realise the important random decision forest machine learning algorithm [START_REF] Tin | Random decision forests[END_REF].

21 st Century

In the current epoch it is perhaps not yet clear which new ideas will stick or not. At the time of writing, it seems like a period with little in the way of fundamentally new or radical ideas in how to build models of artificial intelligence and instead one where existing ideas have been supercharged by ever increasing computational resources. The key piece of computing hardware would become the graphics processing units, the first of which was launched by NVIDIA, rather appropriately, at the turn of the century in 1999 [START_REF]NVIDIA Launches the World's First Graphics Processing Unit: GeForce 256[END_REF]. Graphics processing units would ultimately pave the way for 'deep' multi-layer perceptron models to be trained using the backpropagation algorithm to achieve state of the art performance across a wide range of application domains. The approach was re-branded as 'deep learning' in order to distance it from previous failures regarding the optimisation of multi-layer perceptron models using backpropagation [START_REF] Lecun | Deep learning[END_REF]. If artificial intelligence were a garden, deep learning is a massive Sycamore tree that is blocking out the light for a lot of the smaller, more fragile, plants. The first great success of deep learning was the application of a deep convolutional network model, named AlexNet, in the annual ImageNet image classification competition held in 2012 where it achieved a resounding victory [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. AlexNet, a somewhat monstrous evolution of the 1980's neocognitron counting 60, 000, 000 free synaptic parameters, proposed a regularisation technique called dropout that prevented the model, which with the large number of parameters had so many degrees of freedom, from over-fitting. Dropout, another concept loosely inspired from biology, simply requires that, upon successive forward propagations, synaptic parameter values be temporarily set to be zero with some probability such that they do not contribute to the models prediction.

The next year, a British artificial intelligence company called DeepMind would demonstrate a deep reinforcement learning approach, called Deep-Q learning, that learned to play Atari games from experience [START_REF] Volodymyr | Playing atari with deep reinforcement learning[END_REF] where a deep convolutional neural network model was employed as an approximation of the Q-table proposed in the 1990's [START_REF] Christopher | Q-learning[END_REF]. Again it was a bio-inspired idea, based on theories of the mammalian hippocampus, called experience replay was the would prove to be the trick that would enable the Deep-Q model to obtain such an 'expert-level' of performance. Echoing the marketing strategy of IBM's DeepBlue in the 1990's, DeepMind famously defeated the European champion of 'Go' with their Alpha-Go deep reinforcement learning model in 2016 [START_REF] Silver | Mastering the game of go with deep neural networks and tree search[END_REF]. Similarly to DeepBlue, Alpha-Go was based on a search-tree algorithm called Monte Carlo tree search [START_REF] Cameron B Browne | A survey of monte carlo tree search methods[END_REF] combined with of value and policy encoding deep learning models.

Deep learning models have also had a profound impact in natural language processing. Initially, this was thanks to deep recurrent neural network models, the long short term memory network for example.

However, in 2014 'attention' models were proposed which would come to dominate natural language processing. Attention provided an architectural mechanism allowing models to focus on particular parts of a data sequence -whether it being an a certain area of an image or particular word in a phrase -similar in concept to the mammalian fovea. Initially deep attention models were incorporated into an architecture composed of recurrent neural networks [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF]. In 2017 however, engineers working at Google proposed an attention model called Transformer. Transformer improved on the previous state of the art using an attention architecture that did not feature any recurrence [START_REF] Vaswani | Attention is all you need[END_REF] -leading many in the field to claim that recurrent neural networks were now redundant in such applications.

Another important deep learning architecture was proposed in 2014 called the generative adversarial neural network [START_REF] Goodfellow | Generative adversarial nets[END_REF]. These models are actually a composite of two competing deep neural networks. The first neural network, the generator, is fed with noise it then uses to produce a 'fake' output data point -a fake image for example. The second network, the discriminator, is tasked with distinguishing this fake from a real data point. The two models are optimised simultaneously using backpropagation, whereby the objective of the discriminator is to minimise the error in the detection of the real data points while the generator is trained to maximise the error of the discriminator. Interestingly, the generator uses a de-convolutional neural network architecture to generate the fake images [START_REF] Matthew D Zeiler | Deconvolutional networks[END_REF] -effectively a convolutional network flipped back-to-front.

Despite the truly seismic success of deep learning, serious concerns about model size, therein the number of synaptic parameters in the model, as well as the computational resources required to optimise all of those parameters during training throw its practicality into question [START_REF] Strubell | Energy and policy considerations for deep learning in nlp[END_REF][START_REF] Schwartz | Green ai[END_REF]. At the time of writing the largest models make use of a few hundred billion synaptic parameters [START_REF] Tom B Brown | Language models are few-shot learners[END_REF]. This is particularly relevant in the context of memory and energy constrained 'edge' computing applications and has led to research into deep learning architectures and algorithms that allow model size to be significantly reduced. One such model,

proposed in 2016, is SqueezeNet [START_REF] Forrest N Iandola | Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size[END_REF]. SqueezeNet offers a 50x reduction in model size with respect to the sixty million parameter AlexNet while being able to obtain an equivalent performance. The principal innovation of SqueezeNet was the proposal of the 'fire' module. Each fire module in the architecture is composed of a layer of point-wise convolutional filters followed by an expansions layer containing more point-wise filters but also larger convolutional kernels.

Another direction of research, resonating more with the ideas of John Hopfield and attractor networks, called reservoir computing emerged at the beginning of the 21 st century. Reservoir computing is based on the idea that static, randomly recurrently connected neurons could project an input time-series into a new, possibly higher dimensional, space where temporal information pertaining to different classes of data points could be integrated into linearly separable neuron activation patterns. The two main approaches in reservoir computing are echo state networks [START_REF] Jaeger | The "echo state" approach to analysing and training recurrent neural networks-with an erratum note[END_REF], which employ neurons with continuous activation functions, and liquid-state machines [START_REF] Maass | Real-time computing without stable states: A new framework for neural computation based on perturbations[END_REF], that use spiking neuron models. The two approaches can therefore be broadly dichotomised into favouring either the McCulloch-Pitts or Hodgkin-Huxley paradigms. Both approaches have been applied to problems in temporal pattern recognition, although their achievements have been largely overshadowed by the achievements of deep learning models applied to the same tasks.

Despite being unable to match the performance of deep learning models based on graphics processing units, 'non-von Neumann' reservoir computing models implemented on an emerging class of computer called a 'neuromorphic processor', offers the prospect of a more energy efficient solution. Rooted in the ideas outlined by Carver Mead in the 1980's [START_REF] Mead | Neuromorphic electronic systems[END_REF], the first system bearing the hallmarks of a neuromorphic processor was demonstrated in 2006 and consisted of an array of programmable analogue silicon spiking neuron circuits which could be inter-connected through further circuits models of the synaptic connections between neurons [START_REF] Farquhar | A field programmable neural array[END_REF]. A more developed system, supported by an additional digital 'event routing' protocol was proposed in 2014 [START_REF] Varkey Benjamin | Neurogrid: A mixed-analog-digital multichip system for large-scale neural simulations[END_REF] and would be followed by a similar chip in 2015 which incorporated additional circuit implementations of bio-inspired Hebbian learning algorithms [START_REF] Qiao | A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128k synapses[END_REF]. Fully digital neuromorphic processors were then released released by IBM in 2015 [START_REF] Filipp Akopyan | Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip[END_REF] and then Intel in 2018 [START_REF] Davies | Loihi: A neuromorphic manycore processor with on-chip learning[END_REF] confirming the interest of large corporations in the development of neuromorphic, event-based, hardware. Such systems are referred to as non-von Neumann since they offer a means of distributed memory and parallelised computing, similar to the computation of the animal nervous systems on which the chips are based, in contrast to the centralised and sequential nature of the computing machines developed by the likes of Alan Turing and John von Neumann. However, it is important to note that both Turing and particularly von

Neumann [START_REF] Von | First draft of a report on the edvac[END_REF] also argued in favour of such brain-inspired computing solutions in the 1940's and 50's.

Hardware in artificial intelligence

Artificially intelligent computer models are intrinsically tied to computing hardware. It is no coincidence that the fathers of computer science, like Alan Turing, Donald Michie and Claude Shannon, were also the pioneers that developed early concepts in AI that still persist to this day. At the time of writing, the predominant computing architecture is the von Neumann stored-program computer. Von Neumann machines are coarsely defined as having separate memory and processing centres. Information encoded as strings of binary bits, describing either variables or arithmetic and logic instructions, are stored as digital words in the memory centre. The processing unit fetches these variables and instructions from defined addresses in memory, performs an operation on the variables as defined by the corresponding instruction and then writes the result back to another pre-defined location in the memory. Relative to the modern computers available at the time of writing, early systems were almost unimaginably constrained in terms memory and speed. Around the time that Arthur Samuel wrote his draughts playing program [START_REF] Arthur | Some studies in machine learning using the game of checkers[END_REF] for example, the state of the art computer was the IBM 1401. It disposed a mere 4kb of 8-bit memory and was clocked at a frequency of 87kHz. This permitted the equivalent of approximately one thousand operations per second [START_REF] Jeffrey R Yost | The IBM century: Creating the IT revolution[END_REF]. You can imagine how aware of the underlying hardware the early AI practitioners had to be. As time progressed however, von Neumann computers would become faster and leverage more memory as well as benefit from architectural innovations such as parallelisation, cache memory [START_REF] James R Goodman | Using cache memory to reduce processor-memory traffic[END_REF] and multi-threading [START_REF] Dean M Tullsen | Simultaneous multithreading: Maximizing on-chip parallelism[END_REF], such that gradually less and less thought was given to the underlying computation. However, for memory intensive techniques in AI, such as deep learning, computing hardware has once again become a limiting factor. This is not due to memory capacity, which continues to increase year on year, nor is it due to a limit in clock speed. The problem arises from the time and energy incurred in the transportation information between memory and processing centres. In reference to this, the term 'von Neumann bottleneck' was coined in 1978 [START_REF] Backus | Can programming be liberated from the von neumann style? a functional style and its algebra of programs[END_REF]. This has driven the pursuit of alternative, non-von Neumann, hardware paradigms wherein the objective has been to reduce as much as possible the physical distance separating memory and processing -ideally to zero.

LISP machines

In the first decades of AI research and application, the go-to language for the implementation of tree, logic and knowledge based models was LISP [START_REF] Mccarthy | Recursive functions of symbolic expressions and their computation by machine, part i[END_REF]; deriving its name from LISt Processor in reference to the data structure and syntactic style central to by the language -the linked list. As a result of its popularity, a special purpose computer called a LISP machine was developed in the 1970's [START_REF] Richard D Greenblatt | A lisp machine[END_REF] whereby dedicated circuits allowed the acceleration of various LIPS specific operations and memory structures.

These machines offered a speed-up of several factors compared to equivalent programs implemented on general purpose machines. However, as the use of the LISP language, and the approach to AI which it facilitated, declined during the 1980's and 90's, many LISP machine manufacturers would ultimately go out of business and general purpose von Neumann computers would go on to serve as the predominant substrate for AI.

Field programmable gate arrays

The earliest example of a non-von Neumann computing hardware is the field programmable gate array (FPGA), proposed in 1985 by the American company Xilinx [START_REF] Alfke | It's an fpga![END_REF]. FPGAs are effectively a two-dimensional array of configurable logic blocks (Fig 1 .4a). Each block can be configured independently to implement basic logic operations, typically using a look-up table. The connections between neighbouring blocks can also be configured offering the possibility of realising large reconfigurable digital circuits. FPGAs distribute blocks of memory over the periodic 2D array which are responsible for storing the information which determines the logic and inter-connectivity for a subset of blocks. Naturally a body of work has focused on the implementation neural network architectures and deep learning algorithms, such as backpropagation [START_REF] David E Rumelhart | Learning representations by backpropagating errors[END_REF], on FPGAs [START_REF] Amos | FPGA implementations of neural networks[END_REF]. In particular the parallelisation of the dot(inner)-product operation, which underpins the forward and backward propagations required in deep learning, has been investigated. However, when benchmarked against application-specific integrated circuits (ASICs) implementing neural networks [START_REF] Cavigelli | Origami: A convolutional network accelerator[END_REF],

FPGAs are generally acknowledged to incur penalties an order of magnitude greater in silicon area, power consumption and latency [START_REF] Kuon | Measuring the gap between fpgas and asics[END_REF].

Graphics and tensor processing units

Graphics and, more recently proposed tensor, processing units (GPU/TPUs) represent an evolution of central processing units (CPUs) through the incorporation of a parallelised architecture and multi-threading that allows the acceleration of mathematical operations, namely operations on matrices (Fig. 1.4b). Originally proposed, as suggested by the name, to accelerate the processing of images and video by NVIDIA in 1999 [START_REF]NVIDIA Launches the World's First Graphics Processing Unit: GeForce 256[END_REF], the GPU has also been leveraged to accelerate AI models based on the manipulation of large matrices -principally deep learning [START_REF] Lecun | Deep learning[END_REF]. Like CPUs, current GPUs [START_REF] Lindholm | Nvidia tesla: A unified graphics and computing architecture[END_REF], are configured to execute stored programs through the use of an interface called Compute Unified Device Architecture (CUDA) which provides access to the arithmetic units of the GPU from a high-level programming language and, although memory and processing and more distributed with respect to a CPU, they are still considered as von Neumann computers. Recently Google released the TPU that, unlike the GPU which is intended to be applied to a broad class of applications, is targeted solely on accelerating inference and learning algorithms related to deep learning [START_REF] Norman P Jouppi | In-datacenter performance analysis of a tensor processing unit[END_REF]. While GPUs and TPUs achieve extremely high throughput and accelerate significantly deep learning models, they require a large amount of energy -a typical GPU at the time of writing will consume on the order of hundreds of Watts of power [START_REF] Huang | On the energy efficiency of graphics processing units for scientific computing[END_REF]. Their application is therefore largely limited to cloud computing and data centre settings. This however restricts the class of tasks to which such GPU-based AI models can address -in particular applications at the edge where energy and memory resources are constrained and perhaps intermittent.

Neuromorphic processors

The energy requirements of GPUs and TPUs, when applied to artificial intelligence, are often contrasted with the estimated energy requirements of animal nervous which are capable of performing the same task (while of course performing countless other parallel tasks) but requiring many orders of magnitude less energy. This has driven interest in the field of neuromorphic computing [START_REF] Mead | Neuromorphic electronic systems[END_REF], which aims to develop computing systems that imitate the bottom-up computational primitives of animal nervous systems in configurable, often self-configuring, silicon chips. It is believed that such a bottom-up approach, emulating the massive parallelism and distributed computation of the biological system, will lead to the realisation of a more energy efficient hardware that could support AI models which can be applied in the energy constrained setting of the edge. Numerous analogue circuits modelling the dynamical properties of biological neurons [START_REF] Mahowald | A silicon neuron[END_REF][START_REF] Indiveri | Neuromorphic silicon neuron circuits[END_REF] and synapses [START_REF] Bartolozzi | Synaptic dynamics in analog vlsi[END_REF] have been proposed, leading to the first re-configurable arrays of such models in 2006 [START_REF] Farquhar | A field programmable neural array[END_REF]. Later systems would be composed of multiple 'neuromorphic cores' [START_REF] Qiao | A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128k synapses[END_REF][START_REF] Varkey Benjamin | Neurogrid: A mixed-analog-digital multichip system for large-scale neural simulations[END_REF] that make use of content addressable memories [START_REF] Pagiamtzis | Content-addressable memory (cam) circuits and architectures: A tutorial and survey[END_REF] and a routing protocol called address event representation (AER) [START_REF] Zamarre Ño-Ramos | Multicasting mesh aer: A scalable assembly approach for reconfigurable neuromorphic structured aer systems. application to convnets[END_REF] to send information between neurons on the same core and between cores (Fig. 1.4c). AER assigns an address to each neuron and, since neuromorphic processors implement spiking neuron models like that proposed by Hodgkin and Huxley [START_REF] Hodgkin | A quantitative description of membrane current and its application to conduction and excitation in nerve[END_REF], transmit events from a source address neuron to, potentially, multiple target address neurons asynchronously.

Neuromorphic processors have also been realised using digital, instead of analogue, circuit models of neurons and synapses [START_REF] Filipp Akopyan | Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip[END_REF][START_REF] Davies | Loihi: A neuromorphic manycore processor with on-chip learning[END_REF]. The digital approach is appealing since, as technology nodes scale down, digital circuits become more energy efficient whereas analogue circuits face increasing design challenges and are not able to extract an equivalent benefit [START_REF] Annema | Analog circuit performance and process scaling[END_REF]. Furthermore, due to the increased mismatch between transistor dimensions and properties in more advanced nodes, large behavioural differences can exist between analogue circuit models that would ideally behave identically. To mitigate this, the dimensions of the transistors composing analogue circuit models are generally increased well above the minimum feature size of the node [START_REF] Qiao | Scaling mixed-signal neuromorphic processors to 28 nm fd-soi technologies[END_REF] -negating somewhat the benefit of using the more advanced technology nodes.

Other than providing a silicon substrate for defining dynamical neural network topologies, neuromorphic processors also support biologically-plausible learning and adaptation algorithms, such as Hebbian spike-timing-dependent-plasticity [START_REF] Markram | Spike-timing-dependent plasticity: a comprehensive overview[END_REF] for long-term memory formation as well as circuits implementing short-term plasticity [START_REF] Fioravante | Short-term forms of presynaptic plasticity[END_REF] and also spike frequency adaptation [START_REF] Benda | A universal model for spike-frequency adaptation[END_REF]. At the time of writing, the field of neuromorphic computing, in particular the development of neuromorphic processors, is still emerging and currently faces a broad spectrum of challenges ranging from conceptual ones, regarding how to transfer results from computational neuroscience into neuromorphic computing models, to technical ones such as addressing bandwidth limitations and reducing levels of static power consumption due the AER based communication protocol [START_REF] Moradi | The impact of on-chip communication on memory technologies for neuromorphic systems[END_REF]. The field would do well however, to appreciate the long timescales observed throughout the history of AI between the emergence of new ideas and their successful implementation. , i [START_REF]Edge AI Chipsets: Technology Outlook and Use Cases[END_REF], i [START_REF] Strubell | Energy and policy considerations for deep learning in nlp[END_REF]) that flow out of each row correspond to the vector i.

V[0] V[1] V[2] V[3] i[0] i[1] i[2] i[3] { i = V . G G (d)

Memory technologies

Computing hardware is dominated by memory. This is as true for the distinct memory centres of von Neumann chips as it is for the distributed memory blocks in FPGAs and neuromorphic processors. Models of artificial intelligence also hinge on memory which provides structure and brings them to life -whether it being the giga-bytes of memory required to define the synaptic weights of AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] or the transient memories that are accessed through attractors in Hopfield networks [START_REF] John | Neural networks and physical systems with emergent collective computational abilities[END_REF][START_REF] John | Neurons with graded response have collective computational properties like those of two-state neurons[END_REF]. Memory is a concept that unifies concepts of learning, behaviour and planning in psychology but at the same time is something entirely tangible from a technological perspective. The right kind of memory technology could be the spark that illuminates new opportunities and methods for constructing the next generation of non-von

Neumann machines that can support models of artificial intelligence more efficiently than those that exist today.

Like AI, memory technologies also have an interesting history, whereby a multitude of physical principles have been harnessed over time as they have become available or were sufficiently well understood. From the time of the Jacquard loom [START_REF] Fernaeus | Revisiting the jacquard loom: threads of history and current patterns in hci[END_REF], up until approximately the mid-twentieth century, the best way to store information was using a punch card -literally a thick sheet of paper with holes punched into it. The punch card was eventually replaced by binary patterns written onto, and read off of, ferromagnetic strips on the outer surface of a rotating drum in 1940's and 50's [113]. Early computers would go on to use magnetic tapes, a sensible descendent of the rotating drum, as well as delay-line memories in which signals, often a series of rectangular ultrasonic waveforms, would propagate indefinitely in a delay loop medium, liquid mercury for example, and be restored once per loop using a pulse shaping circuit [START_REF] Isaac L Auerbach | Mercury delay line memory using a pulse rate of several megacycles[END_REF]. A computer could then read out serial sections of this circling waveform and compute with it. In the 1960's, both static random access memory (SRAM) [START_REF] Arnold | Nondestructive memory array[END_REF] (Fig. 1.5a) and dynamic random access memory (DRAM) (Fig. 1.5b) were developed that were respectively based on a transistor circuit with restorative feedback and a transistor gated capacitor which was periodically charge-refreshed. These memory cells have since become favourable as they can be realised using the same technological processes that are also used to make arithmetic logic units and today remain the dominant solutions for on-chip working memory'. They are referred to as volatile memory technologies since the electric charge, stored by the respective cells that represents the bit they encode, leaks away in the absence of a power supply. The major downsides of such volatility are the static power consumption, required for information to be retained, and the fact that each time a system is power cycled information has to be reloaded into working memory from a storage class non-volatile memory. In order to store information in such a non-volatile fashion, magnetic hard-disk drives are largely used in modern computers. In 1968, an alternative means of realising a non-volatile memory, the first since the punch card that did not rely on magnetic properties, called the floating-gate transistor [START_REF] Robert | Field-effect transistor memory[END_REF] was proposed. The floating gate transistor works on the principle of injecting charge into a 'floating gate' by means of electron tunnelling, or hot electron injection, that takes place under application of a large voltage over the device. The charge injected into the floating gate persists in the absence of a power supply and, when a read voltage is applied to the standard gate of the transistor, the current that flows in the transistor channel is determined by the quantity of this stored charge -therefore denoting the value of the stored bit. The floating gate transistor would later be used by engineers working at Toshiba in Japan to realise NOR and NAND flash non-volatile memory cells in 1984 and 1987 which respectively use series or parallel combinations of these transistors [START_REF] Masuoka | A new flash e 2 prom cell using triple polysilicon technology[END_REF][START_REF] Masuoka | New ultra high density eprom and flash eeprom with nand structure cell[END_REF].

Since the turn of the 21 st century, resistive memory (RRAM) devices, which use the resistance of a twoterminal nano-device as a means of non-volatile storage, have been proposed based on a variety of physical mechanisms (Fig 1 .5c). Such mechanisms include the presence of conductive oxygen vacancies (OxRAM) [START_REF] Beck | Reproducible switching effect in thin oxide films for memory applications[END_REF] or a conductive bridge of metal-ions (CBRAM) [START_REF] Liu | Resistive switching: Real-time observation on dynamic growth/dissolution of conductive filaments in oxide-electrolyte-based reram (adv. mater. 14/2012)[END_REF] in an insulating oxide, the extent of crystallisation of a glass alloy in phase change memories (PCM) [START_REF] Wong | Phase change memory[END_REF] and the difference in magnetic polarisation between two ferromagnetic material layers separated by a thin insulating tunnelling layer [START_REF] Chappert | The emergence of spin electronics in data storage[END_REF] (MRAM). Ferroelectric materials have also been investigated in a similar device topology to the floating gate transistor whereby a ferroelectric material is used as a transistor gate-dielectric (FeFET) and, through the non-volatile modification of its polarisation, the read current of the transistor also changes to reflect the value of the stored bit [START_REF] Mikolajick | Feram technology for high density applications[END_REF]. RRAM are claimed to be a 'missing' fourth circuit element predicted in 1976 called the memristor: a device whose resistance varies as a function of the history of charge that has flowed through it [START_REF] Chua | Memristor-the missing circuit element[END_REF][START_REF] Strukov | The missing memristor found[END_REF]. This is hotly disputed however [START_REF] Vongehr | The missing memristor has not been found[END_REF], and in this thesis the term resistive memory, or RRAM, will be used exclusively. In any case, the resistive state (or conductive state)

of RRAM can be programmed by applying voltage and current waveforms over the two device terminals to store binary bits [START_REF] Ig Baek | Highly scalable nonvolatile resistive memory using simple binary oxide driven by asymmetric unipolar voltage pulses[END_REF] or multiple bits per cell [START_REF] Kund | Conductive bridging ram (cbram): An emerging non-volatile memory technology scalable to sub 20nm[END_REF] in a non-volatile fashion. However, the resistive states assumed after programming operations are subject to a high degree of randomness whereby, upon identical programming operations, devices assume, often drastically different, resistance states between successive programming iterations and between devices -respectively termed 'cycle-to-cycle' and 'deviceto-device' variability [START_REF] Ambrogio | Statistical fluctuations in hfo x resistive-switching memory: part i-set/reset variability[END_REF]. Such variability imposes severe limits on the number of bits that can be reliably stored in a cell with respect to other memory device approaches.

One of the principal interests in resistive memory devices is to leverage their non-volatile conductance states as analogue weight elements in non-von Neumann hardware implementations of neural network models. In particular, much like GPUs, as a means of parallelising the dot(inner)-product operations that are required in the forward propagation of an input data point to an output inference and also in the backpropagation operations required during model training. Specifically, a dot-product operation is evaluated between a matrix of conductances, organised in a cross-point array, and a voltage vector, corresponding to features of an input data point, that is applied across the array columns. The currents flowing out the array rows correspond to each element of the vector resulting from the dot-product between the voltage encoded data and the conductance based model (Fig. 1.4d). The key advantage of such an approach is that the computation occurs inside of the memory, exploiting nothing more than the physical laws of Ohm and Kirchoff -potentially circumventing a von Neumann bottleneck in the computing system [START_REF] Chen | Nanoscale molecular-switch crossbar circuits[END_REF]. To accommodate this analogue-domain parallelisation of the dot-product operation however, serious practical issues remain regarding the energy and area required due to the repeated digital-to-analog and analogto-digital conversion [START_REF] Chakraborty | Resistive crossbars as approximate hardware building blocks for machine learning: Opportunities and challenges[END_REF][START_REF] Ni | An energy-efficient matrix multiplication accelerator by distributed in-memory computing on binary rram crossbar[END_REF] and 'pitch matching' the low-area memory cross-points with large area peripheral circuits [START_REF] Levisse | Architecture, design and technology guidelines for crosspoint memories[END_REF][START_REF] Sun | Fully parallel rram synaptic array for implementing binary neural network with (+ 1,-1) weights and (+ 1, 0) neurons[END_REF]. Further problems exist in the high programming voltages, high programming currents and high read currents required [START_REF] Grossi | Experimental investigation of 4-kb rram arrays programming conditions suitable for tcam[END_REF] that not only entail large quantities of energy but pose challenges in the integration with advanced technology nodes that use much smaller nominal supply voltages. The first such neural network inference chip, whereby a neural network model was trained 'ex-situ', or 'off-line', on a von Neumann computer and then transferred onto the hardware which would perform only inference with the model, was proposed in 1991 and used the drain-source conductance of transis-tors, whose gates were biased by charged capacitors, as the analogue memory element [START_REF] Boser | An analog neural network processor with programmable network topology[END_REF]. This was shortly followed by further inference chips employing the conductance states of FeFETs [START_REF] Ishiwara | Proposal of adaptive-learning neuron circuits with ferroelectric analog-memory weights[END_REF] and floating gate transistors [START_REF] Fujita | A floating-gate analog memory device for neural networks[END_REF] as the non-volatile synaptic weight elements. The first inference chips based on arrays of re-configurable two terminal RRAM devices were then proposed in the next decade [START_REF] Choi | An electrically modifiable synapse array of resistive switching memory[END_REF]. Despite proposals to realise neural networks without overlapping memory cross-points [START_REF] Pappachen | Cognitive memory network[END_REF], the crossbar structure has come to dominate proposals for such dot-product (sometimes called multiply and accumulate)

'engines' and their design and fabrication remains, at the time of writing, an active research topic. In order to solve leakage current, or 'sneak path', issues in RRAM arrays, multiple device selection solutions have been proposed whereby diodes [START_REF] Sven M Öller | A polymer/semiconductor write-once read-many-times memory[END_REF], transistors [START_REF] Kim | One transistor-one resistor devices for polymer non-volatile memory applications[END_REF] or ovionic threshold switching thin-films [START_REF] Navarro | Innovative pcm+ ots device with high sub-threshold non-linearity for non-switching reading operations and higher endurance performance[END_REF] act to select specific devices in a memory array for reading and programming. The configurability of resistive memories, under the application of relatively simple voltage pulses that can be generated locally has also prompted research into the implementation of machine learning algorithms 'in-situ' -also referred to as 'on-chip' or 'on-line' -for feed-forward [START_REF] Alibart | Pattern classification by memristive crossbar circuits using ex situ and in situ training[END_REF][START_REF] Ambrogio | Equivalent-accuracy accelerated neural-network training using analogue memory[END_REF][START_REF] Li | Efficient and self-adaptive in-situ learning in multilayer memristor neural networks[END_REF], long-short-term-memory [START_REF] Li | Long short-term memory networks in memristor crossbar arrays[END_REF][START_REF] Wang | In situ training of feed-forward and recurrent convolutional memristor networks[END_REF] and spiking neural networks [START_REF] Snider | Spike-timing-dependent learning in memristive nanodevices[END_REF][START_REF] Querlioz | Simulation of a memristor-based spiking neural network immune to device variations[END_REF][START_REF] Prezioso | Training and operation of an integrated neuromorphic network based on metaloxide memristors[END_REF][START_REF] Woźniak | Deep learning incorporating biologically inspired neural dynamics and in-memory computing[END_REF]. When contrasted with ex-situ model training, in-situ training allows for models to be updated locally and, as such, the approach is therefore a promising avenue for bringing adaptability and learning to edge systems. Additionally, in-situ trained models have also been observed

to accommodate for local hardware and device specific defects and variability [START_REF] Li | Efficient and self-adaptive in-situ learning in multilayer memristor neural networks[END_REF].

Despite open questions regarding how to best harness its properties, RRAM is certainly a candidate for the 'right' kind of memory that could well lead to a new generation of low-energy non-von Neumann computers. Although their application has thus far been largely confined to either ex-situ [START_REF] Hu | Memristor-based analog computation and neural network classification with a dot product engine[END_REF] or in-situ [START_REF] Ambrogio | Equivalent-accuracy accelerated neural-network training using analogue memory[END_REF][START_REF] Li | Efficient and self-adaptive in-situ learning in multilayer memristor neural networks[END_REF][START_REF] Li | Long short-term memory networks in memristor crossbar arrays[END_REF][START_REF] Wang | In situ training of feed-forward and recurrent convolutional memristor networks[END_REF][START_REF] Woźniak | Deep learning incorporating biologically inspired neural dynamics and in-memory computing[END_REF] implementations of backpropagation for training deep learning models, other examples of non-von Neumann computing based on RRAM properties also exist. This includes their use in realising logic circuits [START_REF] Kvatinsky | Magic-memristor-aided logic[END_REF] as well as in true random number generation [START_REF] Chen | Utilizing the variability of resistive random access memory to implement reconfigurable physical unclonable functions[END_REF]. Additionally, intrinsic resistive memory randomness has been leveraged as a means of solving optimisation problems in RRAMbased Hopfield networks [START_REF] Cai | Power-efficient combinatorial optimization using intrinsic noise in memristor hopfield neural networks[END_REF]; in a similar fashion to how the spin-spin coupling of superconducting quantum bits has been used to implement quantum annealing algorithms in conceptually similar Ising machines [START_REF] Mark W Johnson | Quantum annealing with manufactured spins[END_REF]. Furthermore, while strictly not acting as a memory, the random switching behaviour of MRAM when driven by a constant DC bias current has been applied to perform Bayesian inference; using a tree of C-Muller elements [START_REF] Joseph S Friedman | Bayesian inference with muller c-elements[END_REF] to evaluate the joint probability of input random variables [START_REF] Vodenicarevic | Low-energy truly random number generation with superparamagnetic tunnel junctions for unconventional computing[END_REF], each represented by an MRAM device, or by interconnecting the devices to realise a Bayesian network [START_REF] Faria | Implementing bayesian networks with embedded stochastic mram[END_REF].

Future memory concepts such as magnetic racetracks [START_REF] Stuart Sp Parkin | Magnetic domain-wall racetrack memory[END_REF] which store bits as a sequences of magnetic 'domain walls' in nanowires, molecular memory cells that use switchable properties of organic molecular structures to store information [START_REF] Radosavljević | Nonvolatile molecular memory elements based on ambipolar nanotube field effect transistors[END_REF], quantum-dot based flash [START_REF] Tezuka | Quantum dot memory cell[END_REF] as well as quantum mechanical memory [START_REF] Julsgaard | Experimental demonstration of quantum memory for light[END_REF] promise further properties and opportunities that can also be one day harnessed in computer models of artificial intelligence.

Challenges in applying artificial intelligence

There are obvious conceptual challenges in artificial intelligence, namely the question of what intelligence even is and how to go about building models of it on computers are still very much open questions. The best black-box and tree-based approaches developed at the time of writing, although arguably not viable models for real intelligence, are certainly capable of reproducing specific intelligent and useful behaviours in isolation. As such it is now possible to apply AI models, neural networks for example, to solve real-world problems. However, considerable challenges still need to be confronted such that existing approaches can be applied in a practical and responsible fashion.

Training energy consumption

Artificial intelligence, deep learning above all, has benefited enormously from the aggressive transistor scaling that has increased the speed of computers and the availability of on-chip memory. However, to obtain state of the art performance in a specific task the energy and time required to train a model is often truly staggering [START_REF] Strubell | Energy and policy considerations for deep learning in nlp[END_REF][START_REF] Schwartz | Green ai[END_REF]. The AlphaGo model from DeepMind, for example, was trained over the course of a month using on the order of thousands of CPUs and hundreds of GPUs [START_REF] Silver | Mastering the game of go with deep neural networks and tree search[END_REF]. Knowing that a single GPU has a power consumption on the order of hundreds of Watts [START_REF] Huang | On the energy efficiency of graphics processing units for scientific computing[END_REF], a back of the napkin calculation suggests the total energy demanded by the GPUs alone is on the order of millions of kilowatt hours. To put this number in context, an average French household consumes on the order of a thousand kilowatt hours of energy in an entire year [165]. Is it reasonable that the same amount of energy to sustain one thousand households in a year was used to train a deep learning model to play Go in a month? At the time of writing, the AlphaGo model was produced four years ago. Since then model sizes have increased exponentially in order to achieve incremental performance gains [START_REF] Huang | Speed/accuracy trade-offs for modern convolutional object detectors[END_REF]. To train the Transformer model proposed by Google in 2017 [START_REF] Vaswani | Attention is all you need[END_REF], greenhouse gas emissions equivalent to six hundred plane tickets between New York and San Francisco or the total fabrication and running costs of five cars during their lifetimes was estimated to have been required [START_REF] Strubell | Energy and policy considerations for deep learning in nlp[END_REF]. The worlds largest model at the time of writing uses almost two hundred billion parameters [START_REF] Tom B Brown | Language models are few-shot learners[END_REF]. It has been estimated that the electricity usage alone to train it cost five million US dollars.

Limited to a single GPU, this training process would take 355 years to conclude [166]. Considering the rapid growth in the application of these extremely energy intensive AI models, which is projected to be sustained well into the future [1], serious questions have to be posed about the sustainability of allowing the field to continue in this fashion and, ultimately, end up as a major contributor to climate change.

Bringing learning to the edge

As a result of these energetic requirements, the majority of applications for AI hinge on centralised cloud computing infrastructures [167]. This means that models are trained, stored and often queried by edge computing hardware in the cloud -where massive amounts of energy, memory and computing resources are readily available. To reduce the latency and the volume of data transferred between the edge and the cloud, models can also be trained on the cloud and then transferred to dedicated edge hardware that acts to perform local inference with that trained model [START_REF] Nicholas D Lane | An early resource characterization of deep learning on wearables, smartphones and internet-of-things devices[END_REF][START_REF] Shi | Edge computing: Vision and challenges[END_REF][START_REF] Xu | Scaling for edge inference of deep neural networks[END_REF][START_REF] Zhou | Edge intelligence: Paving the last mile of artificial intelligence with edge computing[END_REF]. However, no commercially available hardware exists that can meet energy usage and memory size constraints that would enable intelligent systems to be trained and adapt themselves locally at the edge [START_REF] Shi | Edge computing: Vision and challenges[END_REF][START_REF]Edge AI Chipsets: Technology Outlook and Use Cases[END_REF]. While for many application domains this cloud-centric approach to AI is perfectly adequate, it does mean that a number of potentially revolutionary use cases are being overlooked and not subject to investigation. For example, implantable medical systems such as cardioverter defibrillators [START_REF] Barry J Maron | Efficacy of implantable cardioverter-defibrillators for the prevention of sudden death in patients with hypertrophic cardiomyopathy[END_REF], which are required to intervene with an electric shock upon detection of dangerous heart arrhythmia, have existed for some decades. Such systems might benefit tremendously from the local and patient specific application of AI that could allow their operation to be continuously adapted and optimised. Once implanted however, these medical systems may have intermittent, or no access at all, to an external power supply. Other than the tremendous heat that would be generated, a deep learning model based on a GPU within the implantable system would evidently quickly exhaust any local battery. Further than restricting the areas in which AI can be applied, the cloud-centric approach also influences the direction of research into new modelling and algorithmic ideas. In recent years , research interest has been overwhelmingly focused on the brute-force mathematical optimisation, framed as supervised learning problems, of extremely large models using pre-labelled 'big data' datasets on cloud platforms. Such a learning approach is not suitable in the edge setting. At the edge, by contrast, data is input to the model in a continuous and unlabelled stream in real-time from the environment and, due to memory constraints, likely will not be stored. Rather, ideas focusing on how to understand, learn from and respond to patterns in such a label-less deluge in real-time, on how to deal with previously unseen types of data and compensate for environmental changes and drift where the edge system exists would be preferred. At the time of writing however, since they are not required in the cloud-centric context, such modelling and algorithmic approaches either do not exist or are not being actively pursued at large by the community.

In spite of this, the market for edge learning is expected to emerge and grow rapidly over the coming decade [START_REF]Edge AI Chipsets: Technology Outlook and Use Cases[END_REF]. To enable this, the solution can be viewed from two angles: either edge hardware must be made more efficient or models of AI must be improved such they demand fewer resources. Aspects of both are undoubtedly true. To bring locally adaptive intelligence and learning to the edge, potentially opening up new application domains that are not being considered today, simultaneous and cooperative innovation in hardware and modelling are required.

Ethical application of AI

Unlike the goal-tree based approaches applied in the 1960's [START_REF] James R Slagle | A heuristic program that solves symbolic integration problems in freshman calculus[END_REF] which, after performing an action, could offer a reason as to why, the outputs of black-box models like neural networks often suffer from a lack of, or zero, interpretability [START_REF] Leilani H Gilpin | Explaining explanations: An overview of interpretability of machine learning[END_REF]. While harmless in applications such as product recommendation or in customer service chatterboxes, serious ethical questions arise for safety-critical applications to recognise cats and dogs leading to a prediction that the input data point was a cat. Reciprocally, for models trained to recognise hand-written digits, given an input of a cat, they might tell you a '7' has been presented to it. This problem arises by viewing, in the case of neural networks, output layers of neurons as a discrete probability distribution through, most often, the application of a 'softmax' function.

The output softmax ensures all of the output responses, encoding the probability of an input data point belonging to a pre-determined set of classes, sum to one [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. Therefore, when presented with classes of data on which a model has not been trained, or has observed very few examples of, it is obliged to make a prediction such that the total probability across the output sums up to one in any case without communicating any level of uncertainty about this predictions. For AI models to communicate uncertainty, they have to incorporate uncertainty into their constituent parameters. This uncertainty will then in turn propagate through a model into its output predictions [START_REF] Gal | Uncertainty in deep learning[END_REF][START_REF] Kendall | What uncertainties do we need in bayesian deep learning for computer vision?[END_REF].

A class of models which describe and communicate uncertainty in this fashion are 'Bayesian models' -named in homage to the Reverend Thomas Bayes. Bayesian models, instead of using deterministic values to describe something like a synaptic weight in a neural network, make use instead of probability distributions (i.e., random variables) [START_REF] Ghahramani | Probabilistic machine learning and artificial intelligence[END_REF]. In this fashion, uncertainty in the estimation of each parameter can be incorporated into each of the probability distributions which describe them [START_REF] Radford | Bayesian learning for neural networks[END_REF][START_REF] Gal | Uncertainty in deep learning[END_REF][START_REF] Kendall | What uncertainties do we need in bayesian deep learning for computer vision?[END_REF] and in fact becomes and integral part of their training process. For example a synapse with the value of 2.5 in a deterministic neural network might be described by a normal distribution in a Bayesian neural network where, if it was quite sure about this value, the normal distribution would be narrow (have low standard deviation) around a median of 2.5. If it were less sure about this value, the normal distribution might be very wide and spread its total probability density over a larger range of values. This parameter uncertainty, as aforementioned, then propagates through Bayesian models and manifests in their output prediction uncertainty. As a result, a model trained to recognise cats and dogs, when presented with the digit '0', might tell you, via its output prediction distribution, 'that looks more like a cat than a dog although, in all honesty, I have never seen anything like that before and I am very uncertain'. For some applications, a cloud-based model tasked with selecting car insurance policies for example, the consequence of taking wrong actions with a low frequency is largely unimportant. If the model achieves a prediction accuracy of 99.99%, the company will certainly lose some money in a small number of large money payouts, but the money likely saved through incorporation of AI into their process will overall most likely make it worthwhile.

Uncertainty is extremely important however, just like interpretability, in safety-critical applications where a single wrong decision could cost a human life. Consider an AI model tasked with controlling the actions taken by a cardioverter-defibrillator: an implantable medical device which applies electrical shocks to the hearts of patients in the event of detecting dangerous heart arrhythmias in order to restore a normal heart rhythm [START_REF] Mirowski | The automatic implantable cardioverter-defibrillator: an overview[END_REF][START_REF] Stuart | Meta-analysis of the implantable cardioverter defibrillator secondary prevention trials[END_REF]. While the detection of a true positive could save a patients life, the erroneous application of a shock to a properly functioning heart could prove fatal [START_REF] James P Daubert | Inappropriate implantable cardioverter-defibrillator shocks in madit ii: frequency, mechanisms, predictors, and survival impact[END_REF]. A model which takes such heavy decisions, where there exists an imbalance between inaction and incorrect action, should not be obliged to act when it is unsure. While a deterministic model does not know when it is unsure, it is possible to look at the output prediction distributions of a Bayesian model to understand, with what level of certainty, an action should be taken or not.

Scope of this thesis

The objective of the work presented in this thesis is to explore the inter-disciplinary boundary between the fields of artificial intelligence, emerging technologies and biological nervous systems. Specifically it will consider how resistive memory technologies can be used to support ultra-low energy approaches to AI based on neural network, or connectionist, models that can be applied in the resource constrained setting of the edge. The key will therefore be the exploitation of the non-volatility and in-memory dot-product capabilities of resistive memory, while finding methods to work with, or leverage, the other non-ideal characteristics of resisitve memory. For example, the large levels of conductance variability that currently hold back its practical application.

In order to achieve this, Bayesian machine learning approaches, compatible with the fundamental device properties of RRAM, are developed and applied experimentally to the training of RRAM-based models. Additionally, a computing fabric based on distributed resistive memory devices, capable of supporting re-configurable neural network models and inspired by the organisational principles observed animal nervous systems, is proposed. Circuits, implementing the building blocks of this computing hardware are designed, fabricated in a hybrid 130nm CMOS-RRAM process and then experimentally verified. Two neural network models inspired by two different insect sensory systems, the cricket cercal system and the Drosophila elementary motion detection system, are also developed. These cricket and Drosophila models respectively show that such bio-inspired architectures permit the memory requirements of neural network models to be reduced by orders of magnitude with respect to conventional architectures, and that bio-inspired dynamical neuron and synapse models are important for addressing tasks with temporal dependencies.

The thesis is organised into three chapters, addressing each of these three aspects independently.

While they are written to stand on their own, each chapter compliments aspects of the others and together they are intended to paint a picture of a Bayesian neuromorphic computing approach based on resistive memory. The first chapter presents the two bio-inspired models and their advantages, the second describes RRAM-compatible Bayesian machine learning algorithms that can be applied to train resistive memory based neural networks and the third chapter presents a circuit and system level description for a re-configurable RRAM-based computing fabric on which such models can be implemented. The thesis concludes with a future vision of a full Bayesian neuromorphic computing system. It is discussed how properties of the bio-inspired neural network model architectures from the first chapter might be important for addressing potential limitations foreseen in the presented machine learning and hardware approaches.

Ultimately, the work provides a blueprint for a new approach to artificial intelligence by combining ideas at the interface (Fig. 1.6) of machine learning, technology and biology, each of which plays an important, although normally independent, role in the field. The resulting computing system, by drastically reducing the energy required to implement machine learning, offers a means of bringing learning to extremely resource constrained environments at the edge. Due to certain properties of the bio-inspired and Bayesian AI models developed, namely their abilities to be interpreted and express uncertainty, their value in addressing questions regarding the ethical application AI are also discussed. As a result, it opens up the door on a new set of extremely low-energy safety-critical application domains, which cannot be addressed with existing approaches -implantable intelligent and adaptive medical systems for example.

Chapter 2

Bio-inspired neural network architectures

Chapter introduction

Intelligence in animal nervous systems, contrary to prevailing notions in AI, is more than just learning. A great number of animal behaviours such as the detection and appetitive response to blood in sharks [START_REF] Kara E Yopak | Not all sharks are "swimming noses": variation in olfactory bulb size in cartilaginous fishes[END_REF],

escape responses in mice, flies and crickets [START_REF] Dupuy | Responses of cricket cercal interneurons to realistic naturalistic stimuli in the field[END_REF][START_REF] Yilmaz | Rapid innate defensive responses of mice to looming visual stimuli[END_REF][START_REF] Klapoetke | Ultra-selective looming detection from radial motion opponency[END_REF][START_REF] Ogawa | Cercal System[END_REF][START_REF] Ache | Neural basis for looming size and velocity encoding in the drosophila giant fiber escape pathway[END_REF] as well as complex courtship and burrowing behaviours observed in a wide variety of species [START_REF] Tinbergen | The study of instinct[END_REF][START_REF] Weber | The evolution of burrowing behaviour in deer mice (genus peromyscus)[END_REF] are built-in innately through neural network architecture and neural dynamics from birth. The genome of an animal, carved out and refined during many hundreds of millions of years of evolution, contains the full instruction set which details as to how all of the neural architectures that compose a nervous system are built and interconnected. The mammalian genome however, composed of some billions of nucleotides, stores the equivalent of a mere gigabyte of information [START_REF] Wei | The molecular basis for the development of neural maps[END_REF] -and that is to build the entire animal, not just its brain. A single gigabyte pales in comparison to the memory requirements of even modest neural network models currently being applied in AI. This alludes to the existence of a smaller set of fundamental, task specific, neural architectures and computational principles which are recycled across multiple neural systems. In fact, even across wildly different species separated by half a billion years of evolution, common architectures and computing principles are observed [START_REF] Borst | Common circuit design in fly and mammalian motion vision[END_REF]. This chapter will seek to develop bio-inspired model architectures based on neuro-physiological and neuro-anatomical research currently being conducted into two insect sensory systems -the cricket cercal system and the Drosophila elementary motion detection system. Overall, the purpose is to understand as to how connectionist AI approaches might unshackle themselves from 'pure learning' [START_REF] Anthony M Zador | A critique of pure learning and what artificial neural networks can learn from animal brains[END_REF], and what the advantages may be in doing so, by taking inspiration from the innate mechanisms uncovered by evolution and stored in the animals constrained genome.

Model of the cricket cercal system escape response 2.2.1 Introduction

Neurons are the fundamental computational units of connectionist approaches to artificial intelligence [START_REF] Warren | A logical calculus of the ideas immanent in nervous activity[END_REF][START_REF] Hodgkin | A quantitative description of membrane current and its application to conduction and excitation in nerve[END_REF]. They can be understood computationally as defining a linear hyper-plane in an input feature space through application of an activation function to a weighted sum of these features. While alone these neurons can solve only simple 'linearly separable' tasks; they can, when networked together in a neural network architecture, solve more complex non-linear ones [START_REF] Minsky | Perceptrons: An introduction to computational geometry[END_REF]. The predominant approach in neural networks in recent years has been that of deep learning [START_REF] Lecun | Deep learning[END_REF], which can attribute its success to the combination of two important ideas. The first is the universal approximation theorem [START_REF] Cybenko | Approximation by superpositions of a sigmoidal function[END_REF] which states that a fully-connected feed-forward hidden layer of neurons is capable of approximating any continuous function -given an appropriate combination of synaptic parameters. The second is the use of batch backpropagation stochastic optimisation [START_REF] Linnainmaa | Taylor expansion of the accumulated rounding error[END_REF][START_REF] David E Rumelhart | Learning representations by backpropagating errors[END_REF] as a means of determining these parameter values that link together successive layers of neurons. However, deep learning models with hidden layers are characterised by non-convex loss surfaces. As such, backpropagation will converge to a locally-optimal configuration of parameters -not necessarily the global optimum -as a function of their random initialisation. In order to improve the quality of these local minima the solution has been to include many, hence deep, wide neural network layers [START_REF] Choromanska | The loss surfaces of multilayer networks[END_REF], although this ultimately leads to a unwieldy and memory inefficient model which is difficult, arguably impossible, to interpret [START_REF] Leilani H Gilpin | Explaining explanations: An overview of interpretability of machine learning[END_REF].

An important approach within deep learning, first proposed in the 1980's as the 'neocognitron' [START_REF] Fukushima | Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position[END_REF],

has developed into the field of convolutional neural networks (CNNs) which achieve state of the art performance in applications related to image processing. This is in spite of the fact CNNs are composed of a subset of the parameters of a fully-connected feed-forward network of an equivalent depth [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. Crucially, CNNs, originally inspired by research into the cat visual cortex [START_REF] David | Receptive fields, binocular interaction and functional architecture in the cat's visual cortex[END_REF], have demonstrated the potential of applying backpropagation to task-appropriate neural network architectures inspired by biology -as an alternative to the use of universal approximators. Other deep learning architectures, namely attention models [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF] and generative adversarial networks [START_REF] Goodfellow | Generative adversarial nets[END_REF], have lead to respective leaps in machine translation and novel data-point generation that further reinforce the importance of architecture in neural networks.

In the field of neuromorphic computing, where a more 'bottom-up' approach is taken to artificial intelligence, research into biological nervous systems has, as in the case of CNNs, also provided a source of architectural inspiration. This has led to models which, for example, incorporate dynamical and topological motifs inspired by the Drosophila visual, the honey-bee olfactory, cricket auditory and the cockroach motor systems into motion detection [START_REF] Dalgaty | Insect-inspired elementary motion detection embracing resistive memory and spiking neural networks[END_REF], contrast enhancement [START_REF] Schmuker | A neuromorphic network for generic multivariate data classification[END_REF], temporal pattern detection [START_REF] Sandin | Synaptic delays for insect-inspired temporal feature detection in dynamic neuromorphic processors[END_REF] and locomotion [START_REF] Randall D Beer | A distributed neural network architecture for hexapod robot locomotion[END_REF] models respectively. However, such approaches have been somewhat limited by lack of an effective means of defining model parameters, whereby manual parameter tuning or correlation-based Hebbian learning rules [START_REF] Olding | The organization of behavior: a neuropsychological theory[END_REF] are typically employed.

Here, we propose a computational model for the air current evoked jumping escape response studied in the cricket cercal system. Instead of manual tuning or Hebbian learning, we employ the backpropagation algorithm, more commonly used in the deep learning setting, as a means of model parameterisation. In contrast with deep learning, however, backpropagation is not used to find an arbitrary local loss minimum based on a random parameter initialisation, but instead as a means of steering the parameters towards optimal values consistent with the logical structure of the bio-inspired architecture that relates neurons to one another.

We find that, when applied to the detection of a simulated attacking predator, the optimised cercal system model is able to obtain the same performance as benchmark universal approximators, although requiring between one and two orders of magnitude fewer parameters. Additionally, we also find that, because the model contains neurons and synapses with well defined functional roles, it is possible to interpret how the optimised model functions and makes decisions. Ultimately, the results provide a strong basis for the incorporation of biologically inspired architectures into memory efficient and interpretable neural network models.

Model definition

The cricket cercal system The cricket cercal system has been under investigation for several decades, leading to an understanding of many of its neural and biomechanical components (reviewed extensively in [START_REF] Jacobs | Computational mechanisms of mechanosensory processing in the cricket[END_REF][START_REF] Ogawa | Cercal System[END_REF]). The nervous system of the cricket is characterised by a chain of ganglia along a nerve cord which runs from the head of the animal down to the rear of the body pictured in Fig. 2.3b [START_REF] Insausti | The terminal abdominal ganglion of the wood cricket nemobius sylvestris[END_REF][START_REF] Tc Insausti | The morphology and fine structure of the giant interneurons of the wood cricket nemobius sylvestris[END_REF]. Signals from various sensory and motor systems ascend up or descend down the nerve cord between the ganglia where information is processed to mediate the animal's behaviour. The interneurons making up the cercal sensory system are contained in the Terminal Abdominal Ganglion (TAG) at the very posterior end of the nerve cord (Figs. 2.1b,2.1c,2.1d). This cercal sensory system mediates the detection and analysis of air currents which activate sensory receptors on the crickets two rear cercal appendages. These appendages, or cerci, are long antenna-like structures, each of which is covered by up to a thousand filiform hairs as pictured in the electron microscopy image of Fig. 2.1a [START_REF] Magal | Hair canopy of cricket sensory system tuned to predator signals[END_REF][START_REF] Miller | Quantitative characterization of the filiform mechanosensory hair array on the cricket cercus[END_REF][START_REF] Heys | A model of filiform hair distribution on the cricket cercus[END_REF]. Below each hair there is a sensory neuron which, under mechanical displacement of the hair, will fire action potentials that propagate along an axon into the TAG (see Fig 2.1c). The ensemble of all of the sensory afferents on the two cerci project to specific locations within the TAG and form a sensory feature map [START_REF] Bacon | Receptive fields of cricket giant interneurones are related to their dendritic structure[END_REF][START_REF] Jacobs | Functional organization of a neural map in the cricket cercal sensory system[END_REF][START_REF] Paydar | Neural mapping of direction and frequency in the cricket cercal sensory system[END_REF][START_REF] Jacobs | Extraction of sensory parameters from a neural map by primary sensory interneurons[END_REF]. Different spatial regions of this map represent information about air-current direction and velocity transduced around the animals cercal appendages.

Within the TAG, there are approximately twenty large spiking interneurons, in addition to some thousands of smaller local-interneurons, (Fig. 2.1d) which receive direct excitatory synaptic input from the filiform sensory afferents [START_REF] Jacobs | Segmental origins of the cricket giant interneuron system[END_REF]. These are 'projecting' interneurons: they send axons up the nerve cord to higher ganglia (Fig. 2.3b). There are also an unknown number of small local interneurons within the TAG which do not send axons up the nerve cord [START_REF] Bodnar | Anatomy and physiology of identified windsensitive local interneurons in the cricket cercal sensory system[END_REF][START_REF] Baba | Differing afferent connections of spiking and nonspiking wind-sensitive local interneurons in the terminal abdominal ganglion of the cricket gryllus bimaculatus[END_REF]. Many of these are non-spiking 'graded release' neurons, that interconnect between themselves and the large spiking interneurons. Each of the large spiking interneurons has a unique anatomical projection of its dendritic trees within the sensory afferent map [START_REF] Jacobs | Integrative mechanisms controlling directional sensitivity of an identified sensory interneuron[END_REF][START_REF] Jacobs | Functional organization of a neural map in the cricket cercal sensory system[END_REF]. As a result, each of these interneurons has a unique specific responsiveness to the different properties of the external air current stimuli. The ensemble activity pattern of these interneurons encodes these stimulus parameters, and sends a compressed representation of this stimulus up the nerve cord towards higher ganglia, which in turn use this information to initiate the appropriate behaviours in other areas of the nervous system. For this study, we restrict our terminal abdominal ganglion model to neurons and stimuli that concern the escape response behaviour.

We have developed a reduced model of the cercal system. It consists of twenty-four neurons: sixteen neurons that represent an integrative input layer, seven neurons in a hidden layer which represents the projecting interneurons within the TAG, and a single neuron in the 'output layer', supposed to exist in a higher motor ganglion in the animal, whose response determines whether a jumping escape response should be initiated or not. The characteristics of the neurons in these three layers, and the rationale for their inclusion and inter-connectivty with other neurons, is described in the following sections and diagrammed in Figs. 2.2. Before that however, it is important to introduce the statistical simulation method used to generate the spiking activity of the crickets ensemble of cercal hairs that act as the input for our model.

Statistical Model of Filiform Hairs

In the cricket cercal system, neural processing begins at the cercal filiform hairs, which act as extremely sensitive mechanical filters [START_REF] Shimozawa | Structural scaling and functional design of the cercal windreceptor hairs of cricket[END_REF][START_REF] Miller | Quantitative characterization of the filiform mechanosensory hair array on the cricket cercus[END_REF][START_REF] Ma Landolfa | Stimulus-response properties of cricket cereal filiform receptors[END_REF][START_REF] Ma Landolfa | Direction sensitivity of the filiform hair population of the cricket cereal system[END_REF]. There are hairs of many different lengths on each cercus, and the length of each hair determines the range of frequencies, or speeds, of air currents to which it is most responsive. Longer hairs are more responsive to slower air currents and short ones to faster air currents. Each filiform hair has a spiking neuron at its base which is mechanically activated when the hair is displaced. These sensory neurons have been observed to fire even under extremely low background air current intensity, at rates of up to hundreds of spikes per second [START_REF] Ma Landolfa | Stimulus-response properties of cricket cereal filiform receptors[END_REF]. The base of each hair is constrained in a complex mechanical socket such that it can pivot freely along only one specific axis.

Characterisation of socket orientation distributions on the cerci have revealed four distinct populationshairs that are sensitive to air currents directions coming from 45°, 135°, 225°or 315°relative to the head of the animal, which by convention points to 0°. Further studies on the distribution of hair lengths revealed a bi-modal distribution of hairs into two broad but overlapping length categories [START_REF] Magal | Hair canopy of cricket sensory system tuned to predator signals[END_REF]. In functional terms, this general distinction can be used to categorise the two groups of hairs as responding more readily to either slow air currents (long hairs) or fast air currents (short hairs). Given that there are two cerci, each containing populations of long and short hairs that are sensitive to four directions of air current, there are sixteen sub-populations of cercal filiform hairs which respond optimally to specific, restricted combinations of air current direction and speed. A further study noted another response characteristic of a hair as a function of its length. For an accelerating burst of fast air, shorter filiform hairs respond with a reduced latency with respect to longer ones [START_REF] Steinmann | The morphological heterogeneity of cricket flow-sensing hairs conveys the complex flow signature of predator attacks[END_REF]. Whether an anatomical adaptation or a happy coincidence this latency provides a useful signature in the recognition of an attacking predator which would generate such a high speed burst of air.

Based on these findings we developed a statistical model for the generation of spikes for sixteen subpopulations of hairs: one for each air current angle (45°, 135°, 225°or 315°) for each of two hair lengths (long or short) for each of the two cerci (left or right). Each sub-population is composed of N hairs.

This model was used to generate individual simulation runs, each one second in duration, of the ensemble spiking activity of all 16 × N sensory neurons. Each simulation is characterised by five parameters.

Two of these parameters are (1) the background speed and (2) the predominant background direction of the ambient air currents. These background air currents correspond to the main source of 'noise' that would be expected in a realistic situation that could confound the animal's ability to detect the presence, distance and direction of a predator. Essentially the background air current imposes a variable level of activity in the receptor cells that, in turn, would lead to a background activity in the interneurons of the TAG. The other three simulation parameters correspond to the simulated predator attack. They specify

(3) the attack angle, (4) the attack speed and (5) the amplitude of the air current that is assumed proportional to the attacker size. For each individual one second simulation, values for these five variables are random-uniformly sampled between a lower and upper bound. An attack stimulus is presented at either 350ms or 700ms during this one second. The statistical model generates sixteen sets of N sensory neuron spike-times: one per sub-population.

The specific protocol for defining the neuron firing timestamps in an attack sequence was is as follows.

For each receptor in a long hair sub-population, a number of spike-times are sampled from a uniform random variable between zero and one second (each sample therefore corresponding to a firing time) equal to the background speed. For each receptor in a short hair sub-population, samples were drawn at half of this rate, reflecting their lower baseline firing levels [START_REF] Ma Landolfa | Stimulus-response properties of cricket cereal filiform receptors[END_REF]. The receptors in the sub-populations whose hairs were oriented towards the background direction drew an extra 25% of samples while those contrary to background direction drew 25% less. At either 350ms or 700ms in the simulation, a simulated attack was initiated from behind the animal. The attack is represented by long hair sensory receptors sampling a firing timestamp from a normal random variable centred on the attack time, and the short hair sensory receptors sampling from a normal random variable shifted forwards from the attack time by -10ms -reproducing the response latency effect reported in [START_REF] Steinmann | The morphological heterogeneity of cricket flow-sensing hairs conveys the complex flow signature of predator attacks[END_REF]. The standard deviation of the normal random variable which samples the neuron firing timestamps is equal to the inverse of the attack speed. The probability, p, of each sensory receptor sampling a firing timestamp from the normal random variable, therein the probability of the hair responding during an attack, depends on the attack angle and the attacker size. Hair sub-populations with a preferred direction oriented towards the attack angle and the sub-populations on the cerci that is on the same side of the animal as the attack originate from respond with a higher probability. The probability of a single hair sampling a spike-time during an attack is defined as;

p = p θ × p side × s, (2.1)
p θ = (1 -β 1 ) × cos|θ p -θ| + β 1 , (2.2)
p side = ((1 -β 2 ) × exp( -|π -θ| α ) + β 2 , (2.3)
where θ and θ p are the attack angle and the preferred orientation in radians of the hair respectively.

The attack angle is bounded to be between 120°and 240°. The attacker size, s, is a number between 0.75 and 1, β 1/2 are the minimum response probabilities for p θ and p side , and α defines how the response intensity decays on the contra-lateral cerci to the attack as a function of the attack angle, θ. See Appendix 6.7 for examples of the raster plots generated by this model for different sets of parameters.

Input layer: Feature map neurons

In the cricket TAG, this information on air-current direction and speed, represented by the ensemble firing pattern of the sensory neurons, is preserved through sub-population specific projections of sensory neuron spikes to specific locations of a sensory feature map. In each location of this feature map, the thousands of sensory neuron spikes that propagate from the cerci merge into continuous analogue signals which denote the intensity of a particular environmental feature [START_REF] Bacon | Receptive fields of cricket giant interneurones are related to their dendritic structure[END_REF][START_REF] John P Miller | Representation of sensory information in the cricket cercal sensory system. i. response properties of the primary interneurons[END_REF][START_REF] Jacobs | Functional organization of a neural map in the cricket cercal sensory system[END_REF][START_REF] Paydar | Neural mapping of direction and frequency in the cricket cercal sensory system[END_REF][START_REF] Jacobs | Extraction of sensory parameters from a neural map by primary sensory interneurons[END_REF]. TAG interneurons are then observed to 'tap' into various properties of this feature map through specific dendritic arborisations in particular spatial locations of the feature map.

We propose to model this feature map using the input layer of sixteen neurons depicted in Fig. 2.2a.

Each of these sixteen feature map neurons integrates the spiking input from one of the sixteen subpopulations of hairs on the left and right cerci. Each of our input layer neurons implements a leaky-integration model, in other words a parallel resistor-capacitor circuit fed by a pulsing current source. They can be described using the differential equation;

dV c dt = i in R -V c RC , (2.4)
whereby a current pulse i in , is injected onto a parallel combination of a resistor R and capacitor C, resulting in a capacitor voltage V c which decays in time to zero in the absence of an input current. This model allows the neuron to integrate temporal information within a time window defined by the product of R and C. For an analogue circuit implementation of such a model, see Appendix 6.24a. Based on observations of interneurons in the TAG, which appear to receive excitation above and below a setpoint due respective increases and decreases from ambient background air-current levels, we apply a hyperbolic tangent function to the difference between the instantaneous capacitor voltage, V c , and its average over D previous instances [START_REF] Ma Landolfa | Stimulus-response properties of cricket cereal filiform receptors[END_REF]. In other words, under stimulation by average instantaneous background air currents the output of the feature map neurons should be around zero;

V out = tanh(V c - D x=0 V x ). (2.5)
This results in an activation V out for each of the neurons in the input layer that provides the input for the neurons in the next hidden, layer. Examples of the activations as a function of time for each of these sixteen feature map neurons are plotted in Appendix 6.7.

Hidden layer: Cercal interneurons

The hidden layer is composed of a seven neuron circuit, based on our interpretation of numerous results that have been published in past neurophysiological studies. This group of seven interneurons includes four neurons 'tuned' preferentially to four specific air current directions, two interneurons tuned preferentially to different air current speeds, although relatively insensitive to direction of those air currents, and a final interneuron whose activity indicates the overall 'global' background air current intensity in all directions. These neurons represent these specific quantities through logical connections to specific subsets neurons in the input layer and, through additional lateral interactions between interneurons, shape their responses in the optimal manner. This network architecture is depicted in the three panels of Fig. 2.2.

Each of the sixteen feature map neurons in the input layer are identified using a grid which categorises them by angle, speed and side in Fig. 2.2. The rationale for the connections between the neuron elements are as follows.

• Directional interneurons: Past studies have documented several projecting interneurons that are responsive to low velocity air currents from a very restricted range of directions, and suppressed by air currents from the opposite directions; i.e., they are directionally selective [START_REF] Jacobs | Integrative mechanisms controlling directional sensitivity of an identified sensory interneuron[END_REF][START_REF] John P Miller | Representation of sensory information in the cricket cercal sensory system. i. response properties of the primary interneurons[END_REF]. These interneurons have been observed to have dendritic arbors in locations of the TAG feature map which receive sensory excitation from air currents from the direction the interneurons is responsive to [START_REF] Bacon | Receptive fields of cricket giant interneurones are related to their dendritic structure[END_REF][START_REF] Jacobs | Functional organization of a neural map in the cricket cercal sensory system[END_REF]. In our model these neurons are labelled as d45, d135, d225 and d315. Each of these interneurons receive input from the corresponding slow feature map neurons from both the left and right cerci as in Fig. 2.2a.

• Speed interneurons: Other interneurons have been observed to be responsive to a much broader range of directions and instead appear predominantly sensitive to different air current speeds from the rear of the animal [START_REF] John P Miller | Representation of sensory information in the cricket cercal sensory system. i. response properties of the primary interneurons[END_REF][START_REF] Bodnar | Anatomy and physiology of identified windsensitive local interneurons in the cricket cercal sensory system[END_REF]. In Fig. 2.2a, these are labelled as slow and fast that respectively receive input from the feature map neurons encoding air currents coming from behind due to slow and fast stimuli (originating from long and short receptor sub-populations on the cerci respectively).

• Global regulation interneuron: The seventh interneuron in the model is defined as the 'global regulation' cell, labelled as glob for short in Fig. 2.2c. Extracellular recordings of the ascending spiking activity from the TAG have shown, somewhat counter-intuitively, that the giant interneurons are more active in a relatively calm laboratory setting than in the field, where background air current intensity is considerably greater [START_REF] Dupuy | Responses of cricket cercal interneurons to realistic naturalistic stimuli in the field[END_REF]. That has lead to the proposal of a global regulation, or a background subtraction mechanism, at play in the cercal system. Furthermore, interneurons have been reported in the cricket TAG which are responsive to low speed air currents coming from all directions [START_REF] Baba | Differing afferent connections of spiking and nonspiking wind-sensitive local interneurons in the terminal abdominal ganglion of the cricket gryllus bimaculatus[END_REF]. Inspired by these two findings we propose to include the interneuron glob in the cercal escape system model as depicted in Fig. 2.2c which receives input from the slow feature map neurons in all directions on the left and right cerci and then sends regulatory synapses to the other six interneurons in the hidden layer.

As well as the specific connection pattern from the feature map neurons in the input layer to the interneurons in the hidden layer, we also incorporate functional lateral connections between these interneurons. Lateral inhibitory connections have been proposed in the cercal system [START_REF] Jacobs | Integrative mechanisms controlling directional sensitivity of an identified sensory interneuron[END_REF][START_REF] John P Miller | Representation of sensory information in the cricket cercal sensory system. i. response properties of the primary interneurons[END_REF] based on observations of direction encoding neurons being inhibited under air current stimuli coming from the opposite direction to which they are sensitive. Detailed neuro-physiological data about the neural basis for these interconnections is however very sparse, although several non-spiking local interneurons have been studied [START_REF] Bodnar | Anatomy and physiology of identified windsensitive local interneurons in the cricket cercal sensory system[END_REF][START_REF] Baba | Differing afferent connections of spiking and nonspiking wind-sensitive local interneurons in the terminal abdominal ganglion of the cricket gryllus bimaculatus[END_REF] which could well implement these connections. In our model, we do not explicitly include any extra local interneurons to mediate these lateral interactions. Rather, we define direct all-to-all synaptic connections between each of the four directionally selective neurons and between the two speed encoding neurons in addition to uni-directional synaptic connections from the speed encoding neurons to the four directionally selective ones. These connections are depicted in Fig. 2.2b. Instead of defining these as inhibitory connections we allow their sign and magnitude to be determined through the synaptic parameter optimisation process that is described in the following section. Each of these seven interneurons, like those in the input layer, implement hyperbolic tangent activation functions on a weighted summation of their inputs and a bias term. The bias terms of each of the neurons determines the activation of the neuron in the absence of external input, a parallel to the resting membrane potential in the biological cells.

Output layer: Jump neuron

In the animal, it is supposed that the resulting cercal representation ascends to a higher motor ganglion, one of those pictured in Fig. generator to coordinate an escape motor response. Therefore we propose that each of these seven interneurons in the hidden layer of the model synapse onto a final jump neuron as depicited in Fig. 2.3a.

In our model, the jump neuron implements a sigmoid activation function on the weighted sum of the interneuron activations, and a bias term. In this fashion, the output can be viewed as the probability that the original ensemble spiking activity pattern of the sensory neurons on the two cerci represents the signature of an attacking predator.

In order to determine the synaptic parameters that connect the feature map neurons to the interneurons, the interneurons to the jump neuron as well as the lateral connections between the interneurons, we apply the backpropagation algorithm [START_REF] David E Rumelhart | Learning representations by backpropagating errors[END_REF]. To obtain a dataset that can be used to train and then verify the trained model, the feature map neuron activations, V out , at 350ms and 700ms of 1000 randomly parameterised simulation episodes are used. The two resulting V out vectors, of the sixteen activations per simulation, are each then labelled as 1 (attack) or 0 (ambient) as well as with the predominant background direction during that episode. This yields labelled dataset of 2000 points of sixteen features each. A random subset of half or these points is used to train the model and the remaining half is reserved to test the model.

A multi-objective cost function was defined as the sum of the binary cross-entropy loss of the jump neuron, with respect to the attack or ambient label, and the categorical cross-entropy of the four directional sensitive interneurons, with respect to the one-hot encoded background direction label. Intuitively this optimises the model to simultaneously detect an attack signature and determine the prevailing direction of the background air currents. An additional weight decay term [START_REF] Krogh | A simple weight decay can improve generalization[END_REF] was summed with these two losses in order to penalise the optimisation converging to a solution with large parameters values and, possibly, over-fitting to the test data set.

In order to implement the lateral connections between the interneurons without introducing cyclic loops into the model, these synaptic connections were 'unrolled' by one step. This is required since the gradients that are backpropagated would otherwise circle indefinitely. This was achieved by applying the activation function on the weighted sum of the inputs from the feature map neurons and the glob neuron in a first step and then applying a second activation to the sum of this first activation, the weighted sum of the lateral connections and the bias term.

The derivative of the mean of this loss over a mini-batch of eight training data points was used to update model parameters inline with the root mean square propagation stochastic optimisation algorithm [START_REF] Tieleman | Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude[END_REF] over 50 training epochs. The PyTorch automatic differentiation python framework was used to build and train the proposed TAG model [START_REF] Paszke | Automatic differentiation in pytorch[END_REF].

Model evaluation

In order to evaluate the proposed terminal abdominal ganglion (TAG) cercal escape system model we use a receiver operating characteristic (ROC) curve. The ROC curve plots the true (TPR) and false positive rate (FPR) as the probability threshold, that rectifies the activation of the sigmoidal jump neuron into a jump (1) or don't jump (0) output, is increased from zero to one in regular steps. To reflect the imbalance between the consequences of not detecting true positive (succumbing to a predator) and responding to false positives (wasting a relatively unimportant amount of energy [219]), in the context of the insect, we define the metric 'tolerated' FPR which is the false positive rate that is be tolerated in order to achieve a minimum true positive rate. We fix the minimum TPR to 0.95 in this evaluation.

Universal approximator benchmarking

In order to benchmark the test accuracy and memory requirements of the optimised TAG model, single and three fully-connected hidden-layer neural network models are used -characteristic of a generic deep learning approach based on the universal approximation theorem [START_REF] David E Rumelhart | Learning representations by backpropagating errors[END_REF][START_REF] Lecun | Deep learning[END_REF][START_REF] Cybenko | Approximation by superpositions of a sigmoidal function[END_REF]. These were also implemented using the PyTorch framework. They were trained using backpropagation with the adaptive moment estimation optimisation algorithm [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF]. The models were trained over one thousand epochs using the same mini-batches as the TAG model. On average this was observed to be the number of epochs required for the parameters to converge to a minimum loss. It is also informative to note that this represents over an order of magnitude more training than that required by the TAG model, which required only fifty epochs.

In Figs. However, as the number of hidden layer neurons are further increased the tolerated FPR of these extremely large models drops below that of the TAG model -in some cases obtaining a tolerated FPR of 0.03. In order to understand whether this large investment in memory is worthwhile, we use the Akaike information criterion (AIC) [START_REF] Akaike | Information theory and an extension of the maximum likelihood principle[END_REF]. The AIC provides a number proportional to the sum of the model size and the negative log-likelihood of the model, evaluated on the test data set, and offers a means of comparing to the problem and that, for the extremely large deep learning models which obtain the lowest tolerated FPRs, the AIC score explodes owing to the complexity of the model. Furthermore, it is indicative to note that, even for small hidden layer sizes, it is difficult to justify the choice of a deep learning model over the use of a simpler logistic regression model.

In order to complete this comparison, we extend it to the case of pruned multi-layer perceptronswhereby memory requirements can be reduced by deleting synaptic weights with optimised values close to zero [START_REF] Lecun | Optimal brain damage[END_REF]. Specifically we consider a pruned version of the MLPs in Figs.2.6a and 2.6b and an additional version trained where an L1-norm regularisation term is summed with the loss function in order to enforce sparsity in the resulting model [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF]. The weight distributions of these MLP models are plotted in Supplementary Fig. 7 where it can be seen that the models trained using the L 1 -norm loss function have weights grouped very tightly around zero -appearing almost like a delta function. The tolerated FRP as a function of the percentage of pruned weights in the single and three hidden layer MLPs containing 256 and 32 neurons in each of their hidden layers respectively is plotted in Figs. 2.8a and 2.8b. These dimensions were selected in each case since this was the number of neurons per hidden layer, per MLP, that allowed a lower tolerated false positive rate than the TAG model. In each case, as the extent of pruning increases, the tolerated FPR, after a varying period of robustness, also increases -the point at which the tolerated FPR exceeds that of the TAG model is indicated using a labelled dot in Figs. 2.8a and 2.8b. While the L 1 -norm regularised models permit a considerable reduction in the number of weights with respect to the standard MLP before a degradation in the tolerated FPR is observed, the number of parameters required in order to maintain a performance equivalent to that of the TAG model architecture was still over an order of magnitude greater than this bio-inspired architecture.

Model interpretation

In addition to their low memory efficiency, universal approximation based deep learning approaches suffer from another major drawback in their lack of interpretability [START_REF] Leilani H Gilpin | Explaining explanations: An overview of interpretability of machine learning[END_REF]. Namely, it is difficult to ascertain as to why certain combinations of input features lead to certain output predictions as a function of the cascading layers of synaptic weights that have resulting from the application of backpropagation. This poses ethical and practical problems for many applications of artificial intelligence such as that addressed here which, certainly from the perspective of the cricket, can be considered as safety-critical -the model output entails potentially dangerous consequences (i.e., not jumping in the presence of an attacking predator). In contrast to the wide fully-connected layers of neurons in universal approximators, the proposed TAG model contains eight neurons and 63 synapses -each defining a set of logical relationships between the neurons which each have well defined functional roles. As such, based on its optimised synaptic weights, we make first-order structural and behavioural interpretations of the TAG model.

Structural interpretation

A table of the optimised synaptic weights inter-connecting the input layer neurons with the interneurons is shown in Fig. 2.9a. Each cell in the table contains the synaptic weight value resulting from the backpropagation based training. The majority of these feed-forward connections are excitatory such that, without consideration of the lateral connections, the interneuron activations would be proportional to the activations of the input layer neurons that connect to them. Therefore, the activation of each of the directional neurons for example, encodes the instantaneous intensity of slow air currents coming from each of the four input air current directions. It is interesting to note the imbalance in the excitation received from the left and right cerci in the cases of d135, d225 and fast whereby sensory activity coming from 135°is stronger from the left cerci and activity from 225°is stronger from the right. This in fact appears to model the dependency on attack direction described in Equations 2.2 and 2.3 whereby attacks coming from the left-hand side excite the sensory neurons oriented at from 135°more; and those from the right-hand side excite the 225°sensory neurons more.

Elevated co-activations of d135, d225 and fast will therefore correlate with an attack event -in particular the activation of fast. Another feature that stands out is the weak negative feed-forward connections from the input neurons, corresponding to air-flow from attack angles, that seems to help decouple the high air-currents resulting from an attack and the activation of glob which is intended encode the overall background air-current intensity. Intriguingly, the inhibition coming from the 135°input neurons on the left and right cerci is, in this case, elevated with respect to that coming from 225°. This angular asymmetry is also observed in the weak excitatory inputs from 135°and 225°inputs from the right and left cerci respectively that are pre-synaptic to the fast neuron. It is unclear what is the purpose of this subtle adaptation. A second pair of tables in Fig. 2.9b shows the sign and weight of the optimised synaptic weights inter-connecting the hidden layer neurons as well as their optimised biases. An immediately eye-catching feature is the symmetry observed in the connectivity pattern between the direction encoding interneurons.

The interneurons are all seen to strongly inhibit their opposing counterpart -for example the complimentary negative synaptic connections between d45 and d225. Excitingly this is consistent with predictions that have been made regarding the biological system whereby the reduction in directional interneuron activity, when presented with an air-current stimulus contrary to its preferred direction, has been attributed

to the existence of laterally inhibiting interneurons between them [START_REF] John P Miller | Representation of sensory information in the cricket cercal sensory system. i. response properties of the primary interneurons[END_REF]. Furthermore, these hidden layer interneurons send excitatory lateral connections to their ipsa-lateral and contra-lateral direction encoding neurons -notably the excitation towards the contra-lateral neuron is consistently greater. Based on the presence of the predicted lateral inhibitory interactions, this result hints at the possible existence of additional lateral excitatory connections in the biological system.

The post-synaptic connection weights of glob onto the six interneurons seen in Fig. 2.2a suggest a very interesting functional role of this neuron. Since these weights are negative and glob implements a hyperbolic tangent activation function, therein activating negatively for low background speed and positively for high background speed, glob acts as an excitor given lower background air current levels but, intriguingly, as an inhibitor when the background levels are higher. This latter effect results from the posi-tive product of the negative neuronal activation and the negative synaptic weight value. This is once again an exciting result given the prediction on the presence of a global regulatory or background subtraction mechanism in the biological system [START_REF] Dupuy | Responses of cricket cercal interneurons to realistic naturalistic stimuli in the field[END_REF].

Finally, it is informative to read off the synaptic weights connecting the hidden layer interneurons to the output jump neuron as a logical 'bar-code' -offering a means of understanding what combinations of hidden layer activations should result in elevated activations of the jump neuron. The jump neuron is inhibited due to pre-synaptic connections from d135, d225 and glob. This is likely an adaptation that allows the model to reduce the false positive rate under high background air current levels or high prevailing air-currents originating from the same directions as an attack. Another noteworthy feature of this synaptic bar-code is the opposing strong inhibition and strong excitation from the pre-synaptic slow and fast interneurons. Once again, since the neurons implement hyperbolic tangent functions, the respective negative and positive co-activation of slow and fast will result in a joint strong positive excitation of jump.

This suggests that a key distinguishing feature of an attack is the divergence between the activations of fast and slow. The optimised TAG model further enhances this divergence through the lateral connections betweenslow and fast. In the hidden layer, slow synapses positively onto fast, and fast negatively onto slow -the more positive the activation of fast therefore, the more negative the resulting activation of slow and vice versa.

Functional interpretation

Although it arises as a direct consequence of the structural properties of the model, another interpretation of the model can be made from a functional perspective -therein how the neurons activate when presented with input data-points. In order to build up this picture, the probability distributions of interneuron activations over the 1000 test data-points are plotted and discussed.

The activation distributions of the directionally selective interneurons are shown in Fig. 2.10. Resulting from the excitatory input from the relevant feature map neurons, and the response shaping due to the lateral connections, each of these distribution shows a clear bi-modal response. Specifically, for a background direction in the preferred direction of the interneuron, the activation is strongly positive, and, for a background direction coming from any other angle, the response is strongly negative. Functionally these interneurons are seen to implement a 'one-hot' encoding of the background direction, reminiscent of winner-take-all computational mechanisms that are equally achieved through competitive lateral interactions between neurons [START_REF] Lazzaro | Winner-take-all networks of o (n) complexity[END_REF][START_REF] Maass | On the computational power of winner-take-all[END_REF].

The fascinating computation carried out by the slow and fast interneuron pair is depicted in Fig. points is shown in Fig. 2.12. Consistent with its proposed dual role as an inhibitor and as an excitor the glob activation responds with broad, largely negative or positive, activation distributions for low and high background air current intensities respectively. As reflected by the large area under the ROC curve of the optimised TAG model in Fig. 2.4a, the eventual attack and ambient jump distributions over all of the test data-points for the sigmoidal jump neuron are well separated. The response is predominantly a binary one, with the output layer activating very close to zero or one in most cases. Adaptation of the probability threshold, which is effectively a vertical line drawn through the distribution of Fig. 2.12b, allows the tolerated false positive rate to be tuned.

Model of the Drosophila elementary motion detection system 2.3.1 Introduction

In recent years detailed partial connectomes of insect neural networks have been produced. An example is the elementary motion detection (EMD) network of the Drosophila visual system [START_REF] Takemura | A visual motion detection circuit suggested by drosophila connectomics[END_REF][START_REF] Maisak | A directional tuning map of drosophila elementary motion detectors[END_REF][START_REF] Haag | Complementary mechanisms create direction selectivity in the fly[END_REF]. The insect has two large eyes composed of repeating hexagonal columns called ommatidia (Fig. they indicate a looming stimulus on a collision course [START_REF] Klapoetke | Ultra-selective looming detection from radial motion opponency[END_REF] -like the 'jump neuron in the TAG model of section 2.2 this neuron triggers an escape response in the animal. Furthermore, it has also been found that the neuromodulator octopamine tunes dynamics of the EMDs in the Drosophila visual system as a function of whether the insect is resting or flying [START_REF] Nicola | Flight activity alters velocity tuning of fly motion-sensitive neurons[END_REF][START_REF] Suver | Octopamine neurons mediate flightinduced modulation of visual processing in drosophila[END_REF][START_REF] Arenz | The Temporal Tuning of the Drosophila Motion Detectors Is Determined by the Dynamics of Their Input Elements[END_REF]. This allows the insect to adapt its sensitivity to different velocities of stimulus as well as reduce power consumption whilst in a resting state. In Fig. 2.14(c) the response of an LTPC cell is reported for Drosophila stimulated with a moving grating when it is in resting and flying states. The area under the curve for the insect in its resting state is greatly reduced relative to that of its flying state which is thought to be an evolutionary adaptation to optimise its consumption of energy. The Hassenstein-Reichardt EMD (HR-EMD) is a popular model which reproduces experimental observations of the Drosophila EMDs [START_REF] Hassenstein | Systemtheoretische Analyse der Zeit-, Reihenfolgen-und Vorzeichenauswertung bei der Bewegungsperzeption des R üsselk äfers Chlorophanus[END_REF]. Photo-excitation at adjacent regions in the visual field propagates signals through crossing low pass filters, performing a delay function in the temporal domain, before being recombined at a multiplication unit to detect spatiotemporal correlations.

The output of the unit can be either positive or negative indicating motion in one of two directions as in Fig. 2.14(d). A number of previous works have implemented the HR-EMD in analogue very large scale integrated (aVLSI) systems for point [START_REF] Harrison | An Analog VLSI Model of the Fly Elementary Motion Detector[END_REF], 1D [START_REF] Liu | A neuromorphic aVLSI model of global motion processing in the fly[END_REF], 2D [START_REF] Harrison | A biologically inspired analog IC for visual collision detection[END_REF] and rotational motion [START_REF] Plett | Bio-inspired visual ego-rotation sensor for MAVs[END_REF]. Another perspective on hardware based motion detection are token and feature based EMDs [START_REF] Kramer | Compact integrated motion sensor with three-pixel interaction[END_REF][START_REF] Kramer | Pulse-based analog VLSI velocity sensors[END_REF][START_REF] Sarpeshkar | Analog VLSI architectures for motion processing: from fundamental limits to system applications[END_REF][START_REF] Shoemaker | Implementation of Visual Motion Detection in Analog Neuromorphic Circuitry -A Case Study of the Issue of Circuit Precision[END_REF].

Here we propose an alternative approach for motion detection based on dynamic exponential synapse and leaky-integrate and fire neuron models -that are relatively simple to implement in a computer model as well as with analogue electronic circuits [START_REF] Indiveri | Neuromorphic silicon neuron circuits[END_REF][START_REF] Moritz B Milde | Spiking elementary motion detector in neuromorphic systems[END_REF]. Furthermore, we use a model of the dynamic vision sensor (DVS) to provide spiking, or 'event-based', input for the resulting spiking neural network model.

Model definition

The final model is realised by a two-dimensional tiled matrix of 'coincidence detection modules' whose outputs are networked in a specific fashion to a small number of readout neurons which provide information on global motion in the visual field. The visual field corresponds to the ON and OFF events produced by a simulation of a grating moving either UP, DOWN, LEFT or RIGHT in front of a DVS camera [START_REF] Lichtsteiner | A 128×128 120 dB 15µs Latency Asynchronous Temporal Contrast Vision Sensor[END_REF][START_REF] Serrano-Gotarredona | A 128×128 1.5% Contrast Sensitivity 0.9% FPN 3µs Latency 4 mW Asynchronous Frame-Free Dynamic Vision Sensor Using Transimpedance Preamplifiers[END_REF] at a range of different speeds. The objective of the model is to report the global direction of motion in the presence of local 'salt and pepper' noise.

Coincidence detection module

To detect motion in one direction along a single dimension a temporal delay can be implemented between two spatially adjacent inputs followed by a downstream mechanism for detecting spatiotemporal correlations. In Drosophila this delay is thought to be implemented between pairs of neurons in the medulla [START_REF] Behnia | Processing properties of on and off pathways for drosophila motion detection[END_REF] and the correlation performed by the postsynaptic neurons in the lobula. Such a delay can be implemented by injecting the current of an exponential synapse into a leaky-integrate and fire neuron model;

di syn dt = -i syn τ syn , (2.6)
dV neu dt = i in W -V neu τ neu , (2.7)
where i syn and V neu are the synaptic current and neuron input voltage and τ syn and τ neu control the decay in time of these two variables respectively [START_REF] Sheik | Exploiting device mismatch in neuromorphic vlsi systems to implement axonal delays[END_REF]. The input current of each neuron, i in , is defined as the instantaneous sum of i syn from all pre-synaptic neurons. The value i syn of each synapse can be masked by +1 or -1 to define it as either excitatory or inhibitory. By adjusting the variables τ syn and τ neu , the delay properties can be modified. The variable W is the synaptic weight that determines how much V neu is affected for a non-zero i syn . Although in an electronic circuit implementation of this model tau syn is often a function of W , the parameters are treated independently in this computational model for the means of simplicity. If the value of V neu exceeds a threshold value V th , the neuron will 'fire'. At this point, each of the synapses to connecting the firing neuron to a post-synaptic neuron will have their value of i syn stepped by i spk . A correlation operation can be performed by parameterising a neuron to fire only when two pre-synaptic spikes arrive within a short, pre-determined, time window. Chaining delay and correlation operations in sequence can be achieved using the coincidence detection module shown in Fig. 2.15(a). Neurons in the input layer fire to denote spatial activity, an event produced by the DVS camera for example, and synapse with excitatory connections onto a second delay layer. In this layer the delays implemented by the B1 and B3 neurons are larger than the central B2 neuron. B1 and B2 synapse with excitatory connections onto the correlator neuron C1 in the output layer. Neuron B3 synapses with an inhibitory connection that acts to suppress C1 from firing. As depicted in Fig. 2.15(b), if the input layer is excited in the sequence A1, A2 then A3, therein along its 'preferred direction', the firing times of B1 and B2 should be 'pushed' together in time, and the input from B3 will arrive at C1 last. If the two excitatory spikes from B1 and B2 to arrive at C1 within a sufficiently short time window, it will fire before the arrival of the inhibition from B3. To suppress C1 from firing due to motion contrary to the preferred direction, referred to as the 'null' direction, of the module (i.e. an excitation in the sequence A3, A2 then A1) the firing times of B3 and B2 will be pushed together in time and that of B1 will be delayed. Due to the inhibition resulting from B3, C1 will not be sufficiently excited to fire. In a third case where all input elements are excited simultaneously, due to sensory noise for example, B1, B2 and B3 will fire at approximately the same time and negate each others contribution. The parameters of the model can be set such that C1 does not receive sufficient excitation in this scenario to fire.

To allow for the detection of motion in two dimensions, four identical coincidence detection modules, sharing a common central short delay node, can be arranged orthogonally -such that each module is

Model evaluation

A DVS of resolution 10 × 10 is stimulated using a grating (a series of bars, 1 pixel in width, at 2 pixel intervals) moving UP, DOWN, LEFT or RIGHT in time provides OFF pathway spikes for the inputs of a matrix of the 2D motion detection networks (Fig. 2.16). Noise is simulated through setting each DVS pixel to spike in the absence of an OFF pathway event with a probability of 0.025 per timestep. The frequency of the grating is defined as pixels crossed by the grating per second of simulation time. Twenty 2D motion detection networks are used to span the visual field -each of which contains thirteen neurons and twenty one synapses. In total the topology requires 256 neurons and 580 synapses for the 10 × 10 visual field.

The simulation advances in discrete time steps at which the synaptic currents and neuron input and output voltages are updated with respect to their values at the previous timestep in accordance with equations 2.6 and 2.7. The desired outcome is for the readout layer neuron corresponding to the direction of motion of the grating to fire at an elevated rate relative to the others upward motion to fire at an elevated rate relative to the others. The F 1 score [START_REF] Powers | Evaluation: From precision, recall and f-factor to roc, informedness, markedness correlation[END_REF] in the detection of upward motion is used to quantify the performance of the model;

F 1 = 2U P 2U P + (DOW N + LEF T + RIGHT ) , (2.8)
where UP, DOWN, LEFT and RIGHT are the number of times each of the direction encoding readout neurons fired during the simulation. F 1 score ranges from a minimum value of zero to a maximum value of one. The grating is swept over the input at a range of frequencies and the F 1 score is calculated per frequency. We refer to the plot of F 1 score with grating frequency as the sensitivity tuning curve (STC).

As plotted in Fig. 2.14(c), the LPTCs of Drosophila have been measured to be responsive over a broad range of grating frequencies.

In order to determine the parameters of the model, a genetic algorithm is applied [START_REF] Simon | Evolutionary Optimization Algorithms[END_REF][START_REF] Montana | Training Feedforward Neural Networks Using Genetic Algorithms[END_REF]. Specifically the genetic algorithm determines the parameters of one coincidence detection module that are common to each orthogonal rotation in the 2D motion detection network and to each of twenty modules that span the visual scene and also determines the parameters of the readout neurons -each of which also share the same values. The parameters are the synaptic weights of all synapses W the synaptic and neural time constants τ syn and τ neu as well as the threshold voltages of all neurons v th . The model parameter space therefore has twenty-four dimensions.

Sixty networks are created per generation. First generation parameters are assigned by sampling from a uniform distribution between a lower and upper bound per parameter. A sensitivity tuning curve (STC) is produced for each of the sixty networks. The ten with the largest area under the STC curve (AUC), therefore the ten networks with the best F 1 score over the grating frequency sweep, in addition to two randomly selected ones are recombined in pairs to produce the next generation of networks. Parameters are subject to 'soft' and 'hard' mutations with a certain probability. Hard mutations occur with a probability of 0.05 whereby the parameter is randomly assigned a value from a uniform distribution between an upper and lower bound. A soft mutation occurs with a probability of 0.5 whereby the parameter is reassigned a value from a normal distribution around the value inherited from the parent with a standard deviation of 1%. The average F 1 score of the ten best networks as a function of the generation is shown in Fig. 2.18

where it increases rapidly during the first twenty generations before levelling off an improving in a more incremental fashion. The sensitivity tuning curves resulting from the best performing networks after termination of the genetic optimisation after sixty generations are plotted in Fig. 2. [START_REF] Andreyevich | Extension of the limit theorems of probability theory to a sum of variables connected in a chain[END_REF]. In total two network configurations were determined inspired by the adaptation observed in Drosophila whereby the neurotransmitter octopamine adapts the speed sensitivity of its elementary motion detectors between flying and resting states. Therein, one configuration is optimised to detect lower frequency gratings and the other to detect higher frequency gratings -they are referred to as the Slow and Fast network configurations respectively. The Slow configuration was optimised first and then, using the resulting optimal values for V th , the Fast configuration was determined in a second step. The Slow and Fast configurations are plotted with respective red dashed and green traces in Fig. 2.19(a). The Slow state accurately detects the direction of motion within a range of grating frequencies of 0.7-3Hz while the Fast state does so within a range between 2-20Hz. Further, to strike a greater parallel to experimental measurement of this behaviour, reported in Fig. 2.14(c), firing-rate tuning curves are plotted in Fig. 2. 19(b). This plots the firing rate of the UP neuron, when stimulated with an upward moving grating, over the same range of grating frequencies for both the Slow and Fast network configurations. Since the area under the firing-rate tuning curves of Fig. 2.19(b) is greater over the range of grating frequencies when the network is in the Fast state than the Slow state the energy consumed by firing activity in the readout network in the Slow configuration is reduced by 42% relative to the Fast one.

The result of Fig. 2.19 shows that, like Drosophila, the range of velocities of stimulus to which the system is most sensitive to can be adapted by switching synaptic weights and time constants between two sets of values. Additionally, when the network is switched to the Slow state the power consumption of the system is reduced since it responds less to non-relevant higher speed stimuli. The firing-rate tuning curve corresponding to the response frequency of the neuron defined as the number of times the UP neuron fires per second over the same range of grating frequencies. It can be seen that, resulting from the reduced response in the Slow configuration, the power consumption is reduced in this state relative to that of the Fast configuration inline with the reduced area under the tuning curve.

Chapter discussion

Although the two presented models were inspired by different perception systems found in two different insect species, it is fascinating to note the parallels between them. For instance, just as the filiform hairs on the cricket cerci sort local air-flow into one of four angles, the elementary motion detection circuits in the the lamina and medulla of Drosophila sort local motion into one of four cardinal directions [START_REF] Matthew S Maisak | A directional tuning map of drosophila elementary motion detectors[END_REF]. Also, just as specific regions of the TAG integrate spikes from the sensory neurons of the cerci into distinct regions corresponding to direction, the array of elementary motion detection circuits integrate their global activity in specific regions of the Drosophila lobular plate -forming a directional and spatial feature map of its visual scene. Furthermore, in the same fashion as the seven interneurons in our TAG model form a compressed representation of the thousands of spikes triggered on the two cerci, Drosophila boils down the high dimensional space of its original optical input into a representation of 27 interneurons. As in our model each of these interneurons encodes certain properties of the visual scene [START_REF] Wu | Visual projection neurons in the drosophila lobula link feature detection to distinct behavioral programs[END_REF]. One of the most interesting parallels with the optimised TAG model, although the escape response was not incorporated into the elementary motion detection model, is the observation that the Drosophilas giant fibre neuron, that descends towards the animals motor ganglion to initiate a jumping escape response, sums excitatory and delayed inhibitory input from two of these interneurons -one encoding stimulus speed and the other stimulus size [START_REF] Ache | Neural basis for looming size and velocity encoding in the drosophila giant fiber escape pathway[END_REF]. This resonates with the excitation and delayed (inline with [START_REF] Steinmann | The morphological heterogeneity of cricket flow-sensing hairs conveys the complex flow signature of predator attacks[END_REF]) inhibition from the fast and slow neurons in the optimised TAG model. This alludes to a common set of design principles in, at least, insect sensory systems consistent with the observation that, since the animal genome is relatively constrained in the information it can store [START_REF] Wei | The molecular basis for the development of neural maps[END_REF], animals should be applied to recycle the sets of neural architectures and computational mechanisms across different systems. In the these two systems, information is sorted into fundamental attributes as early-on as possible: the cricket sensory neurons reporting air current in a specific direction for example, and thereafter using a global set of such attributes to build a sensory feature map. By tapping into, and combining in a non-linear fashion, various activity in various spatial locations over this feature map, the ensemble of information is compressed aggressively into a representation, arguably an abstraction, comprising a much smaller set of neurons -seven in our model and twenty-seven in the Drosophila visual system. The output of these neurons is then sent only to where it needs to be , in order to drive appropriate behaviours -escape responses for example.

The objective of this chapter was to develop two neural network model architectures inspired by biology. While the model architectures were inspired from research into the biological systems, several trade-offs were made from the standpoint of practicality. For example, in the case of the cercal system model, the sensory feature map that is realised in the animal through a complex network of sensory afferents, was simplified into sixteen leaky-integrating hyperbolic tangent neurons. Also, instead of modelling the seven interneurons in the model as spiking neurons, which they are in the biological system, hyperbolic tangent functions were used instead. Despite deviating from the exact biology, these practical modelling choices were still able, however, to capture the fundamental underlying computing principles.

This highlights the difference between bio-mimicry and bio-inspiration, at least in the context of neural networks, whereby the former aims to reproduce biology exactly and the latter to extract and apply its key principles -although not necessarily in the same fashion. Surprisingly, despite this deviation from the exact biology, two predictions from electrophysiological studies into the cricket terminal abdominal ganglion were validated in our model to be computationally important in the efficient detection of simulated attacking predators. Specifically the presence of lateral inhibitory connections between the directionally selective interneurons predicted in [START_REF] John P Miller | Representation of sensory information in the cricket cercal sensory system. i. response properties of the primary interneurons[END_REF] and the global regulation, or background subtraction, mechanism

proposed by [START_REF] Dupuy | Responses of cricket cercal interneurons to realistic naturalistic stimuli in the field[END_REF]. These results raise the interesting question as to what else such bio-inspired neural network models might tell us about the mechanisms at play in the cercal system. For example, could the lateral excitatory connections between directionally sensitive interneurons or the divergent connections between air current speed encoding neurons also exist in the biological system?

The two sections of this chapter respectively showed the potential of incorporating bio-inspired architectures, for the purposes of memory efficiency and interpretability, and the importance of dynamical neuron and synapse models in representing time. Therefore an interesting future direction of work is in understanding how to best marry these two facets together -for example, how to properly leverage dynamical neuron and synapse models in the TAG model architecture. Work in this direction has already explored how backpropagation can be applied to training feed-forward spiking neural networks [START_REF] Haeng | Training deep spiking neural networks using backpropagation[END_REF][START_REF] Emre O Neftci | Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks[END_REF] using gradient approximations to the non-differential spiking neuron activation as well as topologies based on dynamical sigmoidal neurons [START_REF] Woźniak | Deep learning incorporating biologically inspired neural dynamics and in-memory computing[END_REF].

What is clear is that, contrary to the direction of deep learning, intelligence in animal nervous systems is more than learning. Rather innate architectures, that have been discovered over the course of evolution, provide built-in solutions for many tasks. Not only looming as discussed here, but also more complex behaviours concerning courtship and burrowing too [START_REF] Tinbergen | The study of instinct[END_REF][START_REF] Weber | The evolution of burrowing behaviour in deer mice (genus peromyscus)[END_REF]. With the advent of recent methods that permit increasingly detailed study of animal nervous systems, ranging from electron microscopy based connectome reconstruction [START_REF] Takemura | A connectome of a learning and memory center in the adult drosophila brain[END_REF] to opto-genetic-based electrophysiology [START_REF] Deisseroth | Optogenetics[END_REF], subsequent years promise that such architectures can be more easily discovered and then transferred into neural network models like those presented here. An important question to address with future work is how to scale the approach to larger models whereby many such bio-inspired modules can be inter-linked in a larger network -resemble something more like an 'artificial nervous system' as opposed to merely an artificial neural network.

Chapter 3

Bayesian machine learning with resistive memory

Chapter introduction

In the previous chapter it was discussed and demonstrated how principles from animal nervous system could be used to inspire neural network model architectures. The two models presented were implemented in software on a von Neumann computer and, as such, their parameters could be determined in an arbitrary fashion -respectively, backpropagation and a genetic algorithm were used. However, this thesis is concerned with exploiting the non-volatility of resistive memory in order to support non-von

Neumann RRAM-based neural network approaches. In using resistive memory, the determination of an optimal set of parameters is non-trivial owing to their random properties and the resultant lack of precision with which parameters can be determined. Therein, this chapter deals with the open question of what approach to machine learning can be compatible with these non-ideal device properties that currently impede the practical application of RRAM-based machine learning models. Unlike the genetic and gradient-based approaches used the previous chapter, this chapter turns instead to the Bayesian machine learning framework where inherent device randomness can be actively harnessed rather than acting as the principal roadblock.

In-situ resistive memory based Markov Chain Monte Carlo

Introduction

A tantalising prospect for the future of computing is the realisation of standalone systems capable of acting, adapting and learning from new experience locally at the edge [START_REF] Shi | Edge computing: Vision and challenges[END_REF] -independent of the cloudwhile simultaneously observing severe constraints on energy consumption, data availability and memory size. While there are no commercial systems currently available that can meet these requirements, edge learning is an application domain that is projected to emerge and grow over the coming decade [START_REF]Edge AI Chipsets: Technology Outlook and Use Cases[END_REF]. Edge learning faces particular challenges given the noisy and limited raw sensory information encountered in the complex environments where such systems could be deployed -an implanted medical system re-quired to locally update its operation based on the evolving state of a patient for example. The models and algorithms within the domain of machine learning offer the enabling tools for such systems. However, until recently, little attention has been given to the hardware that underpins their inherent computation.

Machine learning models are trained using general purpose hardware which inherits from the von Neumann organisation [START_REF] Neumann | First draft of a report on the edvac[END_REF]. This entails spatially separate processing and memory and does not owe itself to energy-efficient learning. For example, state of the art performance in machine learning is currently being obtained with neural network models which feature a very high number of parameters [START_REF] Yann Lecun | Deep learning[END_REF] that are determined using the backpropagation algorithm [START_REF] David E Rumelhart | Learning representations by backpropagating errors[END_REF] and a massive volume of data. The energy required to train such models can be staggering due to the transfer of vast quantities of information between memory and processing centres on the hardware [START_REF] Strubell | Energy and policy considerations for deep learning in nlp[END_REF][START_REF] Li | Evaluating the energy efficiency of deep convolutional neural networks on cpus and gpus[END_REF]. Although cloud computing platforms offer a centralised solution for such energy and data intensive training, these demands are not consistent with the requirements of edge learning [START_REF] Shi | Edge computing: Vision and challenges[END_REF]. For edge applications therefore, it is required to abandon the von Neumann paradigm in favour of another, where memory and processing can co-exist at the same location.

Resistive random access memory (RRAM) technologies, often referred to as memristors [START_REF] Chua | Memristor-the missing circuit element[END_REF][START_REF] Strukov | The missing memristor found[END_REF],

hold fantastic promise for realising such in-memory computing systems, owing to their extremely efficient implementation of the dot-product (or multiply-and-accumulate) operation that pervades machine learning -relying simply on Kirchoffs current law [START_REF] Xia | Memristive crossbar arrays for brain-inspired computing[END_REF][START_REF] Ambrogio | Equivalent-accuracy accelerated neural-network training using analogue memory[END_REF][START_REF] Prezioso | Training and operation of an integrated neuromorphic network based on metal-oxide memristors[END_REF][START_REF] Wang | Reinforcement learning with analogue memristor arrays[END_REF][START_REF] Hu | Memristorbased analog computation and neural network classification with a dot product engine[END_REF][START_REF] Li | Efficient and self-adaptive in-situ learning in multilayer memristor neural networks[END_REF][START_REF] Yao | Fully hardware-implemented memristor convolutional neural network[END_REF] (Fig. 3.2). These technologies come in many flavours [START_REF] Wong | Phase change memory[END_REF][START_REF] Chappert | The emergence of spin electronics in data storage[END_REF][START_REF] Liu | Resistive switching: Real-time observation on dynamic growth/dissolution of conductive filaments in oxide-electrolyte-based reram (adv. mater. 14/2012)[END_REF][START_REF] Beck | Reproducible switching effect in thin oxide films for memory applications[END_REF], and intense effort is currently directed towards their use as synaptic elements in hardware neural networks for edge computing systems [START_REF] Xia | Memristive crossbar arrays for brain-inspired computing[END_REF][START_REF] Ambrogio | Equivalent-accuracy accelerated neural-network training using analogue memory[END_REF][START_REF] Prezioso | Training and operation of an integrated neuromorphic network based on metal-oxide memristors[END_REF][START_REF] Wang | Reinforcement learning with analogue memristor arrays[END_REF][START_REF] Hu | Memristorbased analog computation and neural network classification with a dot product engine[END_REF][START_REF] Li | Efficient and self-adaptive in-situ learning in multilayer memristor neural networks[END_REF]. Currently, approaches for training such systems in-situ revolve around in-memory implementations of the back-propagation algorithm [START_REF] Ambrogio | Equivalent-accuracy accelerated neural-network training using analogue memory[END_REF][START_REF] Gokmen | Training deep convolutional neural networks with resistive cross-point devices[END_REF][START_REF] Wang | Reinforcement learning with analogue memristor arrays[END_REF][START_REF] Hu | Memristorbased analog computation and neural network classification with a dot product engine[END_REF][START_REF] Li | Efficient and self-adaptive in-situ learning in multilayer memristor neural networks[END_REF]. However, implementing back-propagation in such hardware remains a formidable challenge due to multi-fold non-ideal device properties: non-linear conductance modulation [START_REF] Burr | Experimental demonstration and tolerancing of a large-scale neural network (165 000 synapses) using phase-change memory as the synaptic weight element[END_REF], lack of stable multi-level conductance states [START_REF] Garbin | HfO 2 -Based OxRAM Devices as Synapses for Convolutional Neural Networks[END_REF][START_REF] Sebastian | A collective relaxation model for resistance drift in phase change memory cells[END_REF], as well as device variability [START_REF] Guan | On the switching parameter variation of metal-oxide rram-part i: Physical modeling and simulation methodology[END_REF][START_REF] Yu | On the switching parameter variation of metal oxide rram-part ii: Model corroboration and device design strategy[END_REF]. Considering these real device properties, the performance of systems can be lower than that obtained on conventional computing systems [START_REF] Sidler | Large-scale neural networks implemented with non-volatile memory as the synaptic weight element: Impact of conductance response[END_REF][START_REF] Bennett | Wafer-scale taox device variability and implications for neuromorphic computing applications[END_REF][START_REF] Agarwal | Resistive memory device requirements for a neural algorithm accelerator[END_REF]. Several non-ideality mitigation techniques [START_REF] Ambrogio | Equivalent-accuracy accelerated neural-network training using analogue memory[END_REF][START_REF] Gokmen | Training deep convolutional neural networks with resistive cross-point devices[END_REF][START_REF] Nandakumar | Mixedprecision architecture based on computational memory for training deep neural networks[END_REF][START_REF] Hu | Memristorbased analog computation and neural network classification with a dot product engine[END_REF][START_REF] Li | Efficient and self-adaptive in-situ learning in multilayer memristor neural networks[END_REF][START_REF] Boybat | Neuromorphic computing with multi-memristive synapses[END_REF] enhance system accuracy but ultimately curtail the efficiency of the in-memory computing approach. On the contrary, approaches based on neuroscience-inspired learning algorithms, such as spike-timing dependent plasticity feature, resilience and, in fact, sometimes benefit from device non-idealities [START_REF] Serb | Unsupervised learning in probabilistic neural networks with multi-state metal-oxide memristive synapses[END_REF][START_REF] Querlioz | Immunity to device variations in a spiking neural network with memristive nanodevices[END_REF][START_REF] Dalgaty | Hybrid neuromorphic circuits exploiting nonconventional properties of rram for massively parallel local plasticity mechanisms[END_REF]. However, these brain-inspired models cannot yet match state of the art machine learning models when applied to practical tasks. Further research has taken a bolder stance on the issue of resistive memory non-idealities and instead propose that they should be actively embraced. For example, the cycle-to-cycle conductance state variability, which has been leveraged as a source of entropy in random number generation [START_REF] Chen | Utilizing the variability of resistive random access memory to implement reconfigurable physical unclonable functions[END_REF][START_REF] Balatti | True random number generation by variability of resistive switching in oxide-based devices[END_REF], has also been exploited in stochastic artificial intelligence algorithms such as Bayesian reasoning [START_REF] Vodenicarevic | Low-energy truly random number generation with superparamagnetic tunnel junctions for unconventional computing[END_REF][START_REF] Faria | Implementing bayesian networks with embedded stochastic mram[END_REF], population coding neural networks [START_REF] Mizrahi | Neural-like computing with populations of superparamagnetic basis functions[END_REF] and in-memory optimisation [START_REF] Kerem | Stochastic p-bits for invertible logic[END_REF][START_REF] Borders | Integer factorization using stochastic magnetic tunnel junctions[END_REF]. Unfortunately however, these approaches sacrifice the key property of conductance non-volatility -the basis of resistive memory's potential for efficient in-memory computing.

In this work, we present an alternative approach which simultaneously exploits conductance variability and conductance non-volatility without requiring mitigation of other device non-idealities. From the perspective of cycle-to-cycle conductance variability, we propose that resistive memories can be viewed as physical random variables which can be exploited to implement in-memory Markov Chain Monte Carlo (MCMC) sampling algorithms [START_REF] Hastings | Monte carlo sampling methods using markov chains and their applications[END_REF]. We show how a resistive memory based Metropolis-Hastings MCMC sampling approach can be used to train, in-situ, a Bayesian machine learning model realised in an array of resistive memories. Crucially, the devices that perform the critical sampling operations are also those which store the parameters of the Bayesian model in their non-volatile conductance states. This eliminates the need to transport information between processing and memory and instead relies on the physical responses of nanoscale devices under application of voltage pulses inside of the memory circuit itself.

In order to demonstrate the practicality of RRAM-based MCMC sampling, we have implemented an experimental system consisting of a computer-in-the-loop with a fabricated array of 16, 384 resistive memory devices in a one-transistor-one-resistor (1T1R) configuration that are organised by the computer to realise a Bayesian machine learning model (Fig. 3.1). The computer is responsible for performing calculations based on the conductances read from the devices in the array and, as a function of the result, configures the programming voltage waveforms that then are applied over the devices -thereby implementing the learning algorithm in-situ. For information on the experimental setup, the device arrays and the device programming conditions used, see Appendix 6.3.

In a first experimental realisation, we train the system to solve an illustrative classification task, in a second we apply it to the detection of malignant breast tissue samples and, in a third, we address the detection of arrhythmic heartbeats. Finally, we extend the approach using a behavioural simulation calibrated on an array level variability characterisation, to the Cartpole reinforcement learning task. In each of these tasks, we benchmark the resulting performance against software-based neural network models and find that the resulting Bayesian models are not only able to outperform these software benchmarks but also that RRAM-based MCMC is able to train the full model with some orders of magnitude fewer device programming operations relative to existing resistive memory based backpropagation approaches.

Finally, through the design and simulation of a fully-integrated implementation of our proposed approach, we compare the energy required to train the models presented in here relative to conventional CMOS solutions and, once again, observe a reduction comprising several orders of magnitude.

In-memory implementation

Arrays of resistive memory devices are capable of implementing extremely efficient in-memory machine learning models [START_REF] Xia | Memristive crossbar arrays for brain-inspired computing[END_REF][START_REF] Ambrogio | Equivalent-accuracy accelerated neural-network training using analogue memory[END_REF][START_REF] Prezioso | Training and operation of an integrated neuromorphic network based on metal-oxide memristors[END_REF][START_REF] Wang | Reinforcement learning with analogue memristor arrays[END_REF][START_REF] Hu | Memristorbased analog computation and neural network classification with a dot product engine[END_REF][START_REF] Li | Efficient and self-adaptive in-situ learning in multilayer memristor neural networks[END_REF]. Considering an RRAM-based logistic regression classifier, one of the most canonical models in machine learning, the circuit of M parallel devices shown in Fig. In this fashion, we can implement RRAM-based algorithms, whereby programming operations can be determined based on the feedback of the result of the previous programming steps. In the case of the RRAM-based MCMC sampling algorithm implemented here, after reading programmed device conductance values, the computer is then responsible for calculating the dot-product between the data points and the conductance model, evaluating the likelihood of the conductance model and finally performing the acceptance ratio calculation. The ultimate aim of such a system is to experimentally verify RRAM-based algorithms, before a full system can be designed and integrated on a standalone chip.

tance updates which, after being applied over a sufficient number of iterations, guide the model down the slope of an error gradient such that it settles into a minimum (Fig. 3.3a). This results in a locally-optimal model that can then be applied to tasks through inference. However, performing this type of training inmemory is extremely challenging as resistive memory technologies feature highly non-linear and variable conductance updates that do not naturally support the high precision updates required by this class of algorithm [START_REF] Burr | Experimental demonstration and tolerancing of a large-scale neural network (165 000 synapses) using phase-change memory as the synaptic weight element[END_REF][START_REF] Garbin | HfO 2 -Based OxRAM Devices as Synapses for Convolutional Neural Networks[END_REF][START_REF] Sebastian | A collective relaxation model for resistance drift in phase change memory cells[END_REF][START_REF] Guan | On the switching parameter variation of metal-oxide rram-part i: Physical modeling and simulation methodology[END_REF][START_REF] Yu | On the switching parameter variation of metal oxide rram-part ii: Model corroboration and device design strategy[END_REF]. Furthermore, in such a deterministic modelling approach (Fig. 3.2), each parameter is described by a single value, and it is not possible to account for parameter uncertainty. Capturing the uncertainty in parameters is important when faced with the limited, possibly incomplete, data and noisy sensory readings encountered at the edge [START_REF] Ghahramani | Probabilistic machine learning and artificial intelligence[END_REF][START_REF] Radford | Bayesian learning for neural networks[END_REF]. This uncertainty in parameter estimation also allows for a description of uncertainty in the model's predictions, which would be desired in a safety-critical edge application -like an implanted medical system. To account for uncertainty, it is preferable to construct a Bayesian model. In this case, parameters are represented, not by single values, but by probability distributions. The distribution of all of the parameters of a Bayesian model, given observed data, is called the posterior distribution, or simply the posterior. As an analogue to deriving an optimal deterministic model through gradient-based updates, the objective in Bayesian machine learning is to learn an approximation of the posterior distribution. When the posterior has been approximated it can be applied to tasks through model inference. To approximate the posterior, sampling algorithms, most commonly Markov Chain Monte Carlo (MCMC) sampling [START_REF] Hastings | Monte carlo sampling methods using markov chains and their applications[END_REF], are often employed. Instead of descending an error gradient, MCMC sampling algorithms make localised random jumps on the posterior distribution and continuously store models that lie on it (Fig. 3.3b). The algorithm jumps from a current location in the model space g to a proposed location g p , according to a proposal distribution p(g p |g) which is usually a normal random variable. By comparing the proposed and current models, a decision is made on whether to accept or reject the proposed model. If accepted, the next random jump is made from this newly accepted model. MCMC sampling algorithms are configured to accept more samples from regions of high probability density on the posterior, that would correspond to a lower error in the gradient-based case of Fig. 3.3a. After a sufficiently large number of such iterations, the accepted samples can be used together as an approximation of the posterior distribution.

In this thesis we realise that, in stark contrast to the case of gradient-based learning algorithms, the properties of resistive memories are incredibly well suited to the requirements of sampling algorithms. This is because the cycle-to-cycle conductance variability, inherent to device programming, is not a nuisance to be mitigated, but instead, a computational resource that can be leveraged by viewing resistive memory devices as physical random variables. More specifically, we exploit the random variable available in a hafnium dioxide-based filamentary random access memory [START_REF] Beck | Reproducible switching effect in thin oxide films for memory applications[END_REF] (OxRAM), co-integrated at array level Each time the device is SET, a unique HCS conductance is achieved -resulting from the random redistribution of oxygen-vacancies within the oxide [START_REF] Guan | On the switching parameter variation of metal-oxide rram-part i: Physical modeling and simulation methodology[END_REF][START_REF] Yu | On the switching parameter variation of metal oxide rram-part ii: Model corroboration and device design strategy[END_REF] on consecutive programming cycles. If the HCS conductance is measured over successive cycles, a normally distributed cycle-to-cycle conductance probability density emerges (Fig. 3.4a). The SET operation is therefore analogous to drawing a random sample from a normal distribution. In addition, the median conductance of this probability distribution can be controlled by limiting the SET programming current (I SET ) via the gate-source voltage of the series transistor. The relationship between the conductance median and SET programming current follows a power law [START_REF] Ielmini | Modeling the universal set/reset characteristics of bipolar rram by field-and temperaturedriven filament growth[END_REF], and the standard deviation of the distribution also depends on the SET programming current (Fig. 3.4b). Therefore, manifested in the physical response of these nanoscale devices, we find the essential computational ingredient required to implement in-memory MCMC sampling algorithms -a physical normal random variable which can be harnessed to propose new models based on the current one. In addition to this cycle-to-cycle variability, device-to-device variability is also present in the technology. The distribution of the normal physical random variable median and standard deviation of each cell in a memory array, cycled under identical conditions, are plotted in Fig. 3.4c in order to demonstrate the device-to-device variability that exists in this technology. We propose that the N ×M resistive memory array depicted in Fig. 3.5a can be trained through MCMC sampling and then store, in the distribution of its non-volatile conductance states, the resulting posterior distribution of a Bayesian model. A single deterministic model, g n , is stored in each of the rows where its parameters are encoded by the conductance difference between positive g+ n and negative gn sets of devices -allowing for each parameter to be either positive or negative (Fig. 3.5b).

The principle of our in-situ learning approach is to generate at each row a proposed model, based on the model in the previous row, inline with the Metropolis-Hastings MCMC sampling algorithm [START_REF] Hastings | Monte carlo sampling methods using markov chains and their applications[END_REF]. Each parameter of the proposed model can be generated naturally, using the OxRAM physical random variable. This is achieved by allocating pairs of sequential banks of M devices (for an M -parameter model) corresponding to the g + and g -conductance vectors to each of the N rows in Fig. 3.5a. A dot-product is performed by reading the conductances of the devices composing g + and g -and subtracting them in the computer-in-the-loop to arrive at g and then performing the dot-product between V and g. Note that in a future version of the system, by integrating appropriate circuits within the device array the dot-product can be performed by simply applying the data points as read voltages and reading the output current consistent with the row design of Fig. 3.5b.

Resistive memory based MCMC sampling begins by performing a RESET operation on each device in the array, rendering all devices in the LCS. Using the variable n to point to the row containing the current model, the algorithm begins with n = 0. The devices in row n are SET. Because we do not have strong a priori belief on the initial model parameters, each device is SET using the lowest available V WL (in this case 1.4V ) which corresponds to the lowest SET programming current (40µA). As a result the standard deviation of the initial samples will be high (resulting from the relationship measured in Fig. 3.4b). The devices in the following row, n + 1, are then programmed with SET programming currents proportional to the conductances read from the corresponding devices in row n. This results in a proposed model,

In doing this, a new conductance value is sampled for each device in row n + 1 from a normal random variable with a median value corresponding to the same device in row n, offset by device-to-device variability (Fig. 3.4c) that is introduced when moving between successive rows.

The SET programming current in the proposal step is determined by the value of V WL used to program each device in row n + 1. To achieve this a look-up table is determined during an initial sweep whereby the entire 16k device array is RESET/SET cycled once per V WL value that will be used in the experiment. For each of these values of V WL , the median conductance read across the 16k device array is calculated and inserted in the corresponding entry in the table. Therefore, when programming a device in the row containing the proposed model is required to read the conductance of the corresponding device in row n, and use the value of V WL with the closest corresponding conductance, as the V WL used in the SET operation.

In the experiments, the look-up table extended from 1.4V to 1.8V in discrete 20mV steps, corresponding to SET programming currents in the range of 40µA-100µA and permitted median conductances in the range of 40µS-80µS to be used. Programming the devices in row n + 1 implements the model proposal step depicted in Fig. 3.6.

In Metropolis-Hastings MCMC sampling, after proposing a new model, it is required to make a decision on whether to accept and record the proposed model g p or reject and record, once again, the current model g. This decision is made based on the calculation of a quantity named the acceptance ratio a.

Because the proposal density is normally distributed (Equation 3.1), and therefore symmetrical [START_REF] Rogers | A first course in machine learning[END_REF], it can be written as:

a = p(g p ) p(g) p(t|g p , V) p(t|g, V) . ( 3
p (t|g, V) = A a=0 f (V a,t=1 • g) × B b=0 (1 -f (V b,t=0 • g)) . (3.3)
The function f (V • g) depends on the specific formulation of the model. The prior of a model is given by:

p (g) = 1 σ √ 2π exp - (g -µ) 2 2σ 2 , ( 3.4)
where the constant σ, corresponds to the prior belief that the posterior distribution is a multi-dimensional normal distribution with a standard deviation of σ in each dimension. In all examples in this thesis, the value of µ was set to zero. These quantities are calculated on the computer-in-the-loop in logarithmic scale, constituting a summation over D points. Because Markov Chain Monte Carlo sampling methods work with small datasets, where overfitting can be avoided through the incorporation of uncertainty into parameter estimations [START_REF] Radford | Bayesian learning for neural networks[END_REF], it would be possible to sub-sample the original D data points if the dataset was unreasonably large. In order to decide if the proposed model should be accepted or rejected, a is compared to a uniform random number between 0 and 1, u, generated on the computer using the C++ standard random math package. If a is less than u then g p is rejected. This is achieved by programming the devices in row n + 1 back into the LCS and incrementing the counter at row n, C n , by one. The counter is a variable in the C++ on the computer although, as is the case for all functionality of the computer-inthe-loop, could be integrated as a digital circuit on a future implementation of the system. The devices in row n + 1 are then SET once more under the same programming conditions -generating a new proposed model g p at row n + 1. This process repeats until a is found to be greater than u. When this is the case, g p is accepted whereby the counter at row n Performing inference consists of computing the dot-product between a new, previously unseen data point (V new ), and the model recorded in each array row. The response of each row is then multiplied by the value in each row counter. The summed response of each row in the array is then divided by the sum of all row counter values. This is also referred to as taking the expectation, and results in a scalar value which has been inferred from the N weighted samples from the posterior distribution approximation:

P (T new = 1|V, t) = 1 T ot N -1 n=β C n f (V new • g n ) . (3.5)
The summation considers only rows of an index greater than β which determines the number of rows discarded to account for the burn-in period. The variable T ot is the sum of all row counter values recorded after the burn-in period.

Application to supervised learning

In this section it will demonstrated experimentally how this approach can be used to address three supervised learning tasks.

Illustrative 2D separation

First, as an illustrative example, we train a Bayesian logistic regression model, realised in a 2048 × 2 array, to separate two classes of artificially generated data -the red circles from blue squares in Fig. 3.8b To configure the memory array as a Bayesian logistic regression the logistic function is applied on the current flowing out of the row:

f (V • g) = 1 1 + e -S(V•g) , (3.6)
where S is a scaling parameter. This logistic function limits the response of the row dot-product into a probability between 0 and 1. When configured as such, the Bernoulli likelihood of a conductance model is calculated as:

p (t|g, V) = A a=0 1 1 + e -S(V a,t=1 •g) × B b=0 1 - 1 1 + e -S(V b,t=0 •g) . (3.7)
After training, inference can performed on a new data-point V new whereby it is assigned a probability of belonging to the class t = 1 by computing:

P (T new = 1|V, t) = 1 T ot N -1 n=β C n 1 + e -S(Vnew•gn) . (3.8)
After the algorithm terminates, the non-volatile conductance states of the devices in the array give rise to the multi-modal posterior approximation plotted in Fig. 3.8a. Two distinct peaks emerge, denoting regions of high probability density where many of the accepted models are tightly packed. A randomly selected subset of these accepted models are plotted as hyper-planes in the space of the data in Fig. 3.8b, each defining a unique linear boundary separating the two clouds of data points belonging to each class.

Through combination of all the accepted models, therein using the posterior approximation which now exists in the array, a probabilistic boundary between the two classes emerges in Fig.

Malignant tissue recognition

For the second supervised learning task, the Wisconsin breast cancer dataset was used which consists of 569 data points with class labels malignant (t = 1) or benign (t = 0) [START_REF] Wolberg | Multisurface method of pattern separation for medical diagnosis applied to breast cytology[END_REF]. The dataset was shuffled into a training split of 369 points and test split of 200 points that were used in each of the 100 train/test iterations.

The Chi2 feature selection algorithm [START_REF] Liu | Chi2: feature selection and discretization of numeric attributes[END_REF] was used to select sixteen features and these features were After this 'burn-in' period, the algorithm tends to accept models into array rows which have a higher probability density, corresponding to models with higher classification accuracy. Strikingly, the accuracy does not saturate but rather increases sharply during burn-in and then proceeds to oscillate between high and medium accuracy conductance models. This is an important property of MCMC sampling algorithms whereby sub-optimal models are also accepted, although less frequently and with a smaller row counter value, ensuring that the true form of probability density of the posterior is uncovered. After the algorithm terminates, the accuracy achieved on the testing set was 97%, indicated by the horizontal dashed black line in Fig. 3.9b. Notably, the combined accuracy of all of the models in the posterior approximation is greater than the accuracy of any of the single accepted deterministic models alone.

In order to gather statistics on variability between training iterations, the training process was repeated over 100 further experiments and the resulting accuracy distribution is reported in Fig. 3.9c -achieving a median accuracy of 96.3% and, on some iterations, classifying over 98% of test points correctly. It was determined that this median accuracy could be sustained for array sizes down to 96 × 16 (Fig. We next compare the total number of programming operations required to train a full model in our experiment relative to the estimated number required to train RRAM-based implementations of the bench-marks inline with two state of the art RRAM-based backpropagation approaches [START_REF] Ambrogio | Equivalent-accuracy accelerated neural-network training using analogue memory[END_REF][START_REF] Yao | Fully hardware-implemented memristor convolutional neural network[END_REF] -labelled as 'Ambrogio' and 'Yao' in Fig. 3.11a. This is a useful metric in understanding the programming efficiency of the approach. Consistent with the observation that, in our experiments, each device was required on average to be RESET/SET cycled only 7.5 times (Fig. 3.10a); we found that RRAM-based MCMC required between two and four orders of magnitude fewer total programming operations than the neural network benchmarks and between one and two orders of magnitude fewer than the perceptron benchmarks (Fig. Alongside conductance variability, another major drawback of resistive memory technologies is the rapid degradation of their conductive states when subjected to repeated programming operations -referred to as endurance cycling. In order to asses the impact of endurance cycling on RRAM-based MCMC sampling, we RESET/SET cycled an RRAM array 10, 000, 100, 000 and 1, 000, 000 times and, after each decade of cycling, apply RRAM-based MCMC sampling to the same task. After only 100, 000 RESET/SET cycles the array is already no longer suitable for use in a standard memory application due to the overlap of the LCS and HCS cumulative probability distributions plotted in Fig. 3.12a. This situation is further exacerbated by the emergence of permanent write failures over the next two decades of cycling [START_REF] Grossi | Resistive ram endurance: Array-level characterization and correction techniques targeting deep learning applications[END_REF] (green curves in Fig. 3.12b). In spite of this however, the accuracy of the models trained after each decade, plotted as dark blue points in Fig. 3.12b, remain comfortably within the bounds observed during the 100 training iterations using the fresh memory array (Fig. 3.9c). A further light blue point is plotted [START_REF] Ambrogio | Equivalent-accuracy accelerated neural-network training using analogue memory[END_REF][START_REF] Yao | Fully hardware-implemented memristor convolutional neural network[END_REF] applied to both the hidden-layer neural network and perceptron benchmark models. Green (bottom) bars show the total number of operations applied to the resistive memory devices and blue bars, in the case of Ambrogio '18 which uses the change in the voltage on a capacitor for the majority of parameter updates [START_REF] Ambrogio | Equivalent-accuracy accelerated neural-network training using analogue memory[END_REF], show the number programming operations applied to the capacitors. Each bar is based on the application of the approaches in [START_REF] Ambrogio | Equivalent-accuracy accelerated neural-network training using analogue memory[END_REF] and [START_REF] Yao | Fully hardware-implemented memristor convolutional neural network[END_REF] to train RRAM-based implementations of the benchmark perceptron and neural network models used here.

whereby the error probabilities measured on the same technology for 10, 000, 000 cycles are artificially imposed during the experiment by masking read conductances with values indicative of the two error types with a given probability. Here, even when faced with 4% of devices stuck in the HCS and 1.5% of devices stuck in LCS, the accuracy of the resulting model did not degenerate below that of the fresh array.

Heart arrhythmia detection

Finally, to address a task more representative of learning at the edge, we apply RRAM-based MCMC to train a multi-layer Bayesian neural network experimentally to detect heart arrhythmias from electrocardiogram recordings [START_REF] George | The impact of the mit-bih arrhythmia database[END_REF] (see Appendix 6.6). Bayesian neural networks are the probabilistic counterparts of deterministic neural networks whereby probability distributions are used to represent the synaptic connections between neurons, allowing the model to incorporate uncertainty into these synaptic parameters and therefore be robust to over-fitting [START_REF] Radford | Bayesian learning for neural networks[END_REF]. Each neuron in the network, like the in the previous two demon- Calibrated on an array level variability characterisation, a behavioural simulator was developed that was determined to correspond well with the experimental results (see Appendix 6.4). It was used to investigate the impact of cycle-to-cycle and device-to-device variability on RRAM-based MCMC. The algorithm was seen to be robust to considerable deviations from the measured values of variability although, for unnaturally small cycle-to-cycle variability (Fig. 3.14a) or unnaturally large values of both cycle-to-cycle and device-to-device variability (Fig. 3.14b), the test accuracy and the total required number of RESET/SET cycles respectively were adversely affected.

Application to reinforcement learning

We now demonstrate that the approach can be extended to the reinforcement learning setting using the behavioural simulator. In contrast to supervised learning, reinforcement learning does not require a labelled dataset -a potentially appealing prospect for a practical edge learning system. Instead, a model is tasked with determining the actions of an agent in a physical or simulated environment in real-time [START_REF] Sutton | Introduction to Reinforcement Learning[END_REF] based on interaction with that environment. The agent observes, at each timestep, input information pertaining to the current state of the environment (V) and, as a function of the actions taken by the agent (a), a scalar reward (r) is received. The objective in reinforcement learning is to realise a model (often referred to as a policy) that allows an agent to take actions in an environment which maximise its expected reward. Here, we apply RRAM-based MCMC as a policy-search algorithm [START_REF] Hoffman | Trans-dimensional mcmc for bayesian policy learning[END_REF] and learn a posterior distribution in terms of reward.

Specifically, we address the Cartpole control task [START_REF] Barto | Neuronlike adaptive elements that can solve difficult learning control problems[END_REF] that is becoming an important validation task for RRAM-based reinforcement learning approaches [START_REF] Wang | Reinforcement learning with analogue memristor arrays[END_REF][START_REF] Berdan | In-memory reinforcement learning with moderately-stochastic conductance switching of ferroelectric tunnel junctions[END_REF]: an agent learns how to control a pole balanced on top of a cart by accelerating to the left or right as a function of four observed environmental variables describing the velocity and position of the cart and pole (Fig. 3.15a). To achieve this, we employ a perceptron model, effectively realised using two 512 × 4 memory arrays. The output neuron with the largest response, as a function of the input features, dictates the action taken by the agent on each timestep.

The python library gym was used to simulate the Cartpole environment. The Cartpole environment provides four features to the behavioural simulator at each timestep of the simulation. The behavioural simulator then specifies the actions to be taken by the agent in the environment at the next simulation timestep. Under application of a new observation from the environment V the response Instead of using the ratio of likelihoods of the proposed and current models as in the supervised case, the ratio of episodic rewards for a model g, episodic observations V and episodic actions a are used:

f (V • g) = S V • g, (3.9
this is simply the scalar reward value obtained after an episode acting according to a proposed model g p divided by the reward received when the current model g was accepted. This demonstrates a particular strength of applying MCMC sampling in a reinforcement learning setting, relative to supervised learning, since the calculation of the acceptance ratio can be achieved in a single calculation instead of an operation involving a sum over all data points in the log-domain. The prior calculation is the same as in Equation 3.4.

The acceptance ratio is then multiplied by a constant κ -1 which acts a hyper-parameter that determines the extent of exploration from higher to lower reward regions on the posterior. If the acceptance ratio is less than a uniform random number generated between 0 and 1 then the proposed model at the first row is rejected and the row counter, C 0 , is incremented by one. A new model is then proposed at the first row.

If this new model achieves a cumulative reward during the next training episode, (p(r|g p , V, a)), such that the acceptance ratio is now greater than a uniform random number, the two models within the first rows of both arrays are accepted and then become the current models. The reward which was obtained when these current models were accepted is recorded and thereafter used as p(r|g, V, a) in the calculation the acceptance ratio. After training has been completed upon the algorithm reaching the final array row, actions are determined during 100 testing episodes by calculating:

P (a new = 1|V, r) = 1 T ot N -1 n=β C n (V new •g n ), (3.11)
for each array at each simulation timestep. The summation considers only rows greater than index β, which determines the number of rows discarded to account for the burn-in period. The variable T ot is the sum of all row counter values recorded after the burn-in period. The array with the largest response at each timestep determines the action taken during inference in a winner-take-all fashion. It should be noted that although we have used the notation p(r|g, V, a), for means of consistency with the rest of this section, this quantity is not a probability but in fact a reward.

The training process is one again visualised by plotting the reward received by the agent when each In order to assess the impact of device-to-device variability (Fig. 3.4c) on this task, the simulation was repeated without its consideration. The resulting test reward distribution is plotted in Fig. 3.15c where it is seen to be largely equivalent to that case where device-to-device variability was considered. This result is consistent with studied impact of device-to-device variability in the supervised learning task, challenging the longstanding conception that device-to-device variability is a disadvantage in RRAM-based machine learning that requires mitigation. However, this should not necessarily come as a surprise: unlike in gradient-based optimisation, where device-to-device variations impede the proper descent down an error gradient, MCMC sampling is rooted in randomness.

System level energy estimation

To understand the overall energy cost of a system based on RRAM-based MCMC sampling, we evaluated the energy consumption of all the elements of a full system using commercial (Cadence) integrated circuit design tools. This discussion was made in the case of the reinforcement learning task which features a straightforward computation of the acceptance ratio and therefore less designer dependent choices than in the supervised case. Reinforcement learning shows particular promise for eventual edge applications since the system does not need to store a labelled training dataset. As detailed, the reinforcement learning process alternates between three types of phase: evaluating a model (Step 1), accepting or rejecting the model (Step 2), and programming the same row again if the proposal is rejected, or the next row of the array if accepted (Step 3). We evaluated the energy consumption of the operations necessary in these three phases. Our evaluation was made in the 130 nm CMOS process of the test chip featured throughout the chapter, as well as in a more modern technology to evaluate how our approach scales using the validated physical design kit of a commercial 28 nm technology that targets low-power applications. In the 28 nm evaluation, RRAM properties are supposed identical to those in the 130 nm measurements:

due to the filamentary nature of RRAM switching, the RRAM electrical properties of the HCS are relatively independent of the RRAM critical dimension [START_REF] Wong | Metal-oxide rram[END_REF]. We used the following methodology. The energy consumption of the digital circuits were obtained by writing a synthesisable register-transfer level (RTL) description of the circuit in SystemVerilog, generating Value Change Dumps (VCD) files representative of the Cartpole task using the Cadence ncsim simulator, and synthesising the systems to standard logic gates using the Cadence RTL Compiler. Energy estimates are provided by Cadence RTL Compiler using the appropriate VCD files. For analogue circuits, the circuits were designed using Cadence Virtuoso, and their power consumption estimated using the Cadence Spectre simulator. We realised specific designs for all parts of the system, except for Analog-to-Digital conversion and SRAM access, where energy numbers from a commercial vendor and from the literature [START_REF] Hong | A 65-fj/conversion-step 0.9-v 200-ks/s rail-to-rail 8-bit successive approximation adc[END_REF] were used, respectively.

The three different steps required by the system are as follows;

• 1. Evaluation of a model: In our approach, to evaluate a model, a cartpole episode is run. The system receives input from each of the four sensors (Fig. 3.15a), which are presented as analogue voltages to the columns of the two memory arrays of the system using low precision 5-bit digital-toanalogue converters. Then the decision to move the cartpole left or right is obtained by comparing the V•g values of both arrays. To calculate these two quantities and compare them, we evaluated the circuit of Fig. 3.16a, using conservative assumptions. The length of the read operation was chosen based on a worst case (smallest difference between the V•g values of both memory arrays), extracted from the Cartpole results. The simulations were performed in the 130 nm process, and we assumed that the energy consumption would not scale in 28 nm. Then, the system receives a binary reward from the environment, which is counted by our system. The energy consumption of this counter and of the state machine of the process were evaluated using a dedicated SystemVerilog description, in 130 nm and in 28 nm.

• 2. Acceptance/rejection of a model: In our proposed design, the process of accepting and rejecting the model is performed using digital electronics, using 9-bit fixed point (integer) computation.

A full SystemVerilog description was designed for accurate evaluation. The different aspects of the computation are summarised in the simplified data path presented in Fig. 3.16b. For increasing energy efficiency, the prior was not computed according to equation 3.4, but using an approximate value |g|, which was verified to provide equivalent accuracy by simulation of the cartpole task. This prior value can be obtained using a read operation of the memory array, and an 8-bit analogueto-digital conversion (ADC). We did not design an ADC, but used typical energy values from the literature on 8-bit low power ADCs [START_REF] Hong | A 65-fj/conversion-step 0.9-v 200-ks/s rail-to-rail 8-bit successive approximation adc[END_REF]. We performed all computation in the log domain, so that equation 3.10 is transformed into integer sums and subtractions, which consume very low power.

Depending on the value of the resulting log-acceptance ratio, three situations are possible. If the value is lower than a rejection threshold, the model is definitively rejected. If the value is higher than an acceptance threshold, the model is definitively accepted. If the value is between these two values, the actual value of the acceptance ratio (coded in fixed point) needs to be obtained from its log value. For this purpose, we store in a static memory (SRAM) the values of acceptance ratio corresponding to their log value, in the intermediate range. Energy consumption for the SRAM access was not simulated, but obtained using data from a commercial solution. Then, to decide if the model is accepted, an 8-bit pseudo-random number, generated using a linear-feedback shift register (LFSR), is compared to the acceptance ratio. The choice to use fixed point representation in logarithmic scale to represent probabilities, the reliance on a LFSR to generate random numbers, and the use of the SRAM to get acceptance ratio make this step very low power, although it performs relatively sophisticated operations.

• 3. Programming of the RRAM cells of the next row: For programming the next row, each device needs to be RESET and then SET with compliance current proportional to the conductance of the corresponding device of the previous row. To perform this, we evaluated the circuit of Fig. 3.16c, which can perform this operation precisely: it mirrors a multiplied version of the read current of an RRAM device while imposing a virtual ground. The operational amplifier is based on a standard two-stage operational transconductance amplifier design. The current mirror has asymmetrical transistors, so that the read current is multiplied into the appropriate compliance current. This takes advantage of the fact that the relationship between conductance median and SET programming current can be well approximated with a linear relationship in conductance range used in our experimental system. In Fig. 3.16c, the read current from a device in the left memory array is used for programming a device in the right memory array. This is in contrast with the schematics in Fig. 3.5a

where this operation is done within the same array. Because these operations are performed in the analogue domain, and in parallel, it would be required to read and write on the same column of the array presented in the main text -something that is of course impossible. In our design therefore the rows of the two memory arrays alternate between the two arrays -in other words the odd rows of the "go left" actions and the even rows of the "go right" action exist on the same array and vice-versa.

This choice has no impact on functionality, simplifies the RRAM write circuitry and, notably, allows

Step 1 (Model evaluation)

Step for analogue-domain parallelisation.

The results plotted in Figs. [START_REF] Strubell | Energy and policy considerations for deep learning in nlp[END_REF].17a and 3.17b show the programming of RRAM devices (step 3) consumes considerably more energy than steps 1 and 2. However, these steps are repeated a highly nonuniform number of times, as presented in Table 3.1. This Table also shows that when training the full model, step 1 consumes the most energy, for both the 130 nm and the 28 nm technology. This suggests that effort should be spent on optimising step 1, which may be achieved by using comparator designs optimised for low power consumption. However, the relative importance of the different steps will depend tremendously on the details of a task: e.g., if the cartpole experiments are shorter than 92 time steps (215 in the 28 nm technology), step 3 becomes the dominant source of energy consumption -at which point it becomes most important to reduce as much as possible the number of device programming operations.

This resonates well with the results seen in Figs. 3.11a and 3.11b where RRAM-based MCMC sampling was observed to orders of magnitude fewer total programming operations than RRAM-based backpropagation approaches. The total energy consumption summing all three steps is 6.9 µJ in 130 nm, and 3.6 µJ in 28 nm. These values are highly attractive.

To perform an order of magnitude comparison with a conventional CMOS implementation, we implemented the same algorithm (using the same assumption for the prior) in low-level C code, and simulated it on an Intel Xeon Gold 5220 processor (using a single core), based on a 14 nm CMOS technology that notably is more aggressively scaled than the nodes considered in our design. This implementation consumed around 600 mJ. A frequently used metric for evaluating artificial intelligence dedicated hardware is the number of (giga)teraoperations per second and per Watt ((G)TOPS/W). In our case, considering that the total energy consumption is dominated by step 1, and counting only model evaluation/inference, we arrive at estimates of 3.5 TOPS/W in 130 nm and 8.2 TOPS/W in 28 nm when our design is used to support the perceptron model used in the Cartpole task which features four inputs. This TOPS/W metric would increase proportionally for a model with more inputs (i.e. a memory array with more columns), since the read operation energy consumption is largely independent from the model size. This metric, which is very adapted for deep neural network accelerators, ignores the supplementary operations, such as Gaussian random number generation, that are required in MCMC learning algorithms. Therefore the most useful metric of efficiency, that allows it to be compared with approaches based on other fundamental operations, appears to be the total learning energy for a task.

We now discuss, in a qualitative manner, how these results would project to the supervised learning tasks. In these tasks, the acceptance process requires the value of V•g, instead of comparison between two memory arrays. This means that step 1 will require an ADC conversion. This makes this step more energy hungry, and likely to be more dominant than in the reinforcement learning case. Based on this assumption, in the case of the Wisconsin malignant tissue recognition task, we find that the training energy cost is 3.4 µJ in the 130 nm technology, and 1.3 µJ in 28 nm. In the case of the arrhythmic heartbeat detection task, we find a training energy cost of 12 µJ in the 130 nm technology, and 4.7 µJ in 28 nm. These numbers are once again highly attractive. Finally, to perform another order of magnitude comparison we estimate the energy required to train the supervised learning benchmark neural networks using the GOPS/W metrics reported by state of the art GPUs [START_REF] Sun | Summarizing cpu and gpu design trends with product data[END_REF]. It should be noted however that the GOPS/W reported for these systems are normally valid for only very large neural networks (considerably larger than those considered in our benchmarking) trained under optimal conditions. We used the Profiler module available in TensorFlow, where the benchmark models were trained, to determine the number of operations required in each backpropagation update. Multiplying this number by the GOPS/W metric of the most performant GPUs lead to a required energy of approximately 6mJ -although in reality this number would be expected to be greater. Once again, however, this estimate is considerably larger than the energy estimated to experimentally train each of the full models through RRAM-based MCMC.

The considerable differences between traditional CMOS implementations and our approach highlights the benefits of a dedicated in-memory implementation of machine learning algorithms -particularly if their fundamental operations can be realised naturally through inherent nano-device physics.

Ex-situ transfer of a Bayesian neural network

Section introduction

In recent years, neural network models [START_REF] Yann Lecun | Deep learning[END_REF] have demonstrated human-level competency in multiple tasks, such as pattern recognition [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], game playing [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF], and strategy development [START_REF] Vinyals | Grandmaster level in starcraft ii using multi-agent reinforcement learning[END_REF]. This progress has led to the promise that a new generation of intelligent computing systems could be applied to such high complexity tasks at the edge [START_REF] Shi | Edge computing: Vision and challenges[END_REF]. However, the current generation of edge computing hardware cannot support the energetic demands nor the data volume required to train and adapt such neural network models locally at the edge [START_REF] Strubell | Energy and policy considerations for deep learning in nlp[END_REF][START_REF] Li | Evaluating the energy efficiency of deep convolutional neural networks on cpus and gpus[END_REF]. One solution is ex-situ training: a software model is trained on a cloud computing platform and then subsequently transferred onto a hardware system that acts only to perform inference [START_REF] Xu | Scaling for edge inference of deep neural networks[END_REF][START_REF] Zhou | Edge intelligence: Paving the last mile of artificial intelligence with edge computing[END_REF]. The engine-room of such inference hardware is the dotproduct (multiply-and-accumulate) operation that is ubiquitous in machine learning. Non-von Neumann dot-product implementations based on non-volatile resistive memory [START_REF] Wong | Phase change memory[END_REF][START_REF] Chappert | The emergence of spin electronics in data storage[END_REF][START_REF] Liu | Resistive switching: Real-time observation on dynamic growth/dissolution of conductive filaments in oxide-electrolyte-based reram (adv. mater. 14/2012)[END_REF][START_REF] Beck | Reproducible switching effect in thin oxide films for memory applications[END_REF] (RRAM) technologies, otherwise known as memristors, are a particularly promising path towards reducing the energy required during inference.

As was the case in the in-situ learning setting, intrinsic cycle-to-cycle and device-to-device conductance variability once again constitute a considerable challenge. This random variability constrains the number of separable multi-level conductance states that can be achieved, preventing the high-precision transfer of the model parameters in a single programming step. To mitigate against this variability, iterative closed-loop programming schemes (also referred to as program-verify schemes) are often employed whereby devices are repeatedly programmed until their conductance falls within a discretised window of tolerated error. However, such approaches entail costly circuit overheads as well as energy and time during model transfer [START_REF] Mochida | A 4m synapses integrated analog reram based 66.5 tops/w neuralnetwork processor with cell current controlled writing and flexible network architecture[END_REF][START_REF] Mackin | Weight programming in dnn analog hardware accelerators in the presence of nvm variability[END_REF][START_REF] Hu | Memristorbased analog computation and neural network classification with a dot product engine[END_REF]. Other approaches propose to employ multiple 1T1R structures in parallel at each array cross-point [START_REF] Valentian | Fully integrated spiking neural network with analog neurons and rram synapses[END_REF][START_REF] Querlioz | Bioinspired programming of memory devices for implementing an inference engine[END_REF][START_REF] Boybat | Neuromorphic computing with multi-memristive synapses[END_REF] which, although allow for a higher precision in the transferred weight, entail additional costs in area and transfer energy. Other approaches propose to sidestep intrinsic randomness altogether through the quantisation of conductance-levels into one of either a lowconductance or a high-conductance state in binarised neural networks [START_REF] Bocquet | In-memory and error-immune differential rram implementation of binarized deep neural networks[END_REF]. However, such models require a significantly higher number of neuron elements and model parameters to approach the performance of conventional neural networks [START_REF] Hirtzlin | Digital biologically plausible implementation of binarized neural networks with differential hafnium oxide resistive memory arrays[END_REF]. It should be noted that, unlike a separate body of work that leverages the random switching properties of magnetic RRAM [START_REF] Joseph S Friedman | Bayesian inference with muller c-elements[END_REF][START_REF] Faria | Implementing bayesian networks with embedded stochastic mram[END_REF] or stochastic electronic circuits [START_REF] Singh Thakur | Bayesian estimation and inference using stochastic electronics[END_REF][START_REF] Mroszczyk | The accuracy and scalability of continuous-time bayesian inference in analogue cmos circuits[END_REF] to perform Bayesian inference, this chapter is concerned with performing inference of ex-situ trained Bayesian neural networks transferred onto an RRAM-based hardware.

The reality is that the programming of resistive memory is an inherently random process and, the devices therefore, are not well suited to being treated as deterministic quantities. Fortunately however, as has been seen in the previous chapter, this randomness follows stereotyped probability distributions and allows resistive memory devices to be instead yielded as physical random variables [START_REF] Dalgaty | Hybrid neuromorphic circuits exploiting nonconventional properties of rram for massively parallel local plasticity mechanisms[END_REF][START_REF] Dalgaty | In-situ learning harnessing intrinsic resistive memory variability through markov chain monte carlo sampling[END_REF]. This opens up an alternative direction of exploration whereby RRAM conductance states can be manipulated within the frameworks of probabilistic programming and Bayesian modelling [START_REF] Ghahramani | Probabilistic machine learning and artificial intelligence[END_REF]. In this section, we propose and demonstrate experimentally an approach for the ex-situ training and subsequent transfer of a Bayesian neural network on to a resistive memory based inference hardware. Because Bayesian neural networks, like resistive memory conductance states, describe model parameters as probability distributions, and not single high-precision values, it suggests a more natural pairing of algorithm and technology. In this setting, the objective is no longer to precisely transfer a single parameter from a software model to the corresponding device in a resistive memory array but to transfer a probability distribution from the software model into a distribution of device conductance states.

In this work, we first propose to use an expectation-maximisation algorithm to decompose a probability distribution into a small number of random variable components; each corresponding to the cycle-to-cycle conductance distribution of RRAM programmed under a given programming current. We then demonstrate experimentally, with an array consisting of 16, 384 fabricated hafnium dioxide 1T1R structures, that these random variable components can be used to transfer the original probability distribution into the conductance states of a column of RRAM devices. We show that this approach can be leveraged to achieve the transfer of a full Bayesian neural network in a single programming step. We also describe an RRAM-based hardware capable of storing, and performing inference with, such a Bayesian neural network. Finally, a Bayesian neural network model is trained ex-situ and transferred using the proposed technique to the experimental 16k device array. The transferred model is used for inference in a simple illustrative classification task where the decision boundaries that were learned ex-situ are seen to be well preserved in the model transferred into the inference hardware.

Expectation-maximisation based parameter decomposition

As was leveraged in the previous chapter, RRAM devices are normal random variables in their high conductance state. As a further example of this three of the random variables available from a single device under three programming conditions are plotted in Fig. 3.18a. What is evident from this plot is that, as the median of the random variable increases, its standard deviation decreases -the two quantities are intrinsically tied. This relationship between the cycle-to-cycle conductance standard deviation and median, over the full population of 16, 384 devices in the 1T1R array, is plotted in Fig. 3.18b where it is seen to be well approximated by a linear function.

Bayesian neural networks are variants of conventional neural networks whereby parameters are not single values, but probability distributions [START_REF] Radford | Bayesian learning for neural networks[END_REF]. The distribution of each parameter encapsulates the uncertainty in its estimation which allows for a model to avoid over-fitting given, for example, a small training dataset or noisy sensory observations [START_REF] Ghahramani | Probabilistic machine learning and artificial intelligence[END_REF] -this is in contrast to deterministic neural networks which often suffer from severe over-fitting under similar conditions.

Therefore, the challenge is to transfer these software-based probability distributions into a plurality of device conductance states on an RRAM-based inference hardware. The fundamental insight here is that, because RRAM conductance states are also probability distributions, they owe themselves more naturally to the transfer of ex-situ trained Bayesian neural network models than deterministic ones.

We propose that the probability distribution of each Bayesian neural network parameter can be approximated by a linear combination of weighted normal random variable components -determined using a Gaussian mixture modelling approach [START_REF] Moon | The expectation-maximization algorithm[END_REF]. In a Gaussian mixture model, each of the K Gaussian distributions, also referred to as normal random variable components, are characterised by a median, a standard deviation and a weighting factor. These three parameters per component are updated itera- tively using an algorithm called expectation-maximisation until a mixture of components is found that best 'explains' the target parameter distribution [START_REF] Moon | The expectation-maximization algorithm[END_REF]. As seen in the previous section, OxRAM devices are normal physical random variables in the high conductance state [START_REF] Dalgaty | In-situ learning harnessing intrinsic resistive memory variability through markov chain monte carlo sampling[END_REF] (Fig. 3.18a) and can therefore act as the random variable components in such a probabilistic mixture.

However, although the median of each RRAM random variable component can be freely determined by the SET programming current (Fig. 3.18a), the standard deviation is intrinsically tied to this value (Fig. 3.18b). This requires that, instead of treating the standard deviation of each component as free parameters during the expectation-maximisation algorithm, its value must be assigned based on the a known relationship with the median (here the equation in Fig. 3.18b). This may require, for example, additional circuitry on a practical chip to perform an initial calibration step owing to the die-to-die variability that exists across chips on a wafer as well as between wafers [START_REF] Grossi | Resistive ram endurance: Array-level characterization and correction techniques targeting deep learning applications[END_REF].

We apply this technique to decompose the single target probability distribution plotted in green in Fig. 3.19 into K physical random variable components. These components, determined through expectationmaximisation, can then be used to program experimentally a column of N RRAM memory cells such that the distribution of conductance states in the column approximates that of the target distribution. This result is achieved by programming subsets of devices in the column with a SET programming current such that their conductance states are sampled from the Gaussian corresponding to each component. The number of devices programmed per component is equal to the nearest integer value resulting from the multiplication of the total number of available devices by its weighting factor. For this target distribution it was found that five normal components (K = 5) were required to well approximate the target distribution. This result was obtained by performing the expectation-maximisation algorithm over a sweep of K and observing at which value of K the resulting log-likelihood of the mixture saturated -as plotted in Fig. In order to quantify the closeness of the approximation transferred to the hardware we evaluate the Kullback-Leibler (KL) divergence from the transferred to the target distributions over a range of column sizes. The resulting mean KL divergence, over ten experimental transfers, is plotted in Fig. 3.20b for an increasing number of RRAM cells per column. The KL divergence reduces rapidly as the number of devices in the RRAM column is increased, consistent with the law of large numbers [START_REF] Hsu | Complete convergence and the law of large numbers[END_REF]. For this target distribution, it was determined that five components were required to approximate the target distribution (dashed red line). (b) Kullback-Leibler divergence from the target to the transferred distributions calculated for an increasing number of RRAM memory cells per column. For each number of memory cells, the distribution was transferred ten times. The resulting variability in the KL divergence is shown using vertical green bars at each point indicating one standard deviation. An example of the transferred distributions for 32 and 4, 096 devices are plotted as an inset.

Ex-situ training of an RRAM-based Bayesian neural network

Before applying the presented technique to the transfer of a full Bayesian neural network model, we first describe how to perform the ex-situ training of an RRAM-based Bayesian neural network and how the parameters from this software model can be represented using an array of resistive memory devices [START_REF] Dalgaty | In-situ learning harnessing intrinsic resistive memory variability through markov chain monte carlo sampling[END_REF]. In the Bayesian framework, training is typically performed with Markov Chain Monte Carlo (MCMC) sampling or by variational inference algorithms [START_REF] Hinton | Keeping the neural networks simple by minimizing the description length of the weights[END_REF][START_REF] Blundell | Weight uncertainty in neural network[END_REF][START_REF] Kucukelbir | Automatic differentiation variational inference[END_REF]. We employ the No-U-Turn sampler (NUTS) MCMC algorithm [START_REF] Hoffman | The no-u-turn sampler: Adaptively setting path lengths in hamiltonian monte carlo[END_REF]. In contrast to gradient-based approaches, which result a deterministic locally-optimal model, NUTS MCMC results in a collection of sampled models, each with their own parameters (synaptic weights and biases). The distribution of each learned parameter, in other words the distribution of parameters over all of the sampled models, can then be transferred to a distribution of device conductances in a column of RRAM cells. In contrast to deterministic models, which generally require a single device per parameter, the use of a distribution comprising multiple devices per parameter allows uncertainty to be incorporated into its estimation. The ability to represent uncertainty in this manner is advantageous for Bayesian models, permitting them to account for factors such as sensory noise, small training dataset size as well as representing uncertainty in their output predictions. [START_REF] Ghahramani | Probabilistic machine learning and artificial intelligence[END_REF][START_REF] Radford | Bayesian learning for neural networks[END_REF].

One neuron in an RRAM-based Bayesian neural network can be realised as depicted in Fig. 3.21b.

The neuron receives input synapses from M (here three) neurons in the previous network layer (Fig. 3.21a)

-each connecting to one of its three columns. The distribution of the three input synaptic parameters are each stored in a column of size N and therefore necessitates an N × M array of 1T1R structures per neuron.

By applying a voltage vector V across these M columns, corresponding to the activations of the neurons in the previous network layer or the input data for neurons in the first layer, a current equal to V • g n flows out of each array row and into a neuron circuit. Since the conductance values of g n are on the order of microsiemens, the neuron circuit must first multiply this current value by a scaling factor S and then apply an activation function h() to the scaled quantity, resulting in a neuron output voltage

z n = h(S(V • g n )).
This voltage can then in turn be applied to a column of each of the neuron arrays in the next layer. The distribution of these N neuron activation voltages, z, constitutes the output distribution of the neuron. We Each row of the array uses devices that code for positive (g + ) and negative (g -) values that enables each parameter to be positive or negative. The inputs to the columns are the output voltages generated by the M neurons in the previous network layer. As a result of these input voltages, two currents will flow out of each row and into a neuron circuit which subtracts them and then evaluates an activation function. This activation produces an output voltage as a function of this current that can then be applied to the column of neuron arrays in a subsequent layer. The distribution of the N output voltages (blue probability distribution) is the output distribution of the neuron.

In practice, as each parameter distribution can assume positive and negative values, each model parameter should be described by the difference between positive and negative distributions: p(g) = p(g + ) -p(g -) (Fig. 3.21b). Therefore, during MCMC sampling, the parameters which are sampled are p(g + ) and p(g -) and not p(g) directly. In addition, each neuron of the Bayesian neural network requires a bias distribution p(g b ), that can be realised with an extra column of devices, identical to the others, to which a constant voltage V b is applied.

One further technological constraint must be taken into account. Each RRAM device has a limited conductance range; in the technology applied here, extending approximately from 20µS to 120µS (Fig. 3.18b).

As a result, the sampled distributions of each parameter, p(g + ) and p(g -), of the Bayesian neural network must be bounded within these limits during the ex-situ training. Fortunately, in the Bayesian framework, such a bounding can be achieved naturally by placing an appropriate prior distribution over each parameter. To account for this we therefore place a normal prior over each parameter, with a median of 80µS, and a standard deviation of 20µS, such that the sampled distributions exist within the limited conductance range.

Transfer to resistive memory based inference hardware

We now combine the ideas of the two previous sections and present an approach to achieve the transfer of an ex-situ trained Bayesian neural network onto the RRAM-based hardware depicted in Fig. • Quantise -The expectation-maximisation is applied independently to each of the Q columns of the S × Q samples matrix. This allows K component medians to be extracted per synaptic distribution -where the number of K can be different for each parameter. For each matrix column the values in the matrix cells are then quantised by setting them to a new value equal to the nearest component median value.

• Sub-sample -This quantised S × Q samples matrix will likely have many more rows than the number of rows available on the RRAM-based inference hardware (i.e. S N). This is because, to allow NUTS MCMC to adequately explore the posterior distribution of the model, it is useful to collect a large number of samples. Therefore, to reduce it to the same dimensions as the RRAM-based hardware, the matrix must be sub-sampled to an N × Q matrix -ideally by uniform-random removal of rows.

• Slice -Because the hardware is not composed of one large N × Q array of RRAM, but lots of smaller N × M i arrays (one array per neuron) the N × Q matrix must be sliced, column-wise. This groups together all of the pre-synaptic distributions for each neuron into a single array. In this fashion, each cell of each processed N × M i software matrix has a one-to-one correspondence with a device on an RRAM array.

After these processing steps, it is now possible to make a single step transfer by programming each physical device on the hardware with a SET programming current which samples a conductance value from a Gaussian distribution with a median equal to the value in the corresponding software matrix cell. It is important to preserve the order of parameters by transferring in a row-wise fashion: since the sampled parameters obtained through NUTS MCMC will often co-vary with one another (i.e. certain parameters will be correlated per sample). For instance, if we were to mix-up parameters between two of the recorded models from the posterior approximation, the resulting model would not necessarily lie on the posterior distribution of the Bayesian model. Note that the weighting factor resulting from the expectation-maximisation algorithm is not used to determine the number of devices programmed in line with each component as in Fig. 3.19, but is instead incorporated implicitly by the samples that have been accepted by NUTS MCMC.

After the model has been transferred, the hardware can then perform inference on previously unseen data points whose features are presented as voltages to the columns of the neuron arrays in the first hidden-layer. These voltages drive the forward propagation of neuron voltage activation distributions through the subsequent network layers, resulting finally in an activation distribution per output neuron.

These output distributions can then be used to make a prediction regarding as to what output neuron class the input data point belongs. Additionally, the standard deviation of each prediction distribution can be calculated and used to quantify the uncertainty in the prediction of each output neuron.

Since the pre-synaptic distributions of each of the neurons in a Bayesian neural network are stored in multiple rows (i.e. multiple samples), but the physical connections between each neuron consist only of single metal wires, inference with a Bayesian neural network must be performed one row (sample) at a time. At the output layer then, a separate memory structure is required to temporarily store each of the output neuron activations that result from each of the transferred samples -an SRAM memory for instance. In this fashion, after all of the rows have been read in an inference, the prediction distribution of each output neuron is readily available. To achieve this we propose, in Fig. 3.22, that each neuron array contain only a single neuron circuit that is multiplexed between each of the N rows sequentially.

By applying voltage pulses to the gates of the devices in only one row, while grounding the others, this multiplexing can be achieved cheaply and without a dedicated multiplexing circuit. Additionally, the use of a shared neuron circuit also reduces the required circuit overhead to implement the activation function or to perform any required analogue-to-digital conversions by a factor of N . For example, by applying the red pulses in Fig. 3.22 to row N = 0 of all of the neuron arrays simultaneously at time t 0 , each neuron in the output layer will produce a voltage activation, z 0 . In other words, these output activations result from (from the red coloured neurons in the first layer), the hidden layer neurons will produce activation voltages that are in turn applied over the columns of the output layer neurons causing the output layer neurons to produce activation voltages. This forward propagation of voltage continues for an arbitrary number of network layers until reaching the output layer. By sequentially applying gate voltage pulses to each row of all the arrays -the red pulses at t 0 , the green pulses at t 1 and finally the blue pulses at t N -1 -output neuron one will sequentially produce voltage activations z 0 , z 1 and z N -1 . The distribution of all activations, p(z), gives rise to an output distribution of neuron one.

the forward propagation of the input through the devices in row N = 0 of each array only. Thereafter, applying the green pulses at t 1 the outputs z 1 will result from the forward propagation through the devices at rows N = 1 and so on. By proceeding in this fashion up until row z N -1 an output distribution p(z) will be available for each output neuron at t N -1 . The mean value and standard deviation of this distribution can be used to respectively make a prediction and quantify prediction uncertainty.

To demonstrate this technique, we perform the ex-situ training of the Bayesian neural network depicted in Fig. 3.23. We apply it to an illustrative example in the generative moons classification task [START_REF] Wiecki | Bayesian deep learning[END_REF]: Upon performing inference with the hybrid hardware/software model, the decision boundaries for each of the two output neurons for the model transferred to the 1, 024, 128 and 16 row arrays, shown in Fig. 3.23, arise. The output neurons appear, in all situations, capable of discerning the underlying structural separation between the two types of data point that was learned in the software model. The probability contours of the two output neurons are largely similar for the case of 1, 024 and 128 rows, while those for 16 rows appear more erratic. Despite this appearance however, the boundaries drawn at the interface of the two moons with N=16 rows still capture the fundamental curvature of their division. Based on the read currents of the programmed devices, the energy required to read all of the device conductances during inference was 110nJ, 13.7nJ and 1.72pJ for the models transferred to the 1, 024, 128 and 16 row arrays respectively.

each
However, that is important to note that, based on the design of the in-situ RRAM-based MCMC system, the energy required by read circuitry, analogue-to-digital and digital-to-analogue conversions as well as circuits for implementing the neuron activation functions have not been considered and would lead to a considerable increase in these values depending on design choices. The prediction uncertainty of each of these transferred Bayesian models is plotted in Fig. 3.24. This uncertainty, captured in the distribution of each synaptic parameter, naturally propagates through a Bayesian neural network to the output layer where, as might be expected, it is seen to be greatest at the interface between the red and blue points. While the prediction uncertainty contours are largely similar for N = 1024

and N = 128, they are once again degraded for N = 16. In safety-critical edge inference applications the ability of a Bayesian neural network to quantity uncertainty, with respect to deterministic models, is potentially invaluable and, perhaps, indispensable from an ethical perspective [START_REF] David D Fan | Bayesian learning-based adaptive control for safety critical systems[END_REF]. For example, in a medical system, such as the implantable cardioverter-defibrillator [START_REF] Barry J Maron | Efficacy of implantable cardioverter-defibrillators for the prevention of sudden death in patients with hypertrophic cardiomyopathy[END_REF] like that considered in the in-situ setting, these prediction uncertainties can be leveraged to avoid the erroneous application of an electric shock to the heart which can, in some instances, prove fatal [START_REF] James P Daubert | Inappropriate implantable cardioverter-defibrillator shocks in madit ii: frequency, mechanisms, predictors, and survival impact[END_REF]. If the system were presented with a data-point close to a noisy decision boundary (as in Fig. 3.24) or with a data-point from a location in the feature space that the model had not observed during training, perhaps due to a damaged or drifting sensor, the prediction uncertainty of the model will be large. By placing a threshold on a tolerated level of prediction uncertainty, above which the system should not take action, the model is able to express that it dos not known with certainty what action should be taken and erroneous, potentially dangerous, interventions can be avoided.

Allowing a model to say 'I don't know'

We now present a final example that demonstrates how ex-situ Bayesian model transfer can be performed in the closed-loop (program and verify) fashion [START_REF] Hu | Memristorbased analog computation and neural network classification with a dot product engine[END_REF][START_REF] Mochida | A 4m synapses integrated analog reram based 66.5 tops/w neuralnetwork processor with cell current controlled writing and flexible network architecture[END_REF]. The ex-situ trained model is applied to the heart arrhythmia datatset [START_REF] George | The impact of the mit-bih arrhythmia database[END_REF] which, crucially, also allows for a practical demonstration of how Bayesian modelling uncertainty can be applied in a safety-critical application. Specifically, a technique allowing for a model to say 'I don't know', such that it doesn't not carry out a potentially harmful action, is presented. The model is trained to classifying four different types of potentially heart arrhythmia as well as healthy heartbeats. Considering a futuristic use case whereby an implantable cardioverter defibrillator is equipped with an ex-situ trained RRAM-based Bayesian neural network model, the model would be responsible for application of an electric shock to the heart upon detection of certain types of arrhythmia.

As described however, the application of a shock to a normally functioning heart can prove fatal [START_REF] James P Daubert | Inappropriate implantable cardioverter-defibrillator shocks in madit ii: frequency, mechanisms, predictors, and survival impact[END_REF].

The model therefore must apply a shock only when sufficiently certain in order to avoid performing potentially dangerous actions, while also taking actions frequently enough to intervene in the event of sustained arrhythmic beats.

A Bayesian neural network model, with two hidden layers of eight neurons each, is trained ex-situ by NUTS MCMC [START_REF] Hoffman | The no-u-turn sampler: Adaptively setting path lengths in hamiltonian monte carlo[END_REF] using sixty four input features resulting from the fast Fourier transform of each heartbeat (see Appendix 6.6). The training set is composed of two hundred examples of each of the five classes of beat. Once trained it is then transferred in a closed-loop fashion to the hybrid hardwaresoftware experimental 16k device array. Here, instead of exploiting the devices as random variables as presented in the rest of this section, each device is iteratively programmed until its conductance falls within a pre-defined conductance error range -as depicted in Fig. 3.25a. Therefore the high precision weights of the model in the cloud are discretised into a much smaller number of levels. In Fig. 3.25b it is shown how each resistive memory in the 16k device array can be programmed into eight separable conductance levels while being in the HCS, in addition to one further level corresponding to the LCS, using a basic RESET/SET closed-loop programming technique. In this scheme; a specific SET programming current is defined for each of the eight HCS levels and a device is iteratively programmed until its conductance falls within a pre-determined error margin (Fig. 3.25a).

Using the software model of the inference architecture presented in Fig. 3.22, each of the simulated devices is assigned a conductance value from the desired level in the device data presented in Fig. 3.25b.

Considering that subtraction of two devices encodes the positive or negative weight of each parameter, this allows for seventeen distinct conductance combinations. The combination of conductances levels per pair of devices is chosen such that, on average, the resulting differential conductance should be as close as possible to the original high-precision value of the software model.

Chapter discussion

In this chapter it has been demonstrated that, contrary to the case of gradient-based algorithms and deterministic modelling techniques, Bayesian modelling and resistive memory devices result in a harmonious pairing of algorithm and hardware.

In the first section the potential of simultaneously harnessing resistive memories as physical random variables and as high efficiency dot-product engines in the realisation of in-situ Markov Chain Monte Carlo sampling algorithms was demonstrated. This in-situ learning method, which actively exploits cycle-tocycle conductance variability, was shown to also be resilient to device-to-device variability and extensive device aging -orders of magnitude beyond the point where devices could be used as traditional memories.

The appeal of RRAM-based MCMC was seen in two key sets of results. First, the dramatic reduction in the total number of programming operations required to train a full model, with respect to RRAMbased backpropagation approaches, underlined the remarkable compatibility of resistive memory and Markov Chain Monte Carlo sampling. Additionally, the orders of magnitude reduction in the energy of a full RRAM-based MCMC system relative to conventional CMOS solutions, compounded the potential of the approach based on the application of brief voltage pulses, in-memory, to nanoscale devices. In fact, the energy totals which were on the order of micro Joules, correspond to the energy estimated to be consumed by a biological neuron spiking tens of times or less [START_REF] Wang | Neural energy supply-consumption properties based on hodgkin-huxley model[END_REF]. This is particularly eye-catching given the current intense focus within several research communities of developing spike-based brain-inspired learning algorithms in order to reduce the energy requirements of machine learning systems. Secondly, when faced with data representative of the edge, namely the noisy and multi-modal electrocardiograms in the arrhythmia detection task, the robustness of Bayesian machine learning to over-fitting by incorporating uncertainty into learned parameters [START_REF] Radford | Bayesian learning for neural networks[END_REF][START_REF] Ghahramani | Probabilistic machine learning and artificial intelligence[END_REF], was seen to be crucial in outperforming the softwarebased deterministic benchmark neural networks trained through backpropagation. This second result puts in evidence that, beyond simply being compatible with the fundamental properties of resistive memory devices, RRAM-based Bayesian models exhibit a particular suitability for the edge setting.

The Metropolis-Hastings MCMC sampling algorithm presented in the first section is known to lose efficiency when the dimensionality of the model it is applied to train becomes very high. While this could pose a potential issue going forward as we look to scale this approach to higher complexity tasks, a potential solution was stumbled upon in section 2.2. One of the key results from the neural network model of the cricket cercal system was that such bio-inspired neural network models could allow for a reduction in the number of model parameters by some orders of magnitude. The exploration and discovery of bio-inspired architectures, to which RRAM-based MCMC can be applied to whilst remaining at its most effective, may therefore be key to scaling the approach to higher complexity tasks. Furthermore, Hamiltonian Monte

Carlo algorithms [START_REF] Girolami | Riemann manifold langevin and hamiltonian monte carlo methods[END_REF][START_REF] Matthew | The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo[END_REF], such as the No-U-Turn sampler used in section 3.3, are able better address higher dimensional problems and therefore such MCMC sampling algorithms can be incorporated into evolution of the presented Metropolis-Hastings RRAM-based MCMC system presented here. To apply MCMC sampling to larger datasets than those considered in this section, where the Metropolis-Hastings algorithms can also be limited, mini-batch MCMC algorithms should also be investigated [START_REF] Maclaurin | Firefly monte carlo: Exact mcmc with subsets of data[END_REF][START_REF] Korattikara | Austerity in mcmc land: Cutting the metropolishastings budget[END_REF] that allow for samples to be accepted based on subsets of the data.

Ultimately section 3. extremely resource constrained, applications. For example, intelligent and adaptive implantable medical devices which is a currently revolutionary application domain within machine learning that is currently of out reach through existing commercial approaches [START_REF]Edge AI Chipsets: Technology Outlook and Use Cases[END_REF].

The second section of the chapter approaches the learning problem from the ex-situ perspective.

Because Bayesian models require many samples per parameter, existing ex-situ transfer approaches could entail a prohibitive energy cost and latency to transfer a full Bayesian neural network. Therefore the expectation-maximisation based transfer approach was introduced which permitted an ex-situ trained Bayesian model to be transferred in a single step -i.e, each device was required to be programmed only once. The importance of describing uncertainty in the parameters of the Bayesian neural network was also discussed in terms of how the resulting output prediction uncertainty could be leveraged in a safetycritical application. We studied the use case of the application of an electric shock upon the detection of dangerous heart arrhythmias. It was demonstrated that by putting a threshold on the output uncertainty (here called uncertainty thresholding), it was possible to make a Bayesian neural network model say 'I don't know' and take zero potentially dangerous actions. In a variety of emerging, or yet to be discovered, edge computing applications, where systems will interact with humans and whose actions may compromise their safety, it is difficult to imagine using a modelling approach that does not quantify prediction uncertainty. Alongside Bayesian models, other techniques based on deterministic neural networks such as Monte Carlo dropout [START_REF] Gal | Dropout as a bayesian approximation: Representing model uncertainty in deep learning[END_REF] and deep ensembles [START_REF] Balaji Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF] have also been recently found to offer good uncertainty estimates. It will therefore be important to compare the uncertainties available through these approaches with those naturally incorporated into Bayesian models.

Going forward from this initial proposal and experimental demonstration, future work should aim to understand how the expectation-maximisation transfer technique can scale to larger network models and to higher-complexity datasets as well as looking for adaptations to the algorithm that can more naturally incorporate covariance in the posterior -ideally obviating the 'row-wise' transfer protocol used here. In doing so it will be also be useful to explore the application of training algorithms like variational inference [START_REF] Hinton | Keeping the neural networks simple by minimizing the description length of the weights[END_REF][START_REF] Blundell | Weight uncertainty in neural network[END_REF][START_REF] Kucukelbir | Automatic differentiation variational inference[END_REF] which, through use of 'mean-field' variational approximations, can be used to eliminate parameter covariance altogether and allows each synapse to be treated independently. Alternatively, it has also been found that as Bayesian neural networks become deeper, parameter covariance largely disappears altogether [START_REF] Farquhar | Liberty or depth: Deep bayesian neural nets do not need complex weight posterior approximations[END_REF]. A solution therefore, could simply be to perform ex-situ training on models with a greater number of layers.

Finally, it will also be instructive to perform a quantitative comparison between ex-situ trained RRAMbased Bayesian and deterministic neural network models to understand the advantages and trade-offs between the two approaches in terms of inference accuracy, the energy and latency incurred in model transfer and inference as well as memory requirements.

Chapter 4

A non-von Neumann neuromorphic computing fabric

Chapter introduction

The first two chapters have addressed the potential of bio-inspired neural network architectures and a method for training such models, in the ex-situ or in-situ setting, if its parameters were implemented using resistive memory devices. This final chapter completes the picture by proposing how resistive memory based bio-inspired models can be realised in a full non-von Neumann computing system. The two sections of this chapter show respectively how RRAM can be incorporated into neuromorphic circuit models and into a neuromorphic architecture that can interconnects these models in a configurable architecture.

Crucially, in both cases, the non-volatility of RRAM is used a means of distributing memory continuously through the computing fabric and, as observed in biological nervous systems, co-localising the configurable memory elements with the computational units in a fashion that makes them almost indistinguishable.

Hybrid neuromorphic circuits

Proteins exist within the membranes of biological neurons (such as that pictured in Fig. 4.1(a)) which transport ionic charge into and out of the cell [START_REF] Declan A Doyle | The structure of the potassium channel: molecular basis of k+ conduction and selectivity[END_REF]. Under excitatory stimulation positive charge accumulates inside whilst leaking out at a constant rate. In the case of spiking neurons, if the rate of charge accumulation sufficiently exceeds the leak, a chain reaction triggers the generation of an action potential (spike) and a wave of ionic charge propagates down the neuron's axon. As the axon branches, this spike proceeds and divides along parallel bifurcating channels until it reaches synapses composed of pre-synaptic processes and post-synaptic sites belonging to the afferent neuron. At these synapses, electrical excitation elicits release of neurotransmitter that diffuses across a nanoscale cleft from the preto the post-synaptic site and subsequently modulates ion uptake at the destination cell. Properties of neurons (like those depicted in Fig. 4.1(b)), such as the time constant of the input charge integration, the synaptic weight that governs how much charge is injected upon the arrival of neurotransmitter and the length of the refractory period during which the neuron cannot spike are determined by the conductance of these membrane proteins. Neurons continuously adapt these conductances inline with different neural plasticity mechanisms for the purposes of maintaining homeostasis [START_REF] Stemmler | How voltage-dependent conductances can adapt to maximize the information encoded by neuronal firing rate[END_REF][START_REF] Gina | Homeostatic plasticity in the developing nervous system[END_REF] and to form memories and learn [START_REF] Levy | Temporal contiguity requirements for long-term associative potentiation/depression in the hippocampus[END_REF][START_REF] Dan | Hebbian depression of isolated neuromuscular synapses in vitro[END_REF].

Neuromorphic synapse and neuron models, which aim to mimic the dynamic behaviours of the biological cells using electronic circuit components [START_REF] Mead | Neuromorphic electronic systems[END_REF][START_REF] Mahowald | A silicon neuron[END_REF], use the drain-source conductance of a transistor under a 'current-mode' gate bias, mimicking neuronal ion channels, as a means of implementing these conductance parameters [START_REF] Farquhar | A field programmable neural array[END_REF][START_REF] Qiao | A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128k synapses[END_REF][START_REF] Varkey Benjamin | Neurogrid: A mixed-analog-digital multichip system for large-scale neural simulations[END_REF][START_REF] Indiveri | Neuromorphic silicon neuron circuits[END_REF][START_REF] Chicca | Neuromorphic electronic circuits for building autonomous cognitive systems[END_REF]. Essentially, these biased transistors act as the conductances (resistances) used in Hodgkin-Huxley style neuron models [START_REF] Hodgkin | A quantitative description of membrane current and its application to conduction and excitation in nerve[END_REF], which regulate at what rate charge flows onto and off of capacitors.

In the class of non-von Neumann computer chips referred to as neuromorphic processors, these biases are programmed and generated in a centralised unit on the chip [START_REF] Varkey Benjamin | Neurogrid: A mixed-analog-digital multichip system for large-scale neural simulations[END_REF][START_REF] Qiao | A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128k synapses[END_REF][START_REF] Moradi | A scalable multicore architecture with heterogeneous memory structures for dynamic neuromorphic asynchronous processors (dynaps)[END_REF][START_REF] Singh Thakur | Large-scale neuromorphic spiking array processors: A quest to mimic the brain[END_REF]. These bias generators generally store parameter values using a volatile SRAM memory. Such an approach, however, has many drawbacks. To limit the size, and therefore the static power consumption of the volatile memory of the current source generator, as well as the metallic area required to physically deliver biases to individual neurons; groups of thousands of neurons and synapses in neuromorphic processors are generally obliged to share the same set of parameters. In addition, this volatile memory is typically programmed once, via an external computer, either when the neuromorphic processor is power cycled or re-purposed for a new application. However, for networks more closely emulating the computational principles of biological systems where neuron and synapse parameters are updated continuously, such as the Drosophila elementary motion detection model presented in section 2.3, the von Neumann bottleneck that persists between processing and memory in neuromorphic processors becomes a limiting factor. At this point, the separation of memory and the computational elements becomes critically expensive in the implementation of parameter adaptation algorithms due to the large volume of data which is required to be ferried between the neurons and this central memory centre. It should be noted that in some neuromorphic processors that support 'on-line' plasticity this is not necessarily the case for the parameters that encode the synaptic weight. In these cases, whereby synaptic weights are modified inline with spike-timing dependent rules, the synaptic weight parameters are stored and updated locally at each synaptic array using an additional volatile memory. Solutions exploiting certain properties of resistive memory devices, such as the volatile resistance state of metal-insulator-transition Mott devices [START_REF] Pickett | A scalable neuristor built with Mott memristors[END_REF] and the gradual crystallisation of phase change memories [START_REF] Tuma | Stochastic phase-change neurons[END_REF] also been demonstrated. These solutions use the physics of the device to store the information locally, without a static power draw, and therefore allow parameter memory to be distributed throughout a computing substrate instead of centralised in a bias generator. However, the properties of these RRAM-based neurons based on single devices are fixed and cannot be configured.

CMOS-RRAM analogue circuit models

In this section, hybrid CMOS-RRAM electronic circuits models which model biological neurons and synapses like those used in section 2.3 are proposed, designed and fabricated. These analogue circuits incorporate resistive memory elements in order to store parameters that define the behaviour of these models in a non-volatile fashion whilst also playing an role in the physical operation of the circuit. Crucially, this allows configurable memory to be distributed in space throughout a neuromorphic computing system in a truly non-von Neumann fashion -corresponding more closely with the underlying organisational principles observed in animal nervous systems.

The basis of these hybrid neuromorphic circuits is the 1T1R structure [START_REF] Vianello | Resistive memories for ultra-low-power embedded computing design[END_REF][START_REF] Grossi | Fundamental variability limits of filamentbased rram[END_REF] depicted in Fig. 4.2a.

A resistive memory (R1 or R2) is connected in series with either a PMOS or NMOS selector transistor (T1 or T2). The transistor has two roles; (1) to determine the share of total programming voltage V top -V bot (V prog ) that is seen over the resistive memory and (2) to limit the current flowing through the device during a programming operation. Both objectives are achieved by modulating V gate when a non-zero V prog exists. As seen in the previous chapter there are two standard RRAM programming operations called SET and RESET which render the device in a low (LRS) or high (HRS) resistive state. In the case of oxide-based RRAM (OxRAM) the resistance of a thin layer (tens of nanometres) of a transition metal oxide (TMO) material is sandwiched between two metal electrodes and can have its resistance modified through application of electrical pulses which redistributes the conductive oxygen vacancies in the device oxide (Fig. 4.2b).

V gate1 The mechanism of resistance modulation in an oxide based resistive memory. Oxygen vacancies, which form a conductive filament between electrodes, are removed and instantiated due to reduction-oxidation reactions with oxygen ions through the application of RESET and SET pulses. Oxygen ions exists in an 'oxygen reservoir' between the top electrode of the device and the oxide layer.

R1 T1 T2 R2 V gate2 V top V top V bot V bot (a)
As previously noted, ion channels within neuronal membranes regulate the flow of ionic current into and out of the cell's somatic body which acts as a capacitor. Essentially they represent transient or fixed resistances which regulate flow of charge between an extra-cellular battery and this capacitor and serve as a fundamental building block of animal nervous systems. In the same fashion we propose to use the programmed resistance state of resistive memory devices as a direct means of realising the resistance (or conductance) of biological ion channels in neuromorphic circuit models.

In chapter 3, the devices were used in their low resistance state (LRS) (high conductive state). The resistance extremes of devices in the LRS span from 5kΩ to 50kΩ. However, in order to achieve neural and synaptic dynamics which evolve over 'biological timescales', therein the rate at which relevant environmental variables themselves change, much larger values of resistance than this are required. For example, given a capacitor of 1pF and a desired resistor-capacitor (RC) time constant, τ RC , of some hundreds of micro seconds, a resistance on the order of hundreds of MΩ would be required. Therefore it is required to operate the resistive memory in its high resistive state (HRS). In the case of OxRAM, the HRS resistance is not determined by the properties of a conductive oxygen vacancy filament but, on the contrary, the absence of such a filament. Specifically the read resistance across the two terminals is dominated by the distance of the tunnelling gap between the bottom electrode and the remnants of the disrupted conductive filament [START_REF] Yu | On the switching parameter variation of metal oxide rram-part ii: Model corroboration and device design strategy[END_REF]. Since the tunnelling current decays exponentially as a function of the tunnelling gap size, and the magnitude of the voltage applied in a RESET operation is proposed to be linearly proportional to the size of the gap, the relationship between the HRS resistance and the applied RESET voltage follows the exponential relationship as plotted in Fig. The selector transistors had a gate length of 6.7µm. Each histogram of read resistances over these cycles has been fit with a log-normal probability distribution function which here appears as a normal in log-resistance.

Hybrid neuron circuit

The most straightforward, yet still computationally useful, neuron models are the leaky-integrate and fire (LIF) models [START_REF] Gerstner | Time structure of the activity in neural network models[END_REF]. They capture the essence of a neuron's ability to integrate charge on its somatic membrane upon synaptic excitation while simultaneously leaking away this charge in time. Further, upon reaching a threshold of accumulated charge the neuron fires and emits an output pulse which can be propagated to the synaptic inputs of other LIF model neurons. The hybrid differential pair integrator [START_REF] Chicca | Neuromorphic electronic circuits for building autonomous cognitive systems[END_REF] neuron circuit model in Fig. 4.4a captures these behavioural features in an hybrid CMOS-RRAM circuit.

Importantly, the device incorporates three 1T1R structures that influence its dynamical properties. Upon the injection of input current, charge is integrated onto capacitor C1. The amount of integrated charge depends on the ratio of the resistance values 1T1R 2 (green) to 1T1R 1 (blue) and therefore allows for gain tuning. The charge which is integrated onto C1 leaks to ground at a rate defined by the resistance of 1T1R 2 (green). If the rate of integration sufficiently exceeds the rate of the leak then a threshold voltage is reached (V th1 ) (here defined using an OPAMP comparator) and an output inverter sets V out to a logic high. During this firing event, capacitor C2 is charged via the current source M4. As soon as the capacitor exceeds V th2 transistor M5 opens and shunts V in to ground -bringing to an end the pulse. Transistor M5 remains shunted to ground for the period the voltage on capacitor C2 remains in excess of V th2 , defined by the rate the charge leaks to ground through 1T1R 3 . In summary, the 1T1R 1 and 1T1R 2 affect the neuron input time constant and input gain while 1T1R 3 defines the neuronal refractory period. Two example waveforms obtained with different resistance configurations, obtained through SPICE simulation, of V in and V out under a periodic input current pulse train (1µs pulse-width of 100nA every 250µs) are shown in Fig. 4.4b. For the layout of a hybrid CMOS-RRAM neuron circuit see Appendix 6.15a. Further SPICE simulations explore the relationship between the RRAM resistances and the neurons behavioural properties. In Fig. 4.5(a), an increasing value of R2 is seen to increase the RC time constant governing how quickly an initial capacitor voltage of 0.5V decays of 0V. In addition, increasing the ratio between R2 and R1 is seen to reduce the voltage increment resulting from a single input pulse starting from an initial C1 voltage of 0V as well as limiting the maximum steady state value of V in when charged with a constant DC current source in Fig. 4.5(b) and (c) respectively. This ratio should be on the order of the product of R2 and I in for the gain to be significantly affected (Fig. 4.5(b)). Finally, the effect of R3 in tandem with the capacitance of C2 on the refractory period is plotted in Fig. 4.5(d). It is seen that, for a fixed capacitance, the refractory period increases following a power-law with the resistance value.

+ - - + M1 M2 M5 M4 C2 I in V th1 V th2 V out V in C1 V g1 V g2 V g3 M3 T1 R1 (

Hybrid synapse circuit

While the input currents injected into the neuron circuit of Fig. 4.4a were simple rectangular pulses, the synaptic currents injected into biological neurons exhibit temporal properties, reflecting the rate of release and uptake of neurotransmitter in the synaptic cleft. Such temporal behaviours are believed to be impor-tant for neural computation [START_REF] Indiveri | The importance of space and time for signal processing in neuromorphic agents: the challenge of developing low-power, autonomous agents that interact with the environment[END_REF]. Circuit models exist for mimicking biological synaptic dynamics for use in neuromorphic processors [START_REF] Indiveri | Neuromorphic silicon neuron circuits[END_REF]. The simplest model is that of the exponential synapse whereby, during an input voltage pulse (modelling a pre-synaptic action potential), the output current is stepped up and then decays exponentially in time [START_REF] Bartolozzi | Synaptic dynamics in analog vlsi[END_REF]. This is the behaviour of the hybrid differential pair synapse circuit in Fig. 4.6a. Upon an input voltage pulse, V in , a current proportional to the value of 1T1R 2 (green) flows from C1 to ground for as long the voltage drop over the diode-connected transistor M1 is turned on. As this current flows, during an active high V in pulse, the voltage at C1 reduces and turns on transistor M3 allowing an output current to flow (which can be injected into a neuron circuit model). This voltage over C1 continues to reduce for as long as the voltage difference between C1 and the potential divider node formed between 1T1R 1 and 1T1R 2 is sufficient for the diode M1 to remain turned on. Therefore, 1T1R 1 imposes a limit on the magnitude of the output current that flows in M3 as a function of the capacitor voltage at node C1. After the input pulse comes to an end, current stops flowing from C1 to ground. Instead the capacitor charges up again linearly via a leakage current from 1T1R 3 (red). Due to the sub-threshold relationship between gate voltage and drain-source current, this results in an exponential reduction in the output current. A SPICE simulation shown in Fig. 4.6b gives two examples of output current waveforms after the arrival of an input voltage pulse for two configurations of the three 1T1R structures which augment the hybrid circuit. It should be noted that, although in Fig. 4.6a one synapse circuit contains one capacitor, inside neuromorphic processors [START_REF] Qiao | A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128k synapses[END_REF][START_REF] Moradi | A scalable multicore architecture with heterogeneous memory structures for dynamic neuromorphic asynchronous processors (dynaps)[END_REF], multiple synapse circuits share (along a row or column of a synaptic array) a single capacitor and superimpose their currents onto it. This helps to reconcile the small footprint of a synapse circuit with the large footprint of a 1pF capacitor without compromising on large (biological) time constants. For the layout of a hybrid CMOS-RRAM synapse circuit see Appendix 6.14a.

I out V in V g1 V g2 V g3 M2 (

An OxRAM based intrinsic plasticity algorithm

One of the main motivations for the incorporation of resistive memory devices into neuromorphic circuit models is the opportunity of non-volatile memory elements to reduce, or eliminate altogether, static power consumption -since energy is not required in order to maintain information once programmed. Another is that, contrary to the case of neuromorphic processors based on a centralised volatile memory, it can permit the realisation of massively parallelised local adaptive plasticity mechanisms. One such local mechanism, desirable in the context of neuromorphic spiking neural network models, is neuronal intrinsic plasticity (IP) [START_REF] Stemmler | How voltage-dependent conductances can adapt to maximize the information encoded by neuronal firing rate[END_REF]. Intrinsic plasticity is a homeostatic mechanism which, along with synaptic scaling [START_REF] Gina | Homeostatic plasticity in the developing nervous system[END_REF], has proven important in order to maintain healthy neural dynamics in recurrently connected networks [START_REF] Lazar | Sorn: a self-organizing recurrent neural network[END_REF]. The basis of IP is to modify the properties that govern the excitability of a neuron such that it is simultaneously able to minimise the energy it consumes due firing while maximising the amount of information its spike train can encode. For example, if the firing rate of a neuron is bounded between zero and a maximum value determined by its refractory period, then, to transmit the maximum volume of information, its firing rate probability density function (PDF) should be a uniform distribution such that information is transmitted equally over the available bandwidth. However, if the neuron should simultaneously minimise its power consumption while firing around a target rate, its firing rate PDF should assume an exponential distribution [START_REF] Baddeley | Responses of neurons in primary and inferior temporal visual cortices to natural scenes[END_REF]. This exponential PDF is observed in neural recordings across multiple organisms such as that from the macaque inferior temporal cortex plotted in Fig. 4.7. In this experiment, the animal was habituated to a set of natural scenes, to which the neurons in its visual cortex had adapted to and thereby assumed an exponential firing rate PDF. The animal was then presented with unfamiliar noisy, unnatural, imagesresulting in the non-optimal firing rate PDF plotted using a red dashed line in Fig. 4.7.

Figure 4.7: Two probability density functions (PDFs) of the firing rate activity recorded from the temporal inferior cortex of a macaque monkey. The PDF recorded from neurons when the animal was presented with natural scenes to which it has been habituated is plotted in blue while the response of the neurons to an unnatural and noisy scene is plotted in red (dashed). Adapted from [START_REF] Baddeley | Responses of neurons in primary and inferior temporal visual cortices to natural scenes[END_REF].

To implement a naive 'technologically-plausible' intrinsic plasticity algorithm based on OxRAM device properties, we propose that the log-normal random variable, available in the HRS of OxRAM, can be combined with another random property of OxRAM to implement an in-memory Markov Chain [START_REF] Heeger | Markov chains[END_REF] that searches for a stable configuration of the neuron. This additional stochastic property we call the SET Bernoulli random variable.

The SET Bernoulli random variable

Traditionally a SET programming pulse is applied which ensures deterministically that a functioning device transitions from the HRS to the LRS. However, for the case of sub-threshold SET pulses (here below V SET =1.4V) the HfO 2 based RRAM considered in this thesis exhibits a non-deterministic switching mechanism whereby the probability of a device being SET has a dependence on the SET voltage applied over the device [START_REF] Nishi | Crossover from deterministic to stochastic nature of resistive-switching statistics in a tantalum oxide thin film[END_REF]. In order to characterise this probability-voltage relationship, devices in a 4k device array (see Appendix 6. A SET probability is extracted for each value of V SET by calculating the fraction of devices in the array which assume a resistance below the 20kΩ threshold after programming. The resulting relationship is plotted Fig. 4.9a which is seen to be well fit by a sigmoid function. In a further experiment, where each of the devices were RESET/SET cycled 100 times over a range of V SET , a probability-voltage relationship was determined for each device by calculating the fraction of times (out of one hundred) the device was SET per voltage. The relationship for three devices is shown in Fig. 4.9b, whereby a significant deviceto-device variability is observed in their sigmoidal characteristics. The device-to-device variability in this relationship over the full device population is summarised in Fig. 4.9c by plotting the SET probability in a series of boxplots over a range of SET voltages. Strikingly, for a V SET = 0.95V , corresponding to a SET probability of 0.42, the box spans the entire SET probability range and there exists a 10% chance that an arbitrary device will SET every time or not at all under the same programming condition. The variability in the sigmoidal slope and shift, the two parameters that determine the shape of the function, is plotted for the full device population in Fig. 4.9d as a scatter plot with the probability density of each independent sigmoid parameter superimposed onto opposing axes. Both parameters are seen to be normally distributed and largely uncorrelated. 4.9c and 4.9d, its also clear from visual inspection that a substantial device-to-device variability in the switching probability is present. The dispersion is most pronounced at voltages corresponding to probabilities between 21.5% and 80.5% whereby the top and bottom heatmaps of Fig. 4.10 are peppered with divergent coloured cells. The spatial distribution of this device-to-device variability would also appear to be random-uniform. However, to properly assess the absence of spatial correlations, the SET probability variability distribution was evaluated according to the NIST test suite SP800-22 [START_REF] Rukhin | A Statistical Test Suite for Random and Pseudorandom Number Generators for Cryptographic Applications[END_REF]. This test suite is commonly used to validate random number generators by running fifteen tests on the generator output -in particular searching for spatial correlations. The number walk, composed of the complete 4k device over the array and the 100 cycles for each SET voltage, passes the full suite of tests. According to these tests, the device-to-device spatial correlation can be confidently considered as non significant.

Shift Slope

Specification of algorithm

The OxRAM based Markovian IP algorithm, which exploits both the HRS log-normal random variable and the Bernoulli SET random variable, is depicted in Fig. 4.11 which operates as follows. A neuron measures its own firing rate by leaky-integrating (low-pass filtering) its output spike train. Then, at fixed intervals (here 400ms), it compares this integrated rate with a target rate. Based on this firing-rate difference error, a SET/RESET cycle is performed on 1T1R 1 and 1T1R 2 of the hybrid neuron of Fig. 4.4a. These parameters control the input gain and input time constant and thus determine the neurons excitability. By performing a RESET operation on the devices, new parameter values for the neuron circuit model are effectively sampled from a log-normal random variable -changing it's excitability accordingly. However, continually re-sampling these parameter values in this fashion will not allow the neuron to converge towards a stable state. Therefore we propose to modulate the re-sampling using the SET Bernoulli random variable. By generating a voltage over each of the RRAM devices incorporated into the neuron circuit proportional to the firing-rate difference error (between the measured and target rates) the device will transition to the LRS with some probability that is a function of this error. If the device remains in the HRS (i.e., the device did not SET after application of the voltage), then the parameter is not re-sampled. If the device switches, however, the device is immediately RESET and therefore samples a new resistance value from its high resistive state cycle-to-cycle variability distribution. Crucially, in a similar vein to the ideas presented in section 3, this implements an in-memory Markov Chain locally within each neuron by leveraging the random variables within the OxRAM nano-devices that can be harnessed through the application of simple electrical pulses.

To regulate how sensitive the parameter re-sampling is to the measured firing-rate difference error, we introduce a 'tolerance'. This tolerance sets a minimum error that is tolerated before the re-sampling probability for a neuron becomes non-zero. The tolerance is an important quantity in the algorithm since a tolerance which is too small prevents convergence to a stable state since the neuron's configuration is too sensitive to small fluctuations in its input activity. At the other extreme, a tolerance which is excessively large prevents a neuron from self-organising at all. Furthermore, we propose that 1T1R 1 , since it impacts only the gain, should re-sample from an HRS PDF with a median equal to its current resistance value while 1T1R 2 should re-sample from an HRS PDF with a median shifted by a constant learning rate from its previous value. The learning rate multiplied by the current resistance value can then be added to or subtracted from this value give the value of the new median. Since 1T1R 2 has a positive correlation with the firing rate (as it governs the input time constant) this mean shift should be positive for under-firing and negative for over-firing.

Accompanying circuits

Such an algorithm should be implemented locally, inside or near to the neuron circuit, with further analogue or digital CMOS circuits -thereby avoiding a von Neumann bottleneck. Here we provide examples of two CMOS circuits, which respectively detect the sign and magnitude of the error difference between the measured an target firing rates (Fig. 4.12), and then use these quantities to generate an appropriate SET programming voltage pulse over a given 1T1R structure (Fig. 4.13).

The circuit in Fig. 4.12 compares the constant bias voltage V target with the voltage signal corresponding to the integrated output firing rate iV out . We assume that this integration can be performed using a differential-pair integrator circuit [START_REF] Chicca | Neuromorphic electronic circuits for building autonomous cognitive systems[END_REF]. Currents equal to the positive and negative difference between iV out and V target are copied into transistors M6 and M7. If iV out is greater than V target , M7 pulls the node at the input of the inverter down to ground and results in a high output signal. In the contrary case, M6 pulls the node up to VDD, and the output of the inverter is low. The output signal U P/DN therefore provides a logical signal corresponding to the sign of the firing-rate difference error. The voltages V 1 and V 2 vary with the logarithm of this error for negative and positive differences respectively. If the error is large and positive, for example, V 2 would reflect this in assuming a large voltage relative to V 1. Additionally, in the case where iV out and V target are very close, the 'bump' circuit [START_REF] Delbrueck | Bump circuits[END_REF] in the centre of the schematic results in a large stop voltage. These five signals are then input to the circuit in Fig. 4.13 along with one control signal prog and two further bias parameters V r1 and V r2 . Each time that the active-low control signal prog is pulsed, a current flows onto capacitor C1. The resulting voltage ramp is then buffered onto the eventual 1T1R structure -appearing as a ramped voltage pulse over the device. The peak of the voltage ramp corresponds to the applied V SET voltage, and therefore determines the probability of the device being SET into its low resistance state. If the voltage stop is greater than V 1 or V 2, then the current that flows onto the capacitor is negligible and the peak V SET voltage will be zero. When this is not the case, a current proportional to V 1 if U P/DN is low or V 2 if U P/DN is high flows onto C1. This current is also modulated as a function of the sign of U P/DN and the bias voltages V r1 and V r2 . V r1 is used to influence the rate of change of the peak V SET with a negative firing range difference, and V r2 for a positive firing rate difference. This effectively allows for an asymmetrical relationship between firing rate difference and SET probability (Fig. 4.11) which may be favourable, for example, in the situation where positive deviations from the target firing rate are punished more (i.e, there is a greater chance to re-sample parameters) than a negative one since elevated firing rates result in a higher power consumption of the circuit. The peak value of V SET is plotted with the firing rate difference in Fig. 4.14 for three pairs of V r1 and V r2 -demonstrating how their values can be altered to obtain different symmetrical or asymmetrical relationships.

Spiking neural network simulation

We now assess the effectiveness of the algorithm through a set of spiking neural network simulations calibrated on the measured characteristics of the OxRAM devices. First, the case of a single neuron under stimulation from a Gaussian current source, whereby the DC current injected into the neuron resamples from a Gaussian distribution every 20ms, is considered. As described, the difference between PDF is a decaying exponential function. The excitatory neurons have a 0.2 chance to connect recurrently amongst themselves and there is no spatial connectivity kernel as is the case of liquid-state machines [START_REF] Maass | Real-time computing without stable states: A new framework for neural computation based on perturbations[END_REF].

In addition, the excitatory population excites an inhibitory population (red). The neurons in this population recurrently connect amongst themselves and also project inhibitory synapses to the excitatory population -putting on the brakes via negative feedback when the positive population is excessively excited. The

Power Consumption

A SET/RESET cycle, required to re-sample a parameter, incurs a fixed penalty in energy and therefore such an algorithm will consume an amount of energy proportional to the update rate (here 400ms) and the number of devices in a network which have undergone a SET/RESET cycle during this periodic update.

Under standard programming operations (see Appendix 6.3) the 1T1R structures studied in this paper consume approximately 50pJ per SET/RESET cycle. Neurons also pay an energy penalty every time they spike (for the DPI neuron in 180nm CMOS this is 800pJ). This is therefore an order of magnitude more expensive than a SET/RESET cycle as well as being approximately two orders of magnitude more frequent. Clearly, as is the case in biology, it becomes advantageous to expend a small amount of energy to reduce the (comparatively) much greater energy consumed via excessive neural activity. As an illustrative dress is broadcast on some form of global bus. Each of the neurons which contain this address in their pre-synaptic neuron memory generate, locally, a pre-synaptic spike. So-called 'flat' AER uses a global look-up table that sequentially reads and broadcasts address events from several neuromorphic cores, one at a time, onto a global AER bus [START_REF] Mortara | A communication scheme for analog vlsi perceptive systems[END_REF][START_REF] Kwabena | Point-to-point connectivity between neuromorphic chips using address events[END_REF]. Two similar approaches, called broadcast-mesh and hierarchical-fractal AER, extend flat AER by allowing events to 'hop' between neighbouring AER nodes [START_REF] Lin | Programmable connections in neuromorphic grids[END_REF] and to 'climb' up and down a hierarchy of routing nodes [START_REF] Joshi | Scalable event routing in hierarchical neural array architecture with global synaptic connectivity[END_REF] respectively. Such a hierarchical approach gains efficiency by exploiting the observation that, in the biological neural networks that neuromorphic processors aim to emulate, neurons connect most heavily amongst their neighbours and very sparsely with distal cells [START_REF] Sporns | The small world of the cerebral cortex[END_REF][START_REF] Perin | A synaptic organizing principle for cortical neuronal groups[END_REF]. Router-mesh [START_REF] Mukaram Khan | Spinnaker: mapping neural networks onto a massively-parallel chip multiprocessor[END_REF] and multicasting-mesh [START_REF] Zamarre Ño-Ramos | Multicasting mesh aer: A scalable assembly approach for reconfigurable neuromorphic structured aer systems. application to convnets[END_REF] AER, instead of using a global look-up table, incorporate distributed routers with local memory and digital control circuits which, based on the address of the event input to the router, steer it between neighbour neighbouring routers until it arrives at its intended destination.

As an alternative to such 'virtual' AER-based implementations of neural networks whereby digital circuits link up nodes on the fly using volatile memories, hardware exploiting the non-volatility of resistive memory arrays have also been proposed to realise physically connected silicon-based neural network models. Further to storing information on parameters in their non-volatile conductive states, RRAM devices are able serve as a physical, analogue, link between nodes and execute neural network models in-memory as discussed in chapter 3. Arrays of RRAM have been used to implement feed-forward [START_REF] Ambrogio | Equivalent-accuracy accelerated neural-network training using analogue memory[END_REF][START_REF] Li | Efficient and self-adaptive in-situ learning in multilayer memristor neural networks[END_REF] and long-short-term-memory [START_REF] Li | Long short-term memory networks in memristor crossbar arrays[END_REF][START_REF] Wang | In situ training of feed-forward and recurrent convolutional memristor networks[END_REF] neural network models as well as spiking neural networks [START_REF] Prezioso | Training and operation of an integrated neuromorphic network based on metaloxide memristors[END_REF][START_REF] Woźniak | Deep learning incorporating biologically inspired neural dynamics and in-memory computing[END_REF]. The technology has also been used to realise Hopfield networks [START_REF] Cai | Power-efficient combinatorial optimization using intrinsic noise in memristor hopfield neural networks[END_REF] and to perform in-memory inference on Bayesian networks [START_REF] Joseph S Friedman | Bayesian inference with muller c-elements[END_REF][START_REF] Faria | Implementing bayesian networks with embedded stochastic mram[END_REF].

These existing approaches to graphical hardware, however, are each subject to various sets of problems. GPUs suffer from a von Neumann bottleneck between on-chip memory centres and processing units, and therefore expend large sums of energy in the repeated transportation and storage of data. To train and perform inference with deep learning models, a substantial power consumption on the order of hundreds of watts is required [START_REF] Huang | On the energy efficiency of graphics processing units for scientific computing[END_REF]. That makes GPUs unsuitable for certain applications, such as those at the edge. Neuromorphic processors consume many orders of magnitude less power, on the order of hundreds of milliwatts [START_REF] Moradi | A scalable multicore architecture with heterogeneous memory structures for dynamic neuromorphic asynchronous processors (dynaps)[END_REF][START_REF] Filipp Akopyan | Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip[END_REF], but require digital circuitry and volatile memory to manage the routing of address events in an asynchronous manner and are subject to deadlock issues as well as upper-limits on fan-out and event bandwidth [START_REF] Moradi | The impact of on-chip communication on memory technologies for neuromorphic systems[END_REF][START_REF] Kornijcuk | Recent progress in real-time adaptable digital neuromorphic hardware[END_REF]. Both GPUs and neuromorphic processors entail a static power draw required by, potentially, large blocks of volatile memory and digital circuits. The use non-volatile resistive memory arrays partly solve these issues but, based on current approaches, come at a heavy price. To implement the dot-product operation which describes how nodes inter-connect with each other through edges, analogue voltages are required to be generated using a digital-to-analog converter (DAC).

Then, since the result of this dot-product operation is a current, it must be converted back into a voltage using a current sense amplifier. Then, it must once again be digitised using an analog-to-digital converter (ADC) to provide input for the DACs of the next layer of neurons. The energy penalties incurred due to the DAC, sense amplifier and, in particular, the ADC are generally prohibitive [START_REF] Chakraborty | Resistive crossbars as approximate hardware building blocks for machine learning: Opportunities and challenges[END_REF]. In RRAM-based neural network inference chips, even those opting for very low-precision conversions, ADCs have been observed

to consume up to 80% to the total system energy budget and require 70% of the silicon area [START_REF] Ni | An energy-efficient matrix multiplication accelerator by distributed in-memory computing on binary rram crossbar[END_REF].

This section proposes an alternative RRAM-based non-von Neumann computing fabric for the im-plementation of neural networks based on distributed systolic matrix of analogue-domain circuits, called 'tiles', which, in a similar fashion to section 4.2, incorporate local resistive memory arrays. The proposed hardware does not require volatile memory, digital routing circuits nor ADCs. One class of tile, the neuron tile, contains analogue neuron and synapse circuit models. Each of these tiles contains a local resistive memory array which determines the pre-synaptic connection weights to each neuron. Neuron tiles propagate output voltage pulses to neighbouring 'routing tiles' which use a further resistive memory array and analogue circuits to determine how these pulses are then propagated to its own neighbouring routing and neuron tiles, and so on. The resulting mosaic of computational and routing tiles not only succeeds in continuously distributing memory and computing homogeneously throughout the hardware, eliminating any von Neumann bottleneck, but it also avoids the use of volatile memories which necessitate a static power draw as well as costly analog-to-digital signal conversion. This is because all computation is performed in the analogue-domain, leveraging the local non-volatile conductance states of RRAM devices.

We first present the circuit elements and system level composition of what we call the 'neuromorphic memory mosaic', or simply 'the mosaic' for short. We then demonstrate how this substrate can be applied to create small-world neural network, or graphical, models [START_REF] Duncan | Collective dynamics of 'small-world'networks[END_REF][START_REF] Sporns | The small world of the cerebral cortex[END_REF]. Finally, we demonstrate through spiking neural network simulation how a small-world neural network, based on the neuromorphic memory mosaic, can be applied as reservoir computing model [START_REF] Jaeger | The "echo state" approach to analysing and training recurrent neural networks-with an erratum note[END_REF][START_REF] Maass | Real-time computing without stable states: A new framework for neural computation based on perturbations[END_REF] to engineer a static feature space from a input time-series for a simple linear classification model in the heartbeat arrhythmia detection task.

Column circuits

The neuromorphic memory mosaic is based on a fundamental building block referred to here as a 'column circuit'. The purpose of these circuits is to transform input voltage pulses into read currents that depend on the non-volatile conductance states programmed into a plurality of resistive memory devices. These currents can then be used for asynchronous, analogue-domain computation. In the mosaic architecture, these currents are used to update the membrane voltages of neuron circuit models in 'neuron columns' and also to determine if incoming voltage pulses should be blocked or passed through 'routing columns'.

Neuron column

The neuron column, depicted in Fig. 4.20, has at its input N parallel one-transistor-one-resistor (1T1R) structures. Pre-synaptic rectangular voltage pulses, here with a pulse-width of one microsecond, arrive at the selector transistor gate of the 1T1Rs in an asynchronous fashion. The common bottom electrode has a constant DC voltage V bot applied to it and the common top electrode is driven to the voltage V x by an operational amplifier (OPAMP) circuit. The OPAMP output is connected in negative feedback to its non-inverting input (due to the 90°phase-shift between the gate and drain of transistor M 1 in Fig. 4.20)

and has the constant DC bias voltage V top applied to its inverting input. As a result, the output of the OPAMP will modulate the gate voltage of transistor M 1 such that the current it sources onto the node V x will maintain its voltage as close as possible to the DC bias V top . Whenever an input pulse V in < n > arrives, a current i in equal to (V x -V bot )G n will flow out of the bottom electrode. The negative feedback of the OPAMP will then act to ensure that V x = V top , by sourcing an equal current from transistor M 1 . By connecting the OPAMP output to the gate of transistor M 2 , a current equal to i in , will therefore also be buffered, as i buf f , into the branch composed of transistors M 2 and M 3 in series. A transformation is then be applied to i buf f by a synaptic circuit model (here, the circuit in Appendix 6.10a). In this case, like the synapse circuit of Fig. 4.6a, it generates an exponentially decaying current that is in turn in injected onto the membrane capacitor of a leaky-integrate and fire neuron model (here, the circuit in Appendix 6.9a).

In Fig. 4.20, the results of a SPICE simulation accompany the circuit schematic as three insets where all three devices have a conductance of 250µS -representative of an OxRAM device in the HCS. It can be seen that, resulting from the three input pulses, the membrane voltage, V mem , of the neuron circuit then increases twice after two synaptic current injections before firing upon the arrival of a final third pulse.

Note that inhibitory pre-synaptic weights are possible by using an additional column structure feeding into a synapse circuit which, instead of injecting a current via PMOS current mirror into the neuron circuit, sinks current from the neuron by connecting the output of the synapse circuit to an NMOS current mirror.

Such 'differential' column structures have been designed and laid out for fabrication. In a fabricated circuit implementation of the neuron column in Fig. 4.20, the device G 0 in the column was programmed over a sweep of compliance currents that resulted in a series of conductances. After each device had been programmed, this conductance value was recorded and then an input pulse was applied to V in < 0 > resulting in a current waveform with a peak proportional to the programmed conductance state being injected onto the neuron membrane capacitor. The resulting voltage waveform was measured (using the OPAMP shown in Fig. 6.4a in a voltage buffer configuration that had been integrated with the

V top V bot G 0 G 1 G N V x i in i buff M1

Routing column

The routing column, depicted in Fig.

V top V bot G 0 G 1 G N V x i in i ref i buff V out
HCS LCS Figure 4.22: The routing column circuit with example waveforms. Input (red, left) voltage pulses, V in , draw a current i in proportional to the conductance state, G n , of the read 1T1R structures. Two devices are labelled with HCS, indicating that they have been programmed with a conductance corresponding to the high conductance state, and one is labelled LCS in reference to the low conductance state. This resulting currents are buffered (green, centre), i buf f , into a current comparator circuit where it is compared with a reference current i ref . When the buffered current exceeds the reference current a voltage pulse is generated at the column output (blue, right).

Since the conductance state of a device that is read upon the arrival of an input pulse determines i buf f , it can then be used to store in a non-volatile fashion information on what pre-synaptic voltage pulses should propagate through a routing column, and which should not. The HCS and LCS bit count distributions from the 4k device array (see Appendix 6.3), resulting from single SET and RESET operations on each device in the array are plotted in the histogram in Fig. 4.23a. Each distribution is discretised in one hundred conductance bins and the number of bits in each bin is plotted on the y-axis. This allows both the HCS and LCS distributions to viewed on the same scale which would not be possible if the distributions were plotted in terms of probability density. Vertical dashed lines in Fig. 4.23a show the large 'memory window' between the LCS and HCS distributions at two and three standard deviations. This thereby demonstrates that by programming all devices under the same programming conditions, devices in the HCS and LCS can be well distinguished within a population with a negligible bit error rate. An additional vertical dashed line shows a conductance approximately half-way between the LCS and HCS distributions -a value that could be used as a conductance threshold to separate HCS and LCS bits which define whether a pulse should be respectively passed or blocked. In Fig. 4.23b, a SPICE simulation of a DC sweep of the conductance state of an RRAM device at the input of a routing column which is subject to an input pulse, is swept from 1µS to 100µS. This conductance range spans the upper and lower tails of the LCS and HCS distributions plotted in Fig.

Neuron and routing tiles

The presented neuron column realises a bio-inspired spiking neuron model and the routing column offers a means of blocking or passing the voltage pulses generated at the output of this neuron model. These columns can be respectively agglomerated into neuron and routing 'tiles' by stacking consecutive columns side-by-side and connecting their gates, row-wise, to common input lines. A simple neuron tile, composed of only two neuron columns and receiving two inputs, is shown for means of illustration in Fig. 4.24a.

This neuron tile includes two additional rows of RRAM which receive input from the output of each of the neurons in the tile -thereby permitting local recurrent synaptic connections within the tile. Effectively the conductive states of the devices define the synaptic weights of a feed-forward layer of synapses from V in < 0 > and V in < 1 > while the extra two rows define the weights of recurrent connections within the tile as well as the weights of any self-loops that neurons make with themselves. Echoing ideas from FPGAs [START_REF] Alfke | It's an fpga![END_REF] and mesh AER schemes [START_REF] Mukaram Khan | Spinnaker: mapping neural networks onto a massively-parallel chip multiprocessor[END_REF][START_REF] Zamarre Ño-Ramos | Multicasting mesh aer: A scalable assembly approach for reconfigurable neuromorphic structured aer systems. application to convnets[END_REF], each of these input and output voltage pulses can enter from, and exit towards, the neighbouring north (N), south (S), east (E) and west (W). In order to route these pulses generated by the neurons to their neighbouring neuron tiles we require a further tile, based on the routing column circuit, whose constituent devices will direct the flow of the pulses. Such a routing tile, capable of passing and blocking pulses coming from two neurons to and from the N, S, W and E neighbours is shown in Fig. 4.25a. An experiment was performed using a fabricated version of this routing tile where two devices (coloured in green and red in Fig. 4.25a) were programmed in respective high and low conductive states. The other devices were left in the pristine state. This has the effect of allowing incoming pulses from the North to propagate out to the East, while blocking pulses coming from the South. Pairs of pulses were applied to the North and South tile inputs and, as plotted in Fig. 4.25b, an output pulse of observed at the East when input pulses arrive to the North while it remains clamped at gnd for the equivalent pulses arriving from the South. Note that the output pulse does not appear as rectangular due the fact that the output of the circuit is required to charge a pad on the probing station, through which the experiment is conducted, that has a capacitance of some tens of pico Farads.

If this output were connected to the gate of another 1T1R structure on a subsequent routing or neuron tile the rising and falling edges of the pulse would be sharper. The routing tile shown in Fig. 4.25a can pass only one pulse per direction. However, in order to enable greater configurability it is desirable to propagate different pulses that originate from difference sources but travel in the same direction differently. This can be achieved by adding additional 'routing channels' per direction. An example of a routing tile with two channels per direction is shown in Fig. 4.26. It is based on eight stacked routing columns, culminating in an 8 × 8 array of RRAM-devices. Whether input pulses propagate or not between all possible combinations of neighbouring tiles is determined by the conductive state of a corresponding device in the array. An example of how it can be configured is shown in Fig. 4.26, whereby a voltage pulse originating from the tile's southerly neighbour, V in < 1, S >, is blocked by devices in the LCS in six of the routing columns and passed in two columns which have their devices in the HCS -generating output voltage pulses V out < 1, N > and V out < 1, W > that propagate pulses to the respective north and westerly neighbours through the additional channel. Evidently, it is possible to use any number of routing channels per direction to increase configurability as desired; although the memory requirements double for each additional routing column that is used per direction (requiring a 12× 12 array for three columns per direction and a 16 × 16 array for four). The number of neuron and routing columns per tile determines the trade-off between configurability and efficiency as well as the prevalence of two potential issues of the mosaic approach -'path sharing' and 'lost pulses'. The former results when too few routing columns are used per direction and neurons on the same tile are obliged to share the same set of post-synaptic weights and connections. The latter results when pulses originating from two different pre-synaptic sources overlap at the inputs of routing or neuron columns and effectively merge into a single pulse instead of arriving to the destination neuron as two pulses.

LCS HCS

V in <N> V in <S> V in <W> V in <E> V o u t < S > V o u t < N > V o u t < E > V o u t < W > (a) (b)
These two tile types can be pieced together in the pattern depicted in Fig. 4.27a which gives rise to a continuous mosaic of neuromorphic computation and memory. Each of the green squares correspond to a neuron tile and each of the blue ones to a routing tile. The neuron tiles receive input from their neighbouring four routing tiles, update their membrane voltages using the synaptic currents that are generated as a function of the conductive states of the devices in their columns and then, upon exceeding their spiking threshold voltages, emit voltage pulses at their outputs. These pulses are then picked up by their neighbouring routing tiles, those that delivered their input, and then steered onward to their destinations as a function of the non-volatile conductance states of the devices in the blue sea of routing tiles that link up neurons in the graph. An example neural network topology, obtained by randomly programming devices in a computer model of the mosaic to be in the HCS with probabilities p r = 0.075 and p n = 0.4, in the routing and neuron tiles respectively, is shown in Fig. 4.27b. The resulting graph exhibits an intriguing set of connection patterns that strike resemblances to many of the graphical motifs observed in animal nervous systems such as central 'hub-like' neurons that send and receive synapses from numerous nodes, reciprocal connections between pairs of nodes reminiscent of winner-take-all mechanisms, as well a number of heavily connected local neural clusters [START_REF] Bullmore | Complex brain networks: graph theoretical analysis of structural and functional systems[END_REF]. The sixteen (green) neuron tiles in this mosaic model contain four neuron columns each and the forty-eight (blue) routing tiles each comport sixteen routing columns.

The mosaic is evidently better suited for neural network architectures where neurons connect most frequently with physically nearby neurons and sparsely with distal ones. Models such as feed-forward neural networks for example would be impossible or require a large extent of router path sharing. Rather, since the hardware shares the same set of spatial constraints as biological nervous systems, it is much more amenable to implement bio-inspired topologies like those considered in chapter 2 -those that naturally exhibit 'small-world' properties [START_REF] Duncan | Collective dynamics of 'small-world'networks[END_REF][START_REF] Bullmore | Complex brain networks: graph theoretical analysis of structural and functional systems[END_REF]. The use of locally-connected and modular small world networks is frequently employed by biological nervous systems as an adaptation in light of the similar spatial constraints faced by this silicon-based hardware [START_REF] Sporns | The small world of the cerebral cortex[END_REF][START_REF] Perin | A synaptic organizing principle for cortical neuronal groups[END_REF].

LCS HCS

V in <1,N> V in <0,S> V in <1,S> V in <0,W> V in <1,W> V in <0,E> V in <1,E> V in <0,N> V o u t < 1 , N V o u t < 0 , N V o u t < 1 , S V o u t < 0 , S V o u t < 1 , W V o u t < 0 , W V o u t < 1 , E > V o u t < 0 , E

Application to reservoir computing

The field of reservoir computing [START_REF] Jaeger | The "echo state" approach to analysing and training recurrent neural networks-with an erratum note[END_REF][START_REF] Maass | Real-time computing without stable states: A new framework for neural computation based on perturbations[END_REF] is an appealing use case for neuromorphic processors since it requires 'static' randomly connected small-world neural networks whose synapses are not required to be modified by on-chip adaptation circuits which can be costly in terms of memory, silicon area and energy [START_REF] Qiao | A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128k synapses[END_REF][START_REF] Moradi | A scalable multicore architecture with heterogeneous memory structures for dynamic neuromorphic asynchronous processors (dynaps)[END_REF][START_REF] Kornijcuk | Recent progress in real-time adaptable digital neuromorphic hardware[END_REF]. Reservoir computing models are characterised by the random connection of neurons amongst their neighbours with an exponentially decaying spatial connectivity probability kernel -meaning that neighbouring neurons have a higher probability of synapsing onto each other than neurons which are further apart [START_REF] Maass | Real-time computing without stable states: A new framework for neural computation based on perturbations[END_REF]. Reservoir computing models are understood to perform a temporal projection that integrates information of an input time-series into the collective instantaneous activations of the reservoirs neurons. At the end of an input time-series, the collective low-pass filtered spike-trains of the neurons, corresponding to the calcium concentration that encodes instantaneous firing-rate in biological neurons The neuromorphic memory mosaic. Green squares correspond to neuron tiles and blue squares to routing tiles. The bridges drawn between tiles correspond to the north, south, east, west signal buses carrying the V in and V out voltage pulses. (b) An example graph resulting from the random programming of devices in each of the tiles in the mosaic pictured in part (a). The green circles correspond to neurons which exist in the neuron tiles and the blue edges are defined by the resulting paths that are formed between neuron tiles through the routing tiles. The arrows on the end of each of the edges denote their direction. [START_REF] Stosiek | In vivo two-photon calcium imaging of neuronal networks[END_REF][START_REF] Joseph M Brader | Learning real-world stimuli in a neural network with spike-driven synaptic dynamics[END_REF], can be used as input features for a machine learning classification model. In neuromorphic processors, limitations in the fan-in/out and configurability of the reservoir computing model often impose certain constraints on the small-world graph properties that can be achieved [START_REF] Moradi | The impact of on-chip communication on memory technologies for neuromorphic systems[END_REF][START_REF] Kornijcuk | Recent progress in real-time adaptable digital neuromorphic hardware[END_REF]. Here we demonstrate an application of the mosaic to reservoir computing by (as in Fig. 4.27b), randomly programming the devices in routing tiles and neuron tiles with probabilities p r and p n .

Small world graph properties

Small-world neural networks are characterised by two graphical properties; the clustering coefficient and the average path length that respectively describe the tendency for nodes to inter-connect in localised clusters and how few intermediate nodes must be traversed on average to exchange information between a pair of nodes [START_REF] Bullmore | Complex brain networks: graph theoretical analysis of structural and functional systems[END_REF]. Typically, a high clustering coefficient and a short average path length are required for a graph to be considered as small-world. The so-called 'sigma' [START_REF] Mark D Humphries | The brainstem reticular formation is a small-world, not scale-free, network[END_REF][START_REF] Mark | Network 'small-world-ness': a quantitative method for determining canonical network equivalence[END_REF] coefficient is often to used a means of determining whether a graph is a small-world or not. The sigma coefficient is a number proportional to the ratio of the clustering coefficient of the graph and the clustering coefficient of a random network or equivalent size divided by the ratio of the average shortest path length of the graph and of the same random network. The greater the value of sigma above one, the more the graph resembles a small-world.

To investigate the impact of the conductance states (i.e., LCS or HCS) of devices in the neuron and routing tiles on these small-world properties, we randomly program devices in a mosaic model of sixteen neuron tiles, of four neuron columns each, and thirty-eight routing tiles, of sixteen columns each, over a sweep of p n and p r . In Fig. 4.28a a heatmap is plotted that shows how the sigma coefficient varies as these two probabilities are varied. What is apparent is that the most important parameter to obtain a graph that exhibits strong small-world characteristics is the probability of routing tile devices being in the HCS, p r . It is seen that for any value above 0.12 the resulting mosaic-based neural network is not a small-world. This should not be wholly surprising however, given the large number of possible paths that can exist between neurons through a multitude of different routing tile combinations. Therefore, given a moderate fraction of routing tile devices in the HCS, here between 3% and 12%, a variety of complex networks with rich graphical properties can be realised. The number of edges, or synapses between neurons, over the same range of probabilities is plotted in Fig. 4.28b where the probability of the routing tile devices being in the HCS is seen to have a similar effect. For a p r greater than 0.12, the number of edges in the graph rapidly explodes and tends towards a maximum value of 1296 edges.

Heartbeat arrhythmia detection

We now apply a mosaic-based small-world spiking neural network (similar to that in Fig. The results of a SPICE simulation accompany the circuit schematic as insets in Fig. 4.30. At the input it can be seen how, as V in varies, the voltage V x follows it on the same trajectory although being repeatedly, and abruptly, reset to V ref upon exceeding the threshold biases V hi or V lo -resulting in the 'chopped' dark red waveform in the left-most inset in Fig. 4.30. This results in successive waves of UP and DN events at V in and V out whose inter-pulse timing describes the rate at which the envelope of the input sinusoid changes. In this SPICE simulation, these event pulses are input to the gates of two selector transistors in a neuron column. The RRAM device that is read upon UP events has a conductance of 100µS and that on DN events, 67µS. In This demonstrates how such an interfacing scheme can be used to feed a mosaic based reservoir computing model with electrocardiogram waveforms of heartbeats. In this fashion, heartbeats, such that shown in Fig. 4.31, are converted into trains of voltage pulses that describe upward and downward changes of the signal that can then be fed to neurons within the mosaic-based model.

A simulation of a reservoir computing model realised using the mosaic architecture is then used to generate 2700 static data-points for each of 2700 ECG recordings from a single patient (patient 200) from the MIT-BIH heart arrhythmia database [START_REF] George | The impact of the mit-bih arrhythmia database[END_REF]. The patient exhibits approximately half normal and half arrhythmic beats and therefore offers a balanced dataset. The mosaic (as in Fig. 4.27a) is realised using sixteen neuron tiles with four neuron columns and forty-eight routing tiles with sixteen columns. Devices in neuron tiles are programmed in the HCS with p n = 0.35 and in the routing tiles with p r = 0.07. Assuming a differential neuron column is used, the device in the excitatory column has a 0.4 probability of being in the LCS and the corresponding device in the inhibitory column a 0.6 probability of being in the LCS -therefore 40% of synapses in the model are inhibitory and 60% are excitatory. Each of the neurons is equipped with a low-pass filter circuit [START_REF] Chicca | Neuromorphic electronic circuits for building autonomous cognitive systems[END_REF] which gives rise to thirty-six continuously evolving variables which, together, integrate information during each 700ms long ECG recording. The thirty-six instantaneous low-pass filter The full dataset is therefore composed of 2700 data-points of thirty-six features.

V ref V lo V hi V in V up V dn V up V dn V freq V top 100¡S 67¡S C 2 M 1 A 2 A 3
The first two principal components [START_REF] Wold | Principal component analysis[END_REF] for each data-point in the resulting data-set are plotted in Fig. 4.32a. Normal heartbeats are labelled as green points and arrhythmic ones as orange points. It is encouraging that normal and arrhythmic heartbeats form distinct green and orange clouds of points indicating that, at the end of the simulation, the integrated representation of the time-series manifests in two distinctive types of activation patterns that could potentially be well distinguished by a linear classifier.

We train a perceptron model, effectively linking up each up each of the thirty-six input features with two output class encoding neurons (i.e., normal or arrhythmic), with a categorical cross-entropy loss and a training split of 67% of the full dataset. The resulting model is then tested on the remaining 33%. As plotted in the green boxplot in Fig. 4.32b, on average, over 100 train/test iterations where the train and test sets are randomly shuffled on each iteration, the perceptron classifies 94.1% of the test points correctly using the mosaic engineered features as inputs on average. To understand if these integrated, mosaic engineered features, are more useful than using simply the count of pulses in each event channel alone, thereby discarding temporal information in the original signal, we train a perceptron model using the same train-test splits over 100 iterations. With these features, labelled as 'Counts' in Fig. 4.32b, a reduced average test accuracy of 84.2% results. Based on this drop in performance, of approximately 10%, it is fair to conclude that the mosaic-based reservoir computing model is able to apply a useful transformation on the delta-modulated pulse trains and that the resulting features are more linearly separable. To compare with a more traditional approach to time-series processing, a fast-Fourier transform (FFT) is applied to each 700ms ECG waveform -resulting in 125 frequency components in each of the two ECG channels.

From these 250 frequency components, a feature selection algorithm was used to extract the 36 best [START_REF] Liu | Chi2: feature selection and discretization of numeric attributes[END_REF], which were then used to train the same perceptron model with the same train-test split once more. Labelled as 'FFT' in Fig. 4.32b, these features allow the resulting perceptron model to obtain an average accuracy of 95.2%, ultimately 1% better than was possible using the same number of mosaic engineered features. However, perhaps the features generated by the mosaic could be improved through better hyperparameter tuning or through the use different neuron and synapse circuit models. The incorporation of spike frequency adaptation into neuron models, for example, is known to improve the performance of reservoir computing models [START_REF] Nicola | Supervised learning in spiking neural networks with force training[END_REF]. Furthermore, beyond random programming of the memory devices, surrogate gradient-based learning algorithms [START_REF] Zenke | Superspike: Supervised learning in multilayer spiking neural networks[END_REF][START_REF] Emre O Neftci | Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks[END_REF] could be used to learn small-world architectures and weight matrices through backpropagation in an automatic and deterministic fashion that could be transferred into the conductance states of the devices in the mosaic.

Chapter discussion

During the course of the thesis all of the circuits presented in this chapter were designed and laid out for fabrication in a hybrid CMOS-RRAM process -in which resistive memory devices are integrated between metal layers four and five [START_REF] Grossi | Fundamental variability limits of filamentbased rram[END_REF] of a standard 130nm CMOS process. This includes the hybrid neuron

V top V bot G 0 G 1 G N S S S S S 4.8V 1.2V 4.8V 1.2V 4.8V 1.2V V in <0> V in <1> V in <N> V word <0> V word <1> V word <N> V bit V src (a) S S S V bot V bit V top V src 4.8V V word (b)
Figure 4.33: (a) Schematic of the column circuit with the additional multiplexers, level-shifters (the rectangle containing the text 1.2V -4.8V) and programming and control signals required to program the conductance states of the resistive memory devices. The control signal S, selects the column for programming whereby the voltages V bit , V src and the vector V word are applied to the common top, bottom and selector gates respectively. (b) Schematic of the 1T1R structure required in the hybrid neuron and synapse circuits. The control signal, S, switches the incorporated 1T1Rs between existing inside the analogue circuit or having its terminals connected to programming voltages. neuron and synapse circuit models such that their parameters could be, instead of stored in a centralised volatile memory centre, distributed throughout the substrate and exist physically within the computational elements themselves. This avoids burning energy to maintain information in volatile memory circuits and in the generation and distribution of a potentially large number of DC biases. Furthermore it allows for each neuron and synapse to have their own independent set of parameters and the ability to determine these parameters locally using local analogue or digital circuits -an example of this was presented in the implementation of a local intrinsic plasticity Markov chain algorithm. Normally on a neuromorphic processor, groups of some thousands of neurons and synapses that exist on each core are forced to share the same sets of parameters stored in the central bias generator. This is largely imposed due to energy considerations and metal area constraints imposed by the number of metal levels and available silicon area. Considering a state of the art analogue-domain neuromorphic processor [START_REF] Moradi | A scalable multicore architecture with heterogeneous memory structures for dynamic neuromorphic asynchronous processors (dynaps)[END_REF], each bias parameter on average consumes about 4µW of power. If the chip were to have unique parameters for each neuron, assuming only three parameters were required per neuron (time constant, gain and refractory period like in Fig. 4.4a), each circuit would burn 12µW of power. This power consumption means, with only 1000 individually parameterised neurons, an undesirable 12mW of static power is required. Moreover, perhaps the least practical aspect of this would be the physical metal area required to route the three biases from the bias generator to each neuron. Assuming a four metal layer 180 nm technology [START_REF] Moradi | A scalable multicore architecture with heterogeneous memory structures for dynamic neuromorphic asynchronous processors (dynaps)[END_REF],

1.5µm 2 of area would required per neuron. For one thousand neurons, this number grows to 1.5mm 2the equivalent of half a standard silicon chip.

In the second section of this chapter, a system-level means of interconnecting neuron circuits together through a continuous mosaic of resistive memory augmented tiles, what we called the neuromorphic memory mosaic, was proposed. Just as the hybrid neuron and synapse circuits distributed memory regarding the dynamical properties of neuron and synapse circuits, the mosaic distributes information regarding the neural network topology that interconnects such neuron models. This architecture was seen to be particularly well suited for bio-inspired small-world neural network architectures and a use case of a mosaic-based small-world neural network applied as a reservoir computing feature extractor was demonstrated in the context of a heart arrhythmia detection task.

Unlike the alternative hardware means of realising neural networks, GPUs, neuromorphic processors or other RRAM array based graphical hardware, the mosaic does not require volatile memory, digital control circuitry nor digital-to-analog and analog-to-digital converters. Therefore it succeeds in realising a non-von Neumann computer on which custom neural network topologies can be implemented and configured using non-volatile RRAM devices, but without the drawbacks that normally curtail the efficiency of other RRAM-based approaches -principally the use of ADCs.

Although they were considered and presented separately, ultimately these two ideas can be combined by placing hybrid CMOS-RRAM neuromorphic circuit models at the end of the neuron columns (instead of CMOS circuit models). In this fashion, a full silicon full non-von Neumann hardware will be possible where all computation is performed locally, using the memory elements themselves, and information is transported through brief asynchronous voltage pulses. Overall the mosaic architecture arrives at something that resonates strongly with the organisational and communicative principles observed in animal nervous systems. This is perhaps due to the fact they both arise from the consideration of similar sets of spatial connectivity and energy constraints.

In reflecting on the parallels between this silicon architecture and biology, an interesting question arises concerning the spike-based nature of computation and communicating information. On one hand, biological neurons have been proposed to spike because they use inter-spike timing as a means of encoding information [START_REF] Rieke | Spikes: exploring the neural code[END_REF][START_REF] Borst | Information theory and neural coding[END_REF][START_REF] Vanrullen | Spike times make sense[END_REF] and to learn [START_REF] Levy | Temporal contiguity requirements for long-term associative potentiation/depression in the hippocampus[END_REF][START_REF] Dan | Hebbian depression of isolated neuromuscular synapses in vitro[END_REF]. On the other, it could be supposed that the spiking behaviour of neurons is an adaptation permitting the fast transfer of information via action potentials over large distances [START_REF] Cullheim | Relations between cell body size, axon diameter and axon conduction velocity of cat sciatic α-motoneurons stained with horseradish peroxidase[END_REF][START_REF] Gillespie | The relationship between axon diameter, myelin thickness and conduction velocity during atrophy of mammalian peripheral nerves[END_REF] -despite the fact that this conversion from analogue (voltage integration on a neuronal membrane) to digital (spiking) is understood to result in a significant loss of information [START_REF] Sengupta | Consequences of converting graded to action potentials upon neural information coding and energy efficiency[END_REF].

This second idea is supported by the observation that nervous systems often opt to use non-spiking, graded-potential, neurons [START_REF] Kg Pearson | Nonspiking interneurons in walking system of the cockroach[END_REF][START_REF] Katherine Graubard | Synaptic transmission without action potentials: input-output properties of a nonspiking presynaptic neuron[END_REF][START_REF] Baba | Differing afferent connections of spiking and nonspiking wind-sensitive local interneurons in the terminal abdominal ganglion of the cricket gryllus bimaculatus[END_REF], that transmit information in an analogue, diffusive, fashion when neurons can be inter-connected over short distances. From this perspective then, in a neuromorphic computing system, should silicon neurons spike, given the fact that pulses propagate equally as fast as continuous analogue signals when sent through a metal wire?

The question of whether or not spiking is computationally important is an open one. However, from the stand-point of practicality and energy-efficiency in electronic circuits the answer is yes, absolutely! In neuromorphic processors for example [START_REF] Varkey Benjamin | Neurogrid: A mixed-analog-digital multichip system for large-scale neural simulations[END_REF][START_REF] Qiao | A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128k synapses[END_REF][START_REF] Davies | Loihi: A neuromorphic manycore processor with on-chip learning[END_REF], neurons communicate via brief address events [START_REF] Moradi | The impact of on-chip communication on memory technologies for neuromorphic systems[END_REF][START_REF] Kornijcuk | Recent progress in real-time adaptable digital neuromorphic hardware[END_REF]. If it were required to transmit continuous analogue signals between neurons, the bandwidth of an AER-style approach to routing would most likely plummet and necessitate a rethink of how neuromorphic processors might realise neural networks. Equally in the mosaic, not only would it require a re-design in the columns circuits such that they could transmit continuous signals but, most crucially, would mean that RRAM devices must be read continuously. Due to the large conductance values in the HCS (low resistances of the LRS) of most RRAM technologies, this would entail a significant undesirable power consumption. By implementing spiking neuron models in the mosaic, regardless if they are of computational advantage or not, it allows information to be transmitted in brief pulses, obviating the need for DACs and ADCs. In addition, due to their asynchronous nature, energy expensive RRAM device reading operations occur only when the information they store is needed. Overall this underlines the importance of event-based computation in the development of energy efficient neuromorphic computing hardware -in particular for those which leverage resistive memory devices.

Chapter 5 Conclusion

While the three chapters of this thesis were presented largely independently of one another, they are together intended to paint overlapping regions of a larger picture of a full Bayesian neuromorphic computing system. Chapter 2 showed how biological nervous systems could be used as a source of inspiration in the development of memory efficient model architectures in the proposal of a model for the air-current evoked escape response of the cricket cercal system. When compared to a universal approximator (i.e, a generic deep learning approach), the bio-inspired architecture was found to allow for a reduction of up to two orders of magnitude in the required number of synaptic parameters when applied to the detection of simulated attacking predators. Additionally, the specific directed topology consisting of neurons with well defined roles and logical relationships was seen to permit interpretation of the model. It was also shown that neurons and synapses with dynamical properties were important in addressing tasks based on temporal information in a model inspired by the elementary motion detection circuits in the early visual system of Drosophila.

Chapter 3 addressed the question of how the parameters of such neural network models could be determined if they were based on oxide-based resistive memory devices which, if it were possible to overcome their myriad of non-ideal properties, could greatly reduce the energy required in connectionist AI approaches. A solution was presented leveraging the framework of Bayesian machine learning. Bayesian machine learning provides the tools that allow certain non-ideal properties of RRAM, namely their variability, to be harnessed -turning the crippling drawback of the technology into an indispensable, core computational primitive. In-situ and ex-situ realisations of Markov chain Monte Carlo sampling, leveraging RRAM as random variables from the perspective of their variability, were proposed to train RRAM-based Bayesian models. In the in-situ setting (section 3.2) it was observed in experimental implementations of the algorithm that RRAM-based MCMC sampling was capable of training models with five orders of magnitude less energy than a state of the art CMOS alternatives. This result reflected the potential of this approach in bringing adaptive learning capabilities to the edge where they cannot currently be implemented due to energy constraints. Another key result was observed in the ex-situ setting whereby a Bayesian neural network model, transferred onto an RRAM-based hardware, was able to leverage the uncertainty in its output predictions in order to say 'I don't know' in a safety-critical application. By gating decisions using uncertainty thresholding the RRAM-based Bayesian neural network was able to take zero erroneous, potentially harmful, actions. Despite its widespread use across countless domains, one hurdle faced by Metropolis-Hastings MCMC sampling algorithms applied to the training of neural network models is their drop in efficiency as the number of model parameters and the size of datasets increase. This makes it challenging to apply the algorithm to extremely large models and big data datasets like those used in the deep learning setting. A potential solution for this limitation was discovered in section 2.2, where the proposed cricket terminal abdominal ganglion escape response model was able to achieve an equivalent accuracy to model with two orders of magnitude more parameters. Such memory efficient bio-inspired directed neural network architectures offer a means of maintaining model sizes at a reasonable level where Metropolis-Hastings MCMC sampling can be at its most effective, while tackling higher complexity tasks. Approaches using MCMC with mini-batches were also discussed as a means of scaling to bigger datasets.

Chapter 4 completes the picture by detailing a non-von Neumann computing fabric on which RRAMbased bio-inspired models can be realised. By incorporating resistive memory into neuron and synapse circuit models in section 4.2, inspired by the organisation of animal nervous systems, centralised volatile memory centres were avoided -alleviating the von Neumann bottleneck and eliminating static power consumption for parameter storage. This was seen to be of benefit in the implementation of local, massively parallelised, plasticity mechanisms which would be impossible through existing approaches based on centralised volatile memories. An example was presented whereby the high resistive state of an oxidebased RRAM was used to implement a Markov chain mimicking a homeostatic intrinsic plasticity algorithm observed in animal nervous systems. Then in section 4.3, by introducing the spiking neuron column circuit and the routing column circuit, a non-von Neumann means of inter-connecting neurons in a custom topology was proposed, a neuromorphic memory mosaic; based on a continuous array of tiles of such column circuits using RRAM to store information. Crucially, the prohibitive cost of analogue to digital conversion, which dominated the energy budget in the simulation of the RRAM-based MCMC sampling system designed in section 3.2, could be avoided since all computation and routing is performed in the analogue-domain. It was also discussed in section 4.4 that the asynchronous spiking, or event-based, nature of computation employed by the mosaic was also key in avoiding energy expensive continuous reading of RRAM devices. While this mosaic of neuromorphic memory is not amenable to models such as feed-forward networks, it was discussed how biological nervous systems, which face similar spatial connectivity constraints to the silicon-based mosaic, could once again provide a solution to this potential limitation by taking inspiration from the small-world organisation of their neural network architectures. An example of a small-world reservoir computing model, with random local dense inter-connectivity, was implemented in a simulation of the neuromorphic memory mosaic and applied as a means of engineering a feature space for a linear machine learning classifier.

The motivation of carrying out this work was that, by exploring the interface between machine learning, biology and technology and understanding the constraints imposed and the opportunities offered by each, a new hardware-focused approach to AI which could reduce the energy requirements of intelligent, locally-adaptive, edge systems could be proposed. The three chapters of this thesis put forward ideas for how this could be achieved and, along the way, solutions for problems in the ethical application of AI, namely enabling model interpretability through the use of bio-inspired architectures and the quantification of prediction uncertainty through the Bayesian modelling approach, were encountered. Given then that various issues faced by AI can be solved independently through the ideas developed in each of the thesis to describe resistive memory characteristics. Third, if we use resistive memory devices then we want to minimise as much as possible how often they are read. One means to do this is to construct a neural network with a spiking activation function -since memory devices will be read sparsely and in an asynchronous fashion. Finally, looking towards biological nervous systems is important for two reasons. One is that bio-inspired architectures are compatible with the proposed mosaic architecture due their small-world connectivity patterns. The other is that, as shown in the first chapter, bio-inspired neural architectures allow model sizes to be reduced substantially. Given that Metropolis-Hastings MCMC sampling performs better in lower dimensional model spaces, bio-inspiration also offers a means of scaling up the algorithm to more complex tasks.

Perspectives and future work

In addition to the chapter specific perspectives, detailed in each of the chapter discussions, there are some final and more general perspectives for future work listed briefly here;

• Develop the hardware: Starting with the circuits designed, laid out and fabricated during this thesis, it will be important to fully test and validate each of the structures -in particular those which could not be tested in time for the conclusion of this thesis. These circuits can serve as a basis on which to realise a full memory mosaic, as proposed in section 4.3, in silicon that also incorporates the hybrid neuromorphic circuits presented in section 4.2. Secondly, the peripheral control and • Further bio-inspired architectures: Bio-inspired neural network models, because of their memory efficiency and small-world graphical properties, were understood to be key in addressing the scaling issue of the MCMC sampling algorithm as well as easing the spatial connectivity constraint of the neuromorphic memory mosaic architecture. While, admittedly, the models presented in chapter 2 are small and application specific, future work should understand how bio-inspired modules like these can be interconnected and what other mechanisms might be required to glue everything together and realise an artificial nervous system model. Close attention should be paid to research dedicated to the mapping of full brain connectomes in insects like Drosophila [START_REF] Eichler | The complete connectome of a learning and memory centre in an insect brain[END_REF][START_REF] Xu | A connectome of the adult drosophila central brain[END_REF][START_REF] Franconville | Building a functional connectome of the drosophila central complex[END_REF][START_REF] Takemura | A connectome of a learning and memory center in the adult drosophila brain[END_REF],

that can inspire further architectural motifs, as well as to brain structures which are becoming increasingly well understood such as insect mushroom bodies [START_REF] Eichler | The complete connectome of a learning and memory centre in an insect brain[END_REF][START_REF] Xu | A connectome of the adult drosophila central brain[END_REF][START_REF] Takemura | A connectome of a learning and memory center in the adult drosophila brain[END_REF][START_REF] Aso | Mushroom body output neurons encode valence and guide memory-based action selection in drosophila[END_REF] and central complexes [START_REF] Stone | An anatomically constrained model for path integration in the bee brain[END_REF][START_REF] Franconville | Building a functional connectome of the drosophila central complex[END_REF] that seem important in linking up sensory processing with memory and output motor responses. Further to this, bio-inspired neural networks may help to further justify the cause for the use of RRAM as a memory element in computer systems. Von Neumann implementations of neural networks, despite their poor memory efficiency, are extremely flexible -owing to the endless possibilities in how the instructions that execute in their central processing units can be recursively organised. On the other hand, RRAM-based neural networks, although potentially requiring less energy in their execution, may be much more restricted. For example one would have to work within the architectural possibilities of routing and neuron tile dimensions had been fabricated onto a neuromorphic memory mosaic chip. Furthermore, RRAM is also not amenable to models and applications where the memory might have to be re-configured a large number of times -since RRAM technologies are often very limited in terms endurance as was seen in section 3.2. This could be problematic, for example, in the case of dynamic neural networks where the graphical structure of the model changes continuously during its execution [START_REF] Neubig | Dynet: The dynamic neural network toolkit[END_REF]. By contrast, the volatile memory circuit used in the memory hierarchy of von Neumann computers can exchange bits almost endlessly.

Foregoing notions of biological synaptic structural plasticity [START_REF] Caroni | Structural plasticity upon learning: regulation and functions[END_REF], RRAM-based bio-inspired neural networks may simply be more compatible with RRAM-based computers because, just as each synapse is hardwired in biology, physical RRAM devices can have a one-to-one correspondence with each memory element in the neural network that is being modelled.

• Dynamical Bayesian SNNs: The spiking neuron activation was observed to be key to the potential efficiency of the mosaic architecture since it offered a means of leveraging resistive memory without conversions between analogue and digital domains and permitted asynchronous event-based computation which avoids continuous device reading. In the context of a Bayesian neural network model, it was also discussed how event-based computation could having to read each sample of each Bayesian neuron in a sequential manner. Furthermore, the models to which Markov chain Monte Carlo sampling was applied in chapter 3 were 'static'; in that training and inference were performed in sequential and discrete steps and the neuron models did not incorporate dynamical properties. However, neural networks that incorporate dynamical synapse and spiking neuron models, such as those considered in section 2.3 and chapter 4, were seen to be advantageous in tasks containing temporal information. Exploration is required in order to understand what properties of Bayesian spiking neural networks that would be based on such models, such as inter-spike time or average spike rate distributions, can be leveraged for inference and for MCMC sampling based training. Some recent works have started to address questions of how networks of spiking neurons can be viewed as Bayesian models -particularly in how their collective dynamics might perform sampling-based inference [START_REF] Deneve | Bayesian inference in spiking neurons[END_REF][START_REF] Huang | Neurons as monte carlo samplers: Bayesian inference and learning in spiking networks[END_REF][START_REF] Mostafa | Rhythmic inhibition allows neural networks to search for maximally consistent states[END_REF][START_REF] Echeveste | Cortical-like dynamics in recurrent circuits optimized for sampling-based probabilistic inference[END_REF]. Although this offers a starting point for such an investigation, further work is required to understand how these two disciplines can merged effectively.

• Useful edge learning: Throughout the thesis, an overwhelming focus was directed towards supervised learning approaches -with a brief flutter into the reinforcement learning domain in section 3.2. Repeated emphasis was also put on the potential for, and the importance of facilitating, edge learning. However, it is important to pose the question of whether or not local supervised learning at the edge makes sense. While large supervised labelled datasets can be used to train neural network models in an ex-situ fashion at the cloud, edge learning systems will likely have no means of acquiring such a pre-labelled never mind store it in what will likely be a constrained memory.

An end user, or a kind citizen, could perhaps take time to label data locally for the system, something like a teacher, but that seems rather impractical. Reinforcement learning in many applications could be a viable solution, but that also entails considerable trial and error and so would be limited to applications that are not safety-critical. Humans could not learn to cross a busy road by reinforcement learning for example, and autonomous vehicles, except in situations where a few million spare cars are at your disposal, could not realistically learn how to drive through a reinforcement learning approach at the edge. The third domain of machine learning, unsupervised learning, would certainly be the most appropriate. An edge system could observe sensory information and learn representations and latent variables related to the environment in which it has been placed. However, for many applications at the edge, such a passive system may not be useful. For example, considering once again the use case of the implantable cardioverter defibrillator, an unsupervised learning algorithm might learn the underlying characteristics that distinguish a normal heartbeat from an arrhythmic one, but may not have a concept that an arrhythmic heartbeat was dangerous and requires intervention. A solution to this problem might be what is broadly referred to as 'selfsupervised' learning -a sub-domain of unsupervised learning where representations can be given significance using ideas borrowed from the supervised and reinforcement domains [START_REF] Lee | Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks[END_REF][START_REF] Sermanet | Time-contrastive networks: Self-supervised learning from video[END_REF][START_REF] Jang | Grasp2vec: Learning object representations from self-supervised grasping[END_REF].

Self-supervised learning is characterised by the local generation of labels and teacher signals which can be provided to a model that has formed useful representations of its environment through local unsupervised learning. In many respects animals are also obliged to learn in this self-supervised fashion too. Take as an example the olfactory learning process that occurs in the mushroom body of insects [START_REF] Eichler | The complete connectome of a learning and memory centre in an insect brain[END_REF][START_REF] Takemura | A connectome of a learning and memory center in the adult drosophila brain[END_REF][START_REF] Aso | Mushroom body output neurons encode valence and guide memory-based action selection in drosophila[END_REF][START_REF] Aso | The neuronal architecture of the mushroom body provides a logic for associative learning[END_REF]. The mushroom body receives input from olfactory sensory neurons and, in a similar fashion to a reservoir computing model, generates a higher dimensional representation of these inputs in the Kenyon cells of the mushroom body calyx. Based on feedback from the environment, or the gastric system within its own body, dopaminergic and octopaminergic neurons modulate the strength of synaptic connections from the Kenyon cells to mushroom body output neurons that encode the learned valence of certain olfactory stimuli [START_REF] Aso | Mushroom body output neurons encode valence and guide memory-based action selection in drosophila[END_REF][START_REF] Aso | The neuronal architecture of the mushroom body provides a logic for associative learning[END_REF]. Similar principles, using the mathematical toolbox that is under development in the self-supervised learning field should be explored and adapted to the RRAM-based MCMC sampling approach to understand if this is a more viable approach to learning at the edge. Similarly ideas from continuous learning and active learning could also be explored. Alternatively, perhaps edge learning needs to propose a new set of modelling and algorithmic tools and become a domain in its own right that takes a fresh look at what a learning artificially intelligent isolated from external labels and biases can be. where i is the magnitude of the current pulse, C is the capacitance of the MOS capacitor and dV /dt is the rate of change of voltage over the capacitor. By measuring the slope dV /dt, resulting from an input current pulse, the capacitance is given by the ratio of the two quantities.

one hundred times over a range of nine word line voltages (V WL ), corresponding to the range of SET programming currents. Each device was read after each SET operation. Between each step in V WL , the devices were additionally RESET/SET cycled 100 times under standard programming conditions ensuring that, for each of the hundred cycles at different V WL , the initial conditions were the same. The conductance was read after each SET operation and recorded. This conductance data was then processed using the python library NumPy and then using the curve fit tool from SciPy giving rise to the relationships between cycle-to-cycle and device-to-device variability plotted in Fig. 6.18a. The relationships between the cycleto-cycle and device-to-device variability, σ, and the conductance median of the respective distributions were fit using a power law -the equation in Fig. 6.18a, giving parameters a and b.

(a) Figure 6.18: The relationship between the standard deviation, σ, and the conductance median for the cycle-to-cycle (blue crosses) and device-to-device (green plus symbols) variability distributions. Each can be fit with a power law, σ = aG b , where a and b can be used to model this physical relationship in a simulation.

A custom behavioural simulation of the computer-in-the-loop experiment was developed in python implementing the presented Metropolis-Hastings Markov Chain Monte Carlo sampling algorithm using these two power law fits. The proposal distributions were assumed to be normal distributions with a standard deviation equal to that of the cycle-to-cycle variability fit. Each time the simulator moves between array rows it assigns a mean to each of the proposal distributions of each of the devices in the new row using the device-to-device variability power law. Upon each new proposal each device samples from a normal distribution around this mean. The sampling was performed using the random function suite from the NumPy library. The likelihood and acceptance ratio calculations were performed in the log-domain as described in chapter 2 of the thesis.

Appendix -Note on implementation of neural network models

A variety of tools were used to realise the different types of neural network models applied across the chapters. The PyTorch [START_REF] Paszke | Automatic differentiation in pytorch[END_REF] deep learning framework was used in chapter 1 to implement the TAG model and the benchmark models. In chapter 2, TensorFlow [START_REF] Abadi | Tensorflow: A system for large-scale machine learning[END_REF] was used to implement the benchmark neural networks that the RRAM-based MCMC experimental models were compared to. In chapter 2, the PyMC3 framework was used to train and implement the Bayesian neural network models [START_REF] Salvatier | Probabilistic programming in python using pymc3[END_REF]. In the first section of chapter 3, the spiking neural network simulation library Brian2 [START_REF] Dan | The brian simulator[END_REF] was used to implement the intrinsic plasticity models while a custom spiking neural network simulator was developed based on the odeint library, available from the SciPy python package, to perform the simulations in the second section of chapter 1 and the second section of chapter 3.

Appendix -Observed shift of the OxRAM normal random variable

An interesting device behaviour not previously documented in the literature was observed under continual RESET/SET cycling whereby the OxRAM normal random variable, leveraged in chapter two, was seen to shift, expand shift and contract. Each subsequent set of one hundred reads in the high conductance states were fit with a normal distribution. In the plots of Fig. progresses through the colours in the order of the visible spectrum -from red to violet. Therefore the random variable of the first one hundred cycles is red and that of the last one hundred cycles is violet. (a) An example of a device which does not change significantly as it is cycled and all of the fitted normal distributions more or less overlap. (b) An example of a device where the physical normal random variable is seen to shift significantly as it is cycled. The normal distribution fitted from the first one hundred cycles (red) is centred around 260µS and has a low standard deviation. However, as the device is cycled, the median of the fitted normal moves, progressively, towards a lower conductance and its standard deviation increases.

Appendix -Cercal system model input neuron circuit implementation

Vdd Vdd tanh(V c -V)

¡mod mod I spike R C V out V in V c v i in N hairs (a)
Figure 6.24: Circuit implementation of an input neuron. In order to integrate numerous events into a single continuous voltage, a chain of OR gates can be used to drive a single current mirror that injects the current pulse, i in on the parallel resistor-capacitor circuit. An OPAMP connected as a voltage follower can then buffer this voltage into a hyperbolic tangent function circuit whose output can be used as an input feature. This voltage contains information on the recent spiking activity from a population of event-based sensors.

Appendix -Robustness of TAG model to random parameter permutations

Neuromorphic implementations of neural network models also necessitate that the models take into account technological constraints such that they can be implemented in an efficient manner on a silicon substrate. This is at odds with deep learning whereby graphic processing units are typically employed but without a great deal of consideration of the underlying hardware. In contrast to the von Neumann architecture of GPUs which separate processing and memory centres on the chip, neuromorphic computing models are increasingly turning to resistive memory. The pre-synaptic weights to a neuron are defined using a row or RRAM as shown in Fig. 6.25a. However, resistive memory devices are intrinsically random and, as such, it is extremely challenging to program a device conductance state with high-precision [START_REF] Ambrogio | Statistical fluctuations in hfo x resistive-switching memory: part i-set/reset variability[END_REF].

Therefore, when programming RRAM, a closed-loop program and verify scheme is typically employed. In such schemes, a device is iteratively programmed and read until the resulting conductance value obtained after programming falls within a tolerated error margin [START_REF] Hu | Memristorbased analog computation and neural network classification with a dot product engine[END_REF][START_REF] Mochida | A 4m synapses integrated analog reram based 66.5 tops/w neuralnetwork processor with cell current controlled writing and flexible network architecture[END_REF].

(a) Figure 6.25: Scheme indicating how a row of three resistive memory devices realise three synaptic connections between input activations V and a post-synaptic neuron. By applying these activations over the columns of the row of resistive memory the current that flows out of the rows will be equal to the dot product V • g, where g is the conductance vector of the row. In the inset of device an arrow indicates the closed-loop programming routine that is used in order to program the device conductance state, g [START_REF]Edge AI Chipsets: Technology Outlook and Use Cases[END_REF], whereby the memory is iteratively programmed until its conductance falls within a pre-determined range of error.

Therefore it is instructive to understand the robustness of the TAG model to random fluctuations to the synaptic parameters that correspond to different sizes of tolerated error margins. We do this by means of re-sampling each parameter from a normal distribution, centred on the optimised value, over a range of distribution standard deviations. The ROC curves resulting from this sweep are plotted in Fig. 6.26a. The ROC curves are observed to be minimally degraded up to a standard deviation of 10% and, thereafter, somewhat collapse. Like resistive memory devices, the neural components of biological systems are also observed to be very noisy and random. In order to cope with this, nervous systems have been widely understood to employ signal averaging strategies, for example sending multiple synaptic connections between pairs of neurons and averaging out the error [START_REF] Faisal | Noise in the nervous system[END_REF]. This averaging strategy can be easily mimicked in the neuromorphic model by simply adding more rows of resistive memory per post-synaptic neuron and then dividing the current that flows out of the row by the number of rows used, using for example a simple current divider circuit [START_REF] Robert | Very accurate current divider[END_REF] (depicted in Fig 6 .27a). In Fig. 6.28a, we evaluate ROC curves over the same range of normal distribution standard deviations but this time employing the proposed averaging technique over sixteen rows per post-synaptic neuron. In this case, the ROC curves appear not to be degraded even up to 30% standard deviation demonstrating the effectiveness of this strategy -albeit increasing the memory requirements as a linear function of the number of rows used for averaging. The trade-off between the number of rows averaged over and the standard deviation of the normal distribution that re-samples the optimal parameter values are summarised in the plot of Fig. 6.28b. After it spikes it enters a refractory period during which incoming current pulses are no longer integrated. Note that when the membrane voltage approaches the spiking threshold the output almost switches twice. This is an issue present in this circuit since it does not incorporate a positive feedback connection onto the membrane capacitor which helps the device switch as it approaches the threshold. In addition it can be seen that the resting potential of the membrane is not zero but some tens of micro volts. (b) The same circuit as in part (a) but with a different bias configuration. Namely, the bias current governing the input time constant has been reduced resulting a greater input time constant and therefore more output spikes.

Figure 1 . 1 :

 11 Figure 1.1: Early computer systems. (a) A photo of a Jaquard loom. The hole punch card can be seen as a vertically running white sheet on the front face of the apparatus. (b) A model of the analytical engine proposed, but never built, by Charles Babbage. (c) Jon von Neumann stands next to a portion of the Electronic Discrete Variable Automatic Computer (EDVAC). (d) The beast robot consisting of large metal drum which, inside, contained a small network of transistors that controlled its movement.

Figure 1 . 2 :

 12 Figure 1.2: Types of neuron models. (a) The McCulloch-Pitts neuron model whereby the sum of the multiplication of the elements of an input vector X and a synaptic weight vector w is output by the neuron. (b An electrical circuit model for a neuron proposed by Hodgkin and Huxley that describes how parameters, such as the neuron membrane voltage V c , evolves in time.)

Figure 1 . 3 :

 13 Figure 1.3: Examples of computer models applied in artificial intelligence. (a)The perceptron model composed of an input layer of neurons, representing the input data points, and an output layer of neurons each of which corresponds to a prediction, normally a label, about the input. These outputs, y, are inferred using the dot product between the data X and the weight matrix W and then the weight vector can be updated, or trained, through the multiplication of the difference between a teacher signal t and the actual outputs y with the input data X and a learning rate α. (b) A search tree model whereby each green circle represents a node, a specific state in a game of chess for example, and each blue branch that extends downwards a possible decision taken from that node. Blue upward pointing arrows provide a notion of backing-up the search tree, such that information about future states can inform what decision is taken in the current state -drawn as the green node at the top of the tree. (c An example of arch learning, using the task of learning the properties of an arch -drawn as the blue rectangular blocks. Arch learning allows the construction of a semantic network with relations (blue directed arrows) between nodes (green circles) that characterise the object to be recognised. (d) An example of a nearest neighbours unsupervised learning model whereby the distance between the centroid of a data cluster (green and blue crosses) and a new data point (white circle) can be used to determine to which cluster it belongs.

Figure 1 . 4 :

 14 Figure 1.4: Computing hardware supporting AI computer models. (a) A field programmable gate array (FPGA) is composed of an array of configurable logic blocks (blue squares) which receive and send digital signals with their neighbouring blocks. The logic of each block is generally implemented using a look-up table.The logic of the look-up table and block inter-connectivity is determined using a memory that is generally shared by a subset of of blocks (b) Graphics processing units are generally composed of multiple processors, each with their own local memory, that execute in parallel on multiple cores. Their execution is determined by a global threading controller and a global memory. (c) Neuromorphic processors are composed of multiple neuron cores, each containing an array of neurons and synapses that, with either analogue or digital circuits, model their dynamical properties. A content addressable memory is used to interconnect neurons on the same core as well as neurons on other cores which communicate through a global address-event representation (AER) interface. (d) A dot(inner)-product operation can be implemented by applying features (V[0], V[1], V[START_REF]Edge AI Chipsets: Technology Outlook and Use Cases[END_REF] and V[START_REF] Strubell | Energy and policy considerations for deep learning in nlp[END_REF]) of a voltage vector V of the columns of a cross-point array of resistive (conductive) elements G. The currents (i[0], i[1], i[START_REF]Edge AI Chipsets: Technology Outlook and Use Cases[END_REF], i[START_REF] Strubell | Energy and policy considerations for deep learning in nlp[END_REF]) that flow out of each row correspond to the vector i.

Figure 1 . 5 :

 15 Figure 1.5: Common memory bit cells (a) A six transistor static random access memory (SRAM) cell whereby the cell is selected for programming or reading using the wordline (WL) and the binary bit that is present within the cell can be readout based on the current flowing out of the bitlines (BL). (b) A dynamic RAM (DRAM) cell whereby the wordline (WL) is used to read the bit, stored as a voltage on the capacitor, of the cell. DRAM requires a refresh controller in order to keep the capacitors charged or discharged appropriately. (c) Three different types of resistive memory (RRAM) cell are shown -(left) oxide/conductive-bridge RAM (Ox/CBRAM), (centre) phase change memory and (right) magnetic RAM. Arrows on each two-terminal device indicate the conductance mechanism which is used to store the bits within the resistive/conductive states of the cell. White arrows in the MRAM cell indicate the spin polarisation of the magnetic layer.

Figure 1 . 6 :

 16 Figure 1.6: A Venn diagram showing the three predominant research fields, and the emphasis on studying their intersections, covered in this thesis.

Figure 2 . 1 :

 21 Figure 2.1: Optical and scanning electron microscopy images of the cricket and its terminal abdominal ganglion. (a) Three panel image showing different extents of zoom of the cricket cercal hairs. (right) An image of a wood cricket. A green rectangle highlights the base of its right cercal appendage. (centre) a scanning electron microsopy image of the base of a crickets cerci. A green circle highlights one of the sockets in which cercal hairs pivot within. (left) A scanning electron microscopy image of a cercal hair embedded within a socket. (b) An optical microscopy image of an isolated terminal abdominal ganglion. The two extensions at the base of the image correspond to the neuropil where the sensory neurons on the two cerci project their sensory afferent. The two vertical extensions at the top of the image are the two axon bundles composing the nerve chord that ascends up to higher ganglion. (c) An image of the TAG where a red cobalt chloride dye, appearing almost black in high concentrations, has been used to stain a subset of the afferents projecting from the left cerci into the ganglion. (d) An image of the TAG where cobalt dye, appearing purple in the image, has been used to stain the cell bodies, dendrites and the axonal projections up the nerve chord, of a subset of giant interneurons. Neuron cell bodies can be see clearly on the left-most border of the TAG which send projections, both dendritic and axonic, into the middle of the ganglion.

Figure 2 . 2 :

 22 Figure2.2: Architecture of the cercal system escape response model. Colours of synaptic connections do not correspond to synaptic inhibition or excitation but instead are used to help distinguish between sets of synapses. (a) The feed-forward connections between each of the feature maps neurons and the six interneurons each of which encodes a particular property of the input. (b) The lateral connections between interneurons, including the (green) mutual connectivity between the directionally selective neurons, (red and blue) the lateral connections from the air speed sensitive neurons onto the directionally selective interneurons and (black) the mutual lateral connections between the air speed sensitive neurons. (c) The global regulation network which receives (green) synapses from all of the low frequency air current feature map neurons and sends regulatory synaptic connections to each of the six interneurons.

Figure 2 . 3 :

 23 Figure 2.3: (a)The activations of the seven interneurons included in the cercal escape system model ascending up the nerve cord to a higher ganglia. Such a higher ganglion is proposed to contain a neuron or group of neurons, here labelled jump, that looks for certain patterns of activation in the seven interneurons to activate a jump central pattern generator, labelled as jump CPG, to initiate an escape motor routine. (b) An optical microscopy image of the ganglion chain of the wood cricket, stained using a cobalt chloride dye. The terminal abdominal ganglion sits at the base of this chain which ascends up to the central brain contained within the head of the animal. Intermediate ganglion, which for example control one of the insects motor systems, appear as more densely stained red dots along the chain.

Fig. 2 .Figure 2 . 4 :

 224 Fig.2.4a plots the ROC curves resulting from an 'ablation' study of the proposed TAG model whereby combinations of individual components are considered separately. Specifically, the curves correspond to V1 -only the interneurons, V2 -the interneurons plus their lateral connections, V3 -the interneurons plus the global regulation network and V4 -the interneurons plus the lateral connections as well as the global regulation network. For means of comparison, the ROC curve of a logistic regression model, effectively a single neuron taking input from all sixteen feature map neurons, is plotted in black. In the case of model V1, it is observed that, despite increasing the number of neurons from one (i.e. the logistic regression model) to seven, there is a decrease in the tolerated FPR. This indicates that the extraction and combination of specific features alone does lead to an effective model. In model V2, the addition of lateral connections between the interneurons is seen to greatly reduce the tolerated FPR and noticeably increase the area bounded under the ROC curve. This demonstrates the importance of allowing neurons, each encoding different properties of the total sensory landscape, to communicate and compete, via lateral synapses -increasing the contrast between their responses. In model V3, the lateral connections are removed and the global regulation network is added. While the tolerated FPR deteriorates with respect to model V2, there is still a slight improvement with respect to the ROC curve of the logistic model and greatly improved with respect to the neurons acting alone. This suggests a certain importance in regulating the activations of the interneurons as a function of the background air current intensity which, intuitively, should make it easier for the jump neuron to detect the activation pattern of an attack in the presence of fluctuating background conditions. Finally, in model V4, the lateral connections and global regulation network are included together. The resulting ROC curve, plotted in blue, bounds noticeably more area than any of the other models and achieves tolerated FPR of 0.07 -well below that of the other

Figure 2 . 5 :

 25 Figs.[START_REF]Edge AI Chipsets: Technology Outlook and Use Cases[END_REF].6a and 2.6b show the resulting tolerated FPR and the number of parameters required by each of these single hidden layer and three hidden layer models respectively. In each bar plot the tolerated FPR and the number of parameters for the TAG model V4 are plotted as horizontal green and blue dashed lines for comparison. What is striking is that, for the deep learning models to obtain en equivalent performance to the TAG model, they are seen to require between one and two orders of magnitude more synaptic parameters. Specifically, to match the TAG model performance, the single hidden layer neural network is seen to require between 128 and 256 neurons and, for the three hidden layer model, between 16 and 32 neurons per layer.

Figure 2 . 6 :

 26 Figure 2.6: (a) Pairs of bar plots over a range of hidden layer sizes for a single hidden layer universal approximator. The bars show the tolerated FPR (green bars, left y-axis) and the number of parameters required by the model version (blue bars, right y-axis). Dashed horizontal lines show (green) the TAG model tolerated FPR and (blue) the number of parameters required in the TAG model. The minimum TPR is 0.95. (b) Pairs of bar plots over a range of hidden layer sizes for a three hidden layer universal approximator. The bars show the tolerated FPR (green bars, left y-axis) and the number of parameters required by the model version (blue bars, right y-axis). Dashed horizontal lines show (green) the TAG model tolerated FPR and (blue) the number of parameters required in the TAG model. The minimum TPR is 0.95.

Figure 2 . 7 :

 27 Figure 2.7: The Akaike information criterion (AIC) is plotted in log-log scale over range of hidden layer width for (orange) a single hidden layer and (peach) a three hidden layer deep learning models. Horizontal dashed lines show the AICs of a (grey) logistic regression model and (blue) the TAG model.

Figure 2 . 8 :

 28 Figure 2.8: Relationship between the tolerated false positive rate and the percentage of the weights closest to zero pruned for the standard (blue) and L 1 -norm regularised (green) models. The horizontal dashed line shows the tolerated false positive rate achieved by the TAG model. The number associated to each dot, drawn when the relationship crosses the horizontal dashed line, denote how many parameters the model required to match the tolerated FRP for the TAG model. (a) For the case of a single hidden layer MLP with 256 neurons in the hidden layer. (b) For the case of a three hidden layer MLP with 32 neurons per hidden layer.

Figure 2 . 9 :

 29 Figure 2.9: Tables of synaptic weight parameters between pre-and post-synaptic pairs of neurons. The number in each cell shows the value of the weight and the colour of the cell denotes the sign and magnitude of the weight. (a) The table of synaptic weights between each of the sixteen feature map neurons and the seven interneurons. (b) The table of synaptic weights used for each pre-and postsynaptic interneuronal connection. The values of the interneuron biases, corresponding to their resting activations, are shown below the table.

2 . 11 .Figure 2 . 10 :Figure 2 . 11 :

 211210211 Figure 2.10: The activation distributions over the 1000 test-points for each of the directional encoding interneurons. The blue distributions show the hyperbolic tangent activations when the prevailing air-current direction was equivalent to the preferred direction of the sensory neurons that feed into the feature map neurons that each neuron receives excitation from. The red distributions correspond to when the prevailing direction of the air-currents was not in the neurons preferred direction. (a) Directional encoding neuron for air-currents from 45°. (b) Directional encoding neuron for air-currents from 135°. (c) Directional encoding neuron for air-currents from 225°. (d) Directional encoding neuron for air-currents from 315°.

Figure 2 .

 2 Figure 2.12: (a) The hyperbolic tangent activation of the glob neuron due to the feed-forward connections from the feature map neurons. The blue distribution shows the hyperbolic tangent activations for the testpoints where the background air-current intensity was low and the red distribution when the background air-current intensity was high. (b) The sigmoidal activation distributions of the jump neuron for the (blue) attack test-points and (red) ambient test-points. A decision on whether to initiate a jump escape routine can be made by comparing the activation with a probability threshold along the x-axis.

  2.13). Each ommatidia has an identical structure which processes visual information from a small region of the full visual field. It is composed of four distinct layers of neuropil -the retina, lamina, medulla and the lobula as shown in Fig.2.14(a). The neural pathway for detecting elementary motion within this structure is depicted schematically in Fig.2.14(b). It begins at the retina where cells transduce light into electrical signals. These then synapse onto L1 and L2 cells in the lamina. At this stage the information is rectified into ON and OFF pathways -ON pathways carry information on luminescence increments and the OFF pathways luminescence decrements. The L1 and L2 cells in the lamina synapse predominantly onto the Mi1, Tm3 and Tm2, Tm1 cells in the medulla. These cells are implicated in implementing temporal delays between adjacent activity in spatial regions in the visual field[START_REF] Behnia | Processing properties of on and off pathways for drosophila motion detection[END_REF]. The pairs of cells in the medulla synapse onto T4 and T5 cells in the lobula which are then excited if a spatiotemporal correlation exists between it's pre-synaptic cells indicating motion. Subsets of T4 and T5 cells are sensitive to motion in one of the four cardinal directions up, down, left and right and terminate independently in one of four distinct layers in the lobula plate (LP). Groups of lobula plate tangential cells (LPTC) are excited by activity in one layer of the LP and fire to indicate motion in a specific direction. Another group of lobula plate/lobula columnar type two (LPLC2) cells are excited only by diverging motion in each layer of the LP such that

Figure 2 .Figure 2 . 14 :

 2214 Figure 2.13: A scanning electron microscopy image of an the eye of Drosophila. Each 'dot-like' ommitada on the surface extends down through the retina, lamina and medulla of the insect

Figure 2 . 15 :

 215 Figure 2.15: The one dimensional delay and correlate spiking neural network and its functional basis. (a) A 1D delay and correlate SNN. Open blue circles represent LIF CMOS neurons, green arrows excitatory synapses and red dashed arrows inhibitory synapses. Vertical lines separate the three layers of the network into the abstracted functions as performed by the different layers of neuropil in the elementary motion detection system. (b) A time domain plot for the three neurons involved in detection of motion -B1 and B2 being excited in sequence and their resulting signals meeting at C1. Red spikes correspond to the activity of a presynaptic neuron, blue spikes to the activity of the neuron associated with the plot and the green trace is the neuron input voltage resulting from the incoming pre-synaptic spikes.

Figure 2 . 16 :

 216 Figure 2.16: The two dimensional equivalent of the delay and correlate network. Blue filled circles represent LIF CMOS neurons where the fill colour indicates the pathway involved in detecting the directions UP (green), DOWN (purple), LEFT (yellow), and RIGHT (blue) while the central shared low time constant pathway is coloured red. Green arrows represent excitatory synapses and red dashed arrows inhibitory ones. The spatial organisation (left) of the inputs to a 2D delay and correlate SNN (right) is shown.

Figure 2 . 17 :

 217 Figure 2.17: The connection pattern required between the 2D elementary motion detection SNNs that span the visual field in an array, and the four readout neurons. The coloured blue circles correspond to the output layer neurons from Fig. 2.16 with their spatial organisation indicating which direction they detect. Only one set of connections from one 2D motion detector to the four readout neurons has been shown for simplicity.

Figure 2 . 18 :

 218 Figure 2.18: The average F 1 score as a function of the generation during the genetic optimisation of the model parameters.

Figure 2 . 19 :

 219 Figure 2.19: Results of the spiking neural network simulation demonstrating the performance and power consumption of the two network configurations over a range of grating frequencies. (a) The sensitivity tuning curve corresponding to the performance of the topology in detecting the correct direction of motion over a range of grating frequencies. The red points correspond to the Slow network configuration and the green points to the Fast network configuration using the parameters from the genetic optimisations. (b)The firing-rate tuning curve corresponding to the response frequency of the neuron defined as the number of times the UP neuron fires per second over the same range of grating frequencies. It can be seen that, resulting from the reduced response in the Slow configuration, the power consumption is reduced in this state relative to that of the Fast configuration inline with the reduced area under the tuning curve.

Figure 3 . 1 :

 31 Figure 3.1: Diagram of the computer-in-the-loop with the resistive memory array. The computerin-the-loop experiment using the RIFLE NplusT engineering test system, which incorporates a digital sequencer, 100 MHz arbitrary waveform generators, 70Msample/sec cell current measurement capability, as well as a C++ programmable computer. The computer can configure pulses applied by the arbitrary waveform generator to source (SL), bit (BL), and word lines (WL) signals. The CTRL bus controls periphery circuitry integrated within the hybrid CMOS-OxRAM wafer and determines to which device in the array the SL, BL and WL signals will be applied to (here the red coloured 1T1R structure). All these signals are interfaced to our 200 mm (8-inch) wafer through a 25-pin probe-card, which contacts to 25 metal pads integrated on top of the back-end-of-line of the wafer. Using this setup, the computer is therefore able to program the conductance states of devices integrated in the array, read the resulting states, and then, based on these results, re-program the devices in the array in a continuous program-read loop.In this fashion, we can implement RRAM-based algorithms, whereby programming operations can be determined based on the feedback of the result of the previous programming steps. In the case of the RRAM-based MCMC sampling algorithm implemented here, after reading programmed device conductance values, the computer is then responsible for calculating the dot-product between the data points and the conductance model, evaluating the likelihood of the conductance model and finally performing the acceptance ratio calculation. The ultimate aim of such a system is to experimentally verify RRAM-based algorithms, before a full system can be designed and integrated on a standalone chip.

Figure 3 . 2 :

 32 Figure 3.2: (left) A conductance model g, composed of four resistive memory elements, defines a linear boundary which (right) separates two classes of data (red circles from blue squares). Through application of a voltage vector V to the top electrode of the parallel resistive memories, the summed current flowing out of the common node at the the bottom electrode is equivalent to the dot-product V • g, which can then be used to determine to what class the data point V belongs.

Figure 3 . 3 :

 33 Figure 3.3: Strategies for training RRAM-based models. (a) (left) Gradient-based learning algorithms iteratively compute the derivative of an error metric with respect to a conductance model g, multiplied by a learning rate α, to determine updates to be applied to the g parameters. The ideal RRAM device should be capable of high precision and linear conductance updates. (right) The three panels show the gradientdescent algorithm for an increasing number of model updates (green crosses). From an initial model the algorithm performs gradient-based updates until it converges to a local minimum in error. (b) (left) Sampling algorithms use a proposal distribution (p(g n+1 |g n )) to propose random updates to model conductance parameters which are then either accepted or rejected. The ideal RRAM device for sampling algorithms should offer random conductance updates deriving from a known probability distribution. (right) The three panels illustrate how a sampling algorithm performs local random jumps on the posterior distribution for an increasing number of sampling operations.From an initial model, a proposal distribution generates a series of localised random jumps (dashed green lines) which are then either accepted (green cross) or rejected. The algorithm tends to accept models of a higher probability density on the posterior distribution. After a sufficient number of iterations the accepted models can be used together as an approximation of the posterior distribution (blue haze).

Figure 3 . 4 :

 34 Figure 3.4: Electrical characterisation of OxRAM cycle-to-cycle and device-to-device variability. (a) Probability density of the HCS cycle-to-cycle variability for a single OxRAM device, measured over 100 RESET/SET cycles (see Appendix 6.3), and fitted with a normal distribution (dashed line). (b) Cycle-tocycle conductance median and standard deviation for a population of 4, 096 devices, for a range of SET programming current (see Appendix 6.3). Both relationships are fit with a power law. The device-todevice variability in the quantities over the 4096 devices, extending to the 95 th and 5 th percentiles (two standard deviations), is shown with error bars at each point. (c) Joint distribution of the conductance median and standard deviation of each device in the population. 4, 096 devices have been RESET/SET cycled 100 times under the same programming conditions, and the resulting median conductance and standard deviation of each device has been plotted as a single green point -illustrating the device-todevice variability within a population. Two histograms on opposing axes show the probability densities for the conductance median and standard deviation independently.

Figure 3 . 5 :

 35 Figure 3.5: (a) Memory array architecture where RRAM conductances store the posterior approximation and calculation used to perform inference with a learned posterior. Each of the N rows store a single conductance model g n and features a digital counter element, C n . (b)A single array row is the differential conductance between conductance vectors g + and g -. A positive voltage vector, V, is applied over the top electrodes of g + and a negative opposite voltage vector -V is applied over the top electrodes of g -. If the common, bottom node is pinned at a virtual ground, the current flowing into the function block is equal to the dot-product V • g.

Figure 3 . 6 :

 36 Figure 3.6: Model proposal step between two array rows. Using the known relationship between SET programming current and the read conductances in row n, devices (pointed to by the arrows) g + [0] and g -[0] in the n+1 th array row are SET. Their SET programming currents are proportional to the conductances of the corresponding devices in the n th row. The new conductance values of g + [0] and g -[0] in the n + 1 th row are thereby sampled from normal random variables with medians equal to the conductances of the devices in the n th array row (the green probability distributions, left). The SET programming currents are fixed by applying the appropriate voltages V GATE+ and V GATE-to the transistor gates of row n + 1.

Figure 3 . 7 :

 37 Figure 3.7: OxRAM array used in the experiments. An optical microscopy image of the fabricated array is shown in the background. Scanning electron microscopy images are superimposed on top. (left) A focused ion beam a etch reveals the cross-section of a 1T1R structure (centre). In the front-end-of-line a transistor (1T) acts as a selector for the OxRAM device (1R) integrated in the back-end-of-line. (right) Imaged before deposition of the top electrode titanium layer, a 10 nm thick, 300 nm wide mesa of HfO 2 rests upon a T iN bottom electrode.

. 2 )

 2 This acceptance ratio is a number proportional to the product of the likelihood of a proposed model (p(t|g p , V)) and a prior on the proposed model (p(g p )), divided by the product of the likelihood and prior of the current model. Intuitively the acceptance ratio indicates how much more likely is the current model, given a set of observations, than the previous one and how much more does it correspond to prior beliefs on the parameters than the previous one. Given a dataset of D data points where A data points belong to the class the model is required to recognise (t = 1) and B other data points that the model should not recognise (t = 0) the Bernoulli likelihood of a model is given by:

+ 1

 1 is incremented by one and the model at row n + 1 becomes the new current model (n = n + 1). This new current model g is then used to propose a model, g p , at the next row in the array. The model stored at row n -1 is then left preserved in the non-volatile conductance states of the resistive memory devices, weighted by the counter value C n-1 which denotes the probability density of that accepted model. As this process repeats, and progresses down the rows of the memory array, the algorithm randomly walks around the posterior distribution leaving information on its probability density imprinted into the non-volatile conductance states of the OxRAM devices and the row counter values. Upon arriving at the final row of the array, n = N -1, the training process terminates resulting in a physical array of resistive memory devices which contains an approximation of the posterior distribution that can then be used in inference.

Figure 3 . 8 :

 38 Figure 3.8: Experimental results on the illustrative 2-D dataset. (a) Posterior distribution stored within the memory array after the training experiment. The two conductance parameters of each accepted model are plotted as points in the conductance plane (or model space). The initial model stored in the zeroth row is shown by a red dot. The models accepted into the subsequent array rows are plotted as green points with an opacity proportional to the associated row counter value. The transparent lines between green points show the jumps on the posterior made between successive array rows. The resulting posterior distribution is superimposed in a contour plot whereby blue and green contours denote low and high probability density respectively. (b)The two classes of data (red circles and blue squares) and a subset of fifteen models stored in randomly selected rows of the memory array. (c) The probabilistic boundary that is described by the posterior distribution stored within the resistive memory array. Each of the contour lines is annotated with a probability that corresponds to the probability that any point lying on it belongs to the class of the red data. The bounded regions between contours are coloured from red to blue whereby red denotes high confidence that a point within that shaded region belongs to the red class, and blue a low confidence.

  then scaled around zero such that the model does not require an additional bias parameter. If this step were not performed an extra column could be added to the array which would then learn the distribution of the bias parameter. During training, the algorithm was configured to recognise data points corresponding to malignant tissue samples (t = 1). At inference time the 200 previously unseen data-points from the test split were assigned a probability of being malignant using Equation 3.8. Output probabilities greater than or equal to 0.5 corresponded to a prediction of the sample being malignant (t = 1) and probabilities less than 0.5 corresponded to a prediction of the sample being benign (t = 0). The reported test accuracy corresponds to the fraction of the 200 test data points which were correctly classified.A Bayesian logistic model based on a 256 × 16 array was applied to this task. After training has been completed, the differential conductance parameters programmed into the resistive memory array are plotted in a heatmap in Fig.3.9a. Probability distributions of two parameters from the resulting posterior are shown alongside. In order to visualise the learning process, the classification accuracy of the accepted conductance model in each array row, in addition to its corresponding counter value, are plotted in green and blue traces respectively in Fig.3.9b. From an initial conductance model, which achieves a poor classification accuracy, the algorithm quickly converges onto the posterior after approximately 32 rows.

Figure 3 . 9 :

 39 Figure 3.9: Experimental results on the supervised classification of breast tissue samples. (a) (left) Heatmap of the differential conductance pairs in the 256 × 16 array after a training experiment. Cells within the heatmap are coloured from blue to red indicating the sign and magnitude of each conductance parameter. The row counter weighted distributions within columns five and eight, corresponding to two learned model parameters, are plotted in respective green and blue histograms and fitted with kernel density estimations. (b) Accuracy, evaluated on the dataset, of the conductance model (green) and the row counter value (blue) for each of the 256 rows. The first 32 rows, contained within a red rectangle, have been accepted during the burn-in period. Discarding these rows, the combined accuracy of the remaining rows on the test dataset is 97%, indicated by the horizontal dashed line. (c) Boxplots showing the test accuracy distributions over 100 separate train/test iterations for the malignant tissue recognition task using the same train/test split for (left, light green) the experimental setup, and (center and right) software-based neural network (labelled as DNN) benchmark models. The coloured boxes span the upper and lower quartiles of accuracy while the upper and lower whiskers extend to the maximum and minimum accuracies obtained over the 100 iterations. The median accuracy is indicated with a solid horizontal line. (d) The effect of the array size on the test accuracy. The test accuracy distribution shown as a boxplot for an increasing number of rows after the fixed burn-in of 32.

Figure 3 . 10 :

 310 Figure 3.10: Probability distribution of row counter values for the supervised learning tasks. Probability histogram of the values held in the row counters, labelled as counts on the x-axis, over the 100 training iterations for the (a) malignant tissue recognition task and (b) the arrhythmic heartbeat detection task. The mean row counter value over each set of 100 training iterations is shown with a vertical green dashed line -7.47 and 4.56 for (a) and (b) respectively. Because the value in each row counter denotes the number of times the devices in the subsequent row were RESET/SET cycled, this mean value corresponds to the number of times each device in the array is RESET/SET cycled.

Figure 3 . 11 :

 311 Figure 3.11: Bar plots summarising the total number of programming operations required to train a full RRAM-based model for the two supervised learning tasks. Each bar shows the total number of programming operations observed in the (a) malignant tissue recognition task and (b) arrhythmic heartbeat detection task. The proposed RRAM-based MCMC approach (first column) is compared with state of the art RRAM-based backpropagation approaches[START_REF] Ambrogio | Equivalent-accuracy accelerated neural-network training using analogue memory[END_REF][START_REF] Yao | Fully hardware-implemented memristor convolutional neural network[END_REF] applied to both the hidden-layer neural network and perceptron benchmark models. Green (bottom) bars show the total number of operations applied to the resistive memory devices and blue bars, in the case of Ambrogio '18 which uses the change in the voltage on a capacitor for the majority of parameter updates[START_REF] Ambrogio | Equivalent-accuracy accelerated neural-network training using analogue memory[END_REF], show the number programming operations applied to the capacitors. Each bar is based on the application of the approaches in[START_REF] Ambrogio | Equivalent-accuracy accelerated neural-network training using analogue memory[END_REF] and[START_REF] Yao | Fully hardware-implemented memristor convolutional neural network[END_REF] to train RRAM-based implementations of the benchmark perceptron and neural network models used here.

Figure 3 .

 3 Figure 3.12: (a) The experimental accuracy of the model trained on an array after an increasing number of decades of endurance cycles. The permanent write error probabilities of stuck in LCS and stuck in HCS devices are also plotted on a secondary axis. Arrows have been annotated to indicate which curves correspond to which axes and grey horizontal dashed lines denote the lower and upper accuracies obtained over 100 training iterations using a fresh array. (b) The experimental accuracy of the model trained on an array after an increasing number of decades of endurance cycles. The permanent write error probabilities of stuck in LCS and stuck in HCS devices are also plotted on a secondary axis. Arrows have been annotated to indicate which curves correspond to which axes and grey horizontal dashed lines denote the lower and upper accuracies obtained over 100 training iterations using a fresh array.

FFTFigure 3 . 13 :

 313 Figure 3.13: Experimental results on the supervised detection of arrhythmic heartbeats. (a) Depiction of the arrhythmic heartbeat experiment. Electrocardiograms (top left) recorded from the heart, potentially using an implanted cardioverter defibrillator [172] (bottom left), are used to extract ten features through a fast Fourier transform (labelled as FFT) which are then used as the input for a multi-layer Bayesian neural network (top right). The Bayesian neural network is trained to detect arrhythmic beats. Two of the resulting synaptic weight distributions after training are also plotted (bottom right). (b) Boxplots showing the test accuracy distributions over 100 separate train/test iterations of the arrhythmic heartbeat detection task using the same train/test split for (left, light green) the experimental setup, and (center and right) software-based neural network (labelled as DNN) benchmark models. The coloured boxes span the upper and lower quartiles of accuracy while the upper and lower whiskers extend to the maximum and minimum accuracies obtained over the 100 iterations. The median accuracy is indicated with a solid horizontal line.

Figure 3 . 14 :

 314 Figure 3.14: Simulation of the impact of the cycle-to-cycle and device-to-device variability on the test accuracy, for the supervised malignant tissue recognition task. (a)/(b) The cycle-to-cycle / device-to-device variability was multiplied artificially by a constant alpha factor in 10 iterations of simulation per alpha factor. The mean test accuracy of over the resulting ten iterations was plotted, as well as the total number of RESET/SET cycles.

  )was calculated, where S is a scalar constant. Rows of equivalent index in both arrays share a common row counter and are programmed and evaluated at the same time. The devices within the zeroth row of both arrays are initially SET by sampling from a normal random variable with the lowest available conductance median (in this simulation the conductance range extended from 50µS-200µS). The initial model is evaluated during a training episode where the cumulative reward received during the episode is recorded. For each timestep that the agent does not allow the pole to rotate 15 degrees outwith of perpendicular, or does not move outwith the bounds of the environment (both resulting in early termination of the episode), the agent receives a +1 reward. If the agent maintains the pole balanced for 500 timesteps the episode terminates resulting in an episode in which the agent has received the maximum possible score of 500. Respective models are then generated at the first row of each array by sampling new models from normal random variables with medians equal to the conductances of the corresponding devices in the zeroth row, offset by device-to-device variability. The two proposed models in the pair of rows determine the actions of the agent during the following training episode whereby the agent accelerates to either the left or the right at each timestep of the episode as a function of which array exhibited the greater response (Equation 3.9). As a function of the cumulative reward achieved during this training episode a decision is made on whether to accept or reject the proposed model using the acceptance ratio: a = p(g p ) p(g) p(r|g p , V, a) p(r|g, V, a) κ -1 . (3.10)

Figure 3 . 15 :

 315 Figure 3.15: Behavioural simulation results on the Cartpole reinforcement learning task. (a) Diagram of the Cartpole task: learning how to accelerate left or right in order to maintain a pole balanced on top of a cart within 15 degrees of normal (the vertical dashed line). The environment is described by four features; X -x-position of the cart, θ -angle of the pole to vertical, ω -angular velocity at the tip of the pole and ν -velocity of the cart. These four features serve as input to the perceptron model (upper right) where the two output neurons determine whether the agent accelerates to the left or right. (b) Reward obtained during the training episode when each of the conductance models was accepted into each row of the 512 memory array rows (the maximum reward is 500). A burn-in period of 64 rows is denoted with the red rectangle. The mean reward obtained over 100 test episodes using the combination of the models accepted after the burn-in, equal to 484 out of a maximum score of 500, is denoted with a horizontal dashed black line. (c) Boxplots showing the distribution of the mean test reward obtained during 100 testing episodes achieved in the Cartpole task over 100 separate train/test iterations. (left, light green) Behavioural simulation considering device-to-device and cycle-to-cycle variability, (centre, dark green) behavioural simulation considering only cycle-to-cycle variability and (right, blue) a DQN benchmark model. The coloured boxes span the upper and lower quartiles of mean reward, the upper and lower whiskers extend to the maximum and minimum mean rewards obtained, and the median mean reward is indicated with a solid horizontal line.

Figure 3 .Figure 3 . 17 :

 3317 Figure 3.16: (a) Circuit to perform the comparison between the linear activation of two rows from the left and right arrays. The binary output of the comparator determines the action taken at each timestep. (b) Simplified flow chart of the digital circuits used to implement RRAM-based MCMC. (c) Circuit showing the simultaneous read of a device the current rows and programming of the corresponding device in the proposed row.

Figure 3 .

 3 Figure 3.18: (a) Three cycle-to-cycle conductance variability probability distributions for a single device programmed 100 times under three different SET programming currents. The raw data is plotted as three histograms which have each been fitted with a normal distribution (dashed line). (b) The median standard deviation of the cycle-to-cycle and device-to-device conductance variability distribution of all 16, 384 devices is plotted as a function of the median of the medians of all device conductance variability distributions for a sweep of the SET programming current. The relationship can be approximated with a linear function (dashed line) .

Figure 3 .

 3 Figure 3.19: (upper left) A single target distribution (green) is plotted alongside five superimposed normal distributions (dashed lines), corresponding to the five RRAM random variable components determined through the adapted expectation-maximisation algorithm. The red, blue and green components use 85, 333 and 107 devices respectively and the leftmost and rightmost unlabelled components use 177 and 322 devices. (right) Diagram of a column of N resistive memory devices connected in a one-transistorone-resistor configuration. Three groups of devices down the column are highlighted and correspond to the weighted number of devices that will be used to generate samples from the corresponding red, green and blue normal components in the upper left figure. (lower left) Experimentally transferred probability density histogram of device conductances (blue) to a column of 1, 024 devices, superimposed over the target distribution (green), obtained using the medians and weighting factors of the normal components shown in the upper left figure.

Figure 3 .

 3 Figure 3.20: (a) Maximum value of log-likelihood obtained for an increasing number of components, K.For this target distribution, it was determined that five components were required to approximate the target distribution (dashed red line). (b) Kullback-Leibler divergence from the target to the transferred distributions calculated for an increasing number of RRAM memory cells per column. For each number of memory cells, the distribution was transferred ten times. The resulting variability in the KL divergence is shown using vertical green bars at each point indicating one standard deviation. An example of the transferred distributions for 32 and 4, 096 devices are plotted as an inset.

Figure 3 .

 3 Figure 3.21: (a) A feed-forward neural network of three neurons (M = 3) in the first layer and three neurons in the second layer. For the case of a Bayesian neural network the synapses (for example that highlighted in green) and neurons (for example that highlighted in blue) are described by probability distributions. The RRAM-based realisation of the Bayesian neuron and synapses enclosed in the dashed grey triangle is shown in part (b). (b) A proposed structure for realising a Bayesian neuron (and the synapses fanning into it) based on an N × M array of resistive memory. The distribution of conductance states of the devices in a column corresponds to the distribution of a synaptic parameter (for example that highlighted in green).Each row of the array uses devices that code for positive (g + ) and negative (g -) values that enables each parameter to be positive or negative. The inputs to the columns are the output voltages generated by the M neurons in the previous network layer. As a result of these input voltages, two currents will flow out of each row and into a neuron circuit which subtracts them and then evaluates an activation function. This activation produces an output voltage as a function of this current that can then be applied to the column of neuron arrays in a subsequent layer. The distribution of the N output voltages (blue probability distribution) is the output distribution of the neuron.

3 . 22 .

 322 The No-U-Turn Sampler (NUTS) Markov Chain Monte Carlo (MCMC) algorithm is used to explore the posterior distribution of the Bayesian model by generating proposed models through random permutations to the current model parameters -i.e. exploration through construction of a Markov Chain[START_REF] Hoffman | The no-u-turn sampler: Adaptively setting path lengths in hamiltonian monte carlo[END_REF]. Models are accepted or rejected based on a ratio of their likelihoods and priors, and all of the accepted models are stored in memory. After NUTS MCMC has collected a pre-defined number of samples, S, the algorithm terminates. All of these stored samples constitute an approximation of the posterior distribution of the model. Considering a Bayesian neural network model, with Q parameters (synapses and biases) and where NUTS MCMC has been used to record S samples, this posterior distribution is described in an S × Q matrix (upper left in Fig.3.22) where each matrix cell is, typically, a floating-point precision number. Each of the S rows of the matrix is an accepted model and each of the Q columns of the matrix store all of values of a single parameter over all of these models. Each of the Q columns therefore corresponds to the distribution of each synaptic parameter in a Bayesian neural network. The objective is to transfer the posterior approximation in the S × Q matrix into multiple N × M i resistive memory arrays. This requires three matrix pre-processing steps before the eventual single step transfer (see Fig.3.22 for a depiction of the steps).

Figure 3 . 22 :

 322 Figure 3.22: Proposed hardware realisation of feed-forward layers in the RRAM-based Bayesian neural network shown in part Fig 3.23. The output of three hidden-layer neuron arrays, corresponding to neurons one, two and eight in part Fig. 3.23, are connected to the inputs of three columns of RRAM of another neuron array, neuron one in the output layer in Fig. 3.23.As a function of the input data feature voltages (from the red coloured neurons in the first layer), the hidden layer neurons will produce activation voltages that are in turn applied over the columns of the output layer neurons causing the output layer neurons to produce activation voltages. This forward propagation of voltage continues for an arbitrary number of network layers until reaching the output layer. By sequentially applying gate voltage pulses to each row of all the arrays -the red pulses at t 0 , the green pulses at t 1 and finally the blue pulses at t N -1 -output neuron one will sequentially produce voltage activations z 0 , z 1 and z N -1 . The distribution of all activations, p(z), gives rise to an output distribution of neuron one.

  of the two output neurons of the network must learn a non-linear decision boundary that separates its respective class of noisy data points from the other. To evaluate the transfer of the Bayesian neural network model, we perform a hybrid hardware/software experiment. After termination of NUTS MCMC, the normal random variable components required for all of the model parameters are identified using expectation-maximisation. Then, 1, 024 devices in the experimental RRAM array are programmed, using the corresponding SET programming currents based on the median value of each of these random variable components. The resulting conductance values are then used to build up a computer model of the the proposed hardware depicted in Fig. 3.22 in order to perform an inference. Each RRAM cell of this computer model is randomly assigned one of the 1, 024 transferred conductances which resulted from the SET programming current that would have been used to program the equivalent device on the physical array. Examples of the resulting distributions transferred to the synaptic parameter highlighted in green in Fig. 3.23 are plotted for 1, 024, 128 and 16 rows. On average, based on the measured SET programming currents, the programming energy required to perform the transfer of the full Bayesian neural network model to the array was 1.37µJ, 172nJ and 21.5nJ for the models based on 1, 024, 128 and 16 rows respectively.

N=1024 N 128 N=16Figure 3 .

 1283 Figure 3.23: (centre) A single hidden layer feed-forward Bayesian neural network. Circles and lines in bold correspond to the neuron arrays and connections shown in part Fig. 3.22. (left) The probability density histograms and kernel density estimates for a synaptic parameter (green) using 16, 128 and 1, 024 memory cells per column. (right) The predictive probability contours of neuron one (recognising points from the red moon) and neuron two (blue moon) for 16, 128 and 1, 024 memory cells per column. Each of the red and blue moons data points are described by two feature voltages that are applied as inputs to the columns of the green neuron arrays.

Figure 3 . 24 :

 324 Figure 3.24: The uncertainty contours for the models transferred to the 1024, 128 and 16 row RRAM-based Bayesian neural network hardware. This uncertainty is determined by calculating the standard deviation of each output neuron's prediction distribution. The uncertainty in all cases is seen to be greatest at the noisy interface between the moons.

Figure 3 .Figure 3 .

 33 Figure3.25: (a) Scheme indicating how a row of three resistive memory devices realise three synaptic connections between input activations V and a post-synaptic neuron. By applying these activations over the columns of the row of resistive memory the current that flows out of the rows will be equal to the dot product V • g, where g is the conductance vector of the row. In the inset of device an arrow indicates the closed-loop programming routine that is used in order to program the device conductance state, g[START_REF]Edge AI Chipsets: Technology Outlook and Use Cases[END_REF], whereby the memory is iteratively programmed until its conductance falls within a pre-determined range of error. (b) The multilevel conductance distributions obtained with the 16k device array using a RESET/SET closed-loop programming technique. Each cumulative distribution is composed of a series of points which are ordered, per separable level, from lowest to highest conductance.

  2 demonstrated that, by embracing what have been previously considered as nonideal device properties, resistive memory based Markov Chain Monte Carlo sampling hardware can sit at the core of a new generation of intelligent edge learning systems that promise to open a door to new,

Figure 4 . 1 :

 41 Figure 4.1: (a) SEM image of a neuron. The spherical structure is the soma where ions are integrated. Further linear structures are the dendrites and axon [343]. (b) The neuron time constant (blue) is defined by the rate of change of input voltage upon a stimulus. The neuron spikes (green) above a threshold followed by a refractory period (red) during which input signals cannot be integrated.

Figure 4 . 2 :

 42 Figure 4.2: (a)The one-transistor-one-resistor (1T1R) structure whereby a resistive memory device is connected in series with either a (left) PMOS or (right) an NMOS. (b)The mechanism of resistance modulation in an oxide based resistive memory. Oxygen vacancies, which form a conductive filament between electrodes, are removed and instantiated due to reduction-oxidation reactions with oxygen ions through the application of RESET and SET pulses. Oxygen ions exists in an 'oxygen reservoir' between the top electrode of the device and the oxide layer.

Figure 4 . 3 :

 43 Figure 4.3: The high resistive state of oxide-based resistive memory devices (a) The relationship between the high resistive state resistance and the absolute RESET voltage applied over the device terminals for a single device. The mean (blue) as well as one (black) and two (brown) standard deviations are shown that correspond to the cycle-to-cycle variability distribution of the device. (b) Two cycle-tocycle variability distributions, comprising 1000 SET/RESET cycles, are shown for a single device due to two different absolute RESET voltages -blue: V r eset=1.5V, V r eset=4V and red: V r eset=2V, V r eset=4V.The selector transistors had a gate length of 6.7µm. Each histogram of read resistances over these cycles has been fit with a log-normal probability distribution function which here appears as a normal in log-resistance.

Figure 4 . 4 :

 44 Figure 4.4: The hybrid differential pair integrator neuron circuit and its behaviour. (a) Hybrid differential pair integrator neuron circuit where 1T1R structures are used to set input gain, time constant and the refractory period. All NMOS transistors have a width/length of 650nm/250nm while the PMOS transistors are 1.2µm/250nm. Capacitors C1 and C2 are both 1pF. (b) The circuit is stimulated with a train of square current pulses (1µs pulse-width of 100nA every 250µs) where the input voltage, output voltage and neuron firing threshold are shown. The supply voltage is 1.2V consistent with the voltage rating for the 130nm CMOS technology used to design the circuits. Two resistance configurations are presented which are (top) R1=1GΩ,R2=1GΩ,R3=1GΩ and (bottom) R1=40MΩ,R2=1GΩ,R3=40MΩ.

Figure 4 . 5 :

 45 Figure 4.5: (a) Vin plotted in log time for R2 swept from 10kΩ to 10GΩ for a fixed R1. As R2 increases so does the input time constant and the rate of change of voltage decay following an input spike decreases. (b) For a spike atIin and Vin at 0V the instantaneous increment of Vin is plotted with the ratio of R2 and R1 (fixed R2). Only when the ratio is on the order of magnitude as R2 multiplied by Iin (red box) is the gain affected. (c) For a DC Iin, Vin is plotted in time for a range of ratios of R2 over R1 from 2000 (red) to 0.1 (violet). The steady state value indicates the maximum Vin obtainable for a given ratio. As the ratio reduces the maximum obtainable Vin increases (for a given Iin). (d) The effect of R3 and C2 on the refractory period (during which the neuron cannot integrate incoming signals). The refractory period increases with both quantities.

Figure 4 . 6 :

 46 Figure 4.6: The hybrid differential pair integrator synapse circuit and its behaviour (a) Hybrid differential pair integrator synapse circuit where 1T1R structures are used to set a weight per synapse and current gain and the time constant of the exponential current decay per column in a synaptic array. All NMOS transistors have a width/length of 650nm/250nm while the PMOS transistors are 1.2µm/250nm. Capacitor C1 is 1pF. (b) During an input voltage pulse (red pulse of 1.2V with a 1µs pulse-width) the output current is incremented. After the pulse the current exponentially decays to zero. The current waveform for two different configurations are shown -Current 1 (R1=10MΩ,R2=500kΩ,R3=1GΩ) and Current 2 (R1=10MΩ,R2=375kΩ,R3=500MΩ). The supply voltage is 1.2V consistent with the voltage rating for the 130nm CMOS technology used in simulation.

3 )Figure 4 . 8 :

 348 Figure 4.8: Cumulative distributions of device states after sub-threshold 100ns wide programming pulses for a sweep of SET voltages. Those assuming a resistance below a 20kΩ threshold (vertical red dashed line) are defined as SET in the low resistive state. An arrow indicates the CDFs resulting from larger SET voltages.

Figure 4 . 9 :

 49 Figure 4.9: The stochastic SET operation measured in oxide-based resistive memory (a) The average relationship over an array of 4096 OxRAM devices between the probability of a device being SET into its low resistive state as a function of the SET voltage applied over the device. The relationship can be well approximated as a sigmoidal function. Two free parameters of the sigmoid function are the shift (in SET voltage for p=0.5) and the slope (the rate at which the probability changes with SET voltage) which are annotated in the plot. (b) Three single device relationships between SET probability and the applied SET voltage. Each device was cycled 100 times over a range of SET voltages and the fraction of times the device switched corresponds to the SET probability at that SET voltage for that device. (c) Boxplots showing the SET probability distribution over an array of 4096 devices for a range of SET voltages. A blue horizontal bar shows the SET probability median, limits of the box show a 25% deviation from the median, whiskers show a 45% deviation and blue points correspond to outlying devices. (d) A scatter plot where each point corresponds to the sigmoidal slope and shift fitted to a single device. In total 4096 devices are plotted. The resulting probability density function of the slope and shift are plotted on opposing axes. The device-to-device variability in slope and shift are both seen to be normally distributed and uncorrelated.

Figure 4 . 10 :

 410 Figure 4.10: Switching probability for each device in a 16x256 cell 4k device array coloured based on its deviation from a the mean switching probability over a 4k 1T1R array. Strong reds and blues indicate a significant deviation while softer shades show cells close to the mean probability.

Figure 4 . 11 :Figure 4 . 12 :Figure 4 . 13 :

 411412413 Figure 4.11: Diagram of the proposed OxRAM-based intrinsic plasticity algorithm. The hybrid neuron circuit has two 1T1R that set the properties of the neuron model (green circle).The neuron propagates a spike/pulse train to an integrator circuit (light blue block) which transforms the discrete voltage pulses into a continuous analogue voltage encoding it's activity. This signal (blue waveform) is compared with a target (black dashed line) and periodically (black pulse train) the error to this target is evaluated (red pulses). Based on these differences, SET voltage pulses are generated (red block) over the incorporated 1T1R structures in their high resistive states. This intrinsically makes use of the SET Bernoulli random variable to make a decision on whether the resistance value should be re-sampled. If the device is SET, such that the resistance after the SET operations is below 20kΩ, then it is immediately RESET at which point the resistance values of the neurons incorporated RRAM are re-sampled (green vertical bar) using a log-normal random variable (navy blue block). This random variable is accessed through the intrinsic probability density of a the HRS cycle-to-cycle variability.

Figure 4 . 14 :

 414 Figure 4.14: Effect of the biases V rl and V r2 on the SET pulse peak voltage value. For three bias configurations, which are labelled in the legend with the currents which are mirrored into the circuit instead of the actual voltage applied to the transistor gates, the peak value of the V SET ramp as a function of the firing rate difference is plotted. It should be noted that for uneven biases, an asymmetrical relationship can be obtained.

Figure 4 .

 4 Figure 4.15: (a) The neuron adapts R1 and R2 such that it fires (blue) around a target rate (purple dashed) with a Gaussian current source (red). After a change in the properties of the current (50s) the neuron re-adapts R1 and R2 to fire around the target rate once again. The rate has been normalised to the target. (b) Target (purple dashed) normalised firing rate probability densities (PDF). The PDF after organisation (green) is closer to the ideal (blue) than that of a randomly initialised neuron (red). (c) The evolving Kullback-Leibler divergence plotted as the neuron organises R1 and R2. A value of zero indicates the neuron has found the ideal configuration that maximises information and minimises power.

Figure 4 . 16 :

 416 Figure 4.16: The recurrent spiking neural network topology used in simulation. An input Poisson group (blue) feeds forward to an excitatory neuron population which are hybrid DPI neurons with intrinsic plasticity (green). A population of inhibitory neurons (red) is excited by the excitatory population and feeds back with inhibiting synapses that impose an upper limit for the mean firing rate in the excitatory population.

Figure 4 . 17 :Figure 4 . 18 :

 417418 Figure 4.17: Spiking neural network simulations employing intrinsic plasticity allows the recurrent network to self-organise to fire at a target rate. Firing rates are plotted in red, standard deviations in firing rates amongst the population in green and the number of SET/RESET cycles per periodic update is plotted in blue. (a) A plot of the average firing rate and the standard deviation between firing rates of the excitatory population plotted without the application of IP. (b) A plot of the average firing rate and the standard deviation between firing rates of the excitatory population plotted in the presence of the proposed IP algorithm. The variable T denotes the time after initialisation for the network to converge and B the standard deviation amongst firing rates after convergence. (c) Plot of the reduction in the number of RESET/SET operations as a function of time per 400ms re-sampling period. The annotated variable C denotes the number of RESET/SET cycles per re-sample after convergence.

Figure 4 . 20 :

 420 Figure 4.20:The neuron column circuit with example waveforms. Input (red, left) voltage pulses, V in , draw a current i in proportional to the conductance state, G n , of the read 1T1R structures. This current is buffered (green, centre), i buf f , into a synapse circuit model which applies a transform on it and in turn injects it into a neuron circuit model. The neuron circuit integrates this current into a membrane voltage (blue, right), V mem which causes the neuron to fire at the output after exceeding a threshold V th .

Figure 4 .

 4 Figure 4.21: (a) Five cumulative distributions resulting from the application of a single SET programming pulse on each device in an array of 4096 OxRAM devices over a range of SET programming currents, I SET . (b) Measured voltage traces from a fabricated neuron column circuit due to an input voltage pulse to the column. From an initial resting membrane voltage of 0.05V, the membrane voltage waveform obtained due to increasing series of conductance values in the neuron column circuit is shown.

4 . 22 ,

 422 works largely on the same set of principles as the neuron column. An OPAMP connected in the same negative feedback configuration mirrors the current i buf f into the circuit drawn at the right-hand side of Fig.4.22 upon the arrival on an input pulse to one of the 1T1R structure gates. This circuit compares i buf f with a DC bias current i ref using a pair of 'back-to-back' current mirrors. Intuitively the two current mirrors compete to either pull the common node, that at the input of the inverter, up to vdd (due to the two PMOS) or down to gnd (due to the two NMOS). If i ref is greater than i buf f , the inverter will output a low voltage and, on the contrary, a high voltage. A set of graphs in the inset of Fig.4.22 plot the results of a SPICE simulation of the routing column. The devices read by the first and final input voltage pulses are assigned an HCS conductance of 250µS while the remaining device, that is read upon the arrival of the second pulse, has an LCS conductance of 2µS. The magnitude of the corresponding buffered current due to each pulse, the green pulses in the central figure inset, changes accordingly whereby the buffered current due to the second pulse is negligible relative to the first and final ones. This results in only the first and final input pulses passing through the routing column, therein generating a voltage pulse at its output, while the second pulse is blocked.

Figure 4 .

 4 Figure 4.23: (a) The bit count histograms of the low (green) and high (blue) resistive states of 4096 devices. Vertical dashed lines show the memory window at two (darker green and blue) and three (lighter green and blue) standard deviations (labelled in the figure legend as 2/3σ). A black vertical dashed line shows the conductance approximately half-way between the two distributions. (b) The results of a DC SPICE simulation whereby the conductance of a resistive memory device at the input of a routing column circuit was swept between 1µS to 100µS. A black dashed vertical shows the threshold at which the output of routing column switches from low to high due to a threshold reference current, i ref = 600nA. This reference current is set equal to the current that flows due column device conductance equal to that of the vertical black line drawn in part (a).

Figure 4 .

 4 Figure 4.24: (a) Diagrams of the neuron tile. (left) Two stacked neuron columns realise a neuron tile.Four of the devices (top of the array) define the synaptic connections from inputs V in < 0 > and V in < 1 > to the two neurons and an additional four devices (bottom of the array) define the recurrent connections between neurons and neuron self-loops. Devices are coloured in black or grey to indicate respectively whether they were in the high or low conductive state respectively during the experimental results plotted in part (b). V bot corresponds to the voltage of the same name in Fig.4.20 and i buf f shows the direction of the buffered current that is mirrored into the synapse and neuron circuits which are represented by open circles (top right). This tile circuit permits the configuration of the two neuron, two input network shown here. (bottom right) These input and output voltage pulses can come from and be propagated to four neighbouring tiles to the north (N), south (S), east (E) and west (W). (b) Voltage traces measured from a fabricated neuron tile circuit. Due to an input pulse train (grey pulses) at V in < 0 > the membrane of the zeroth neuron column in the tile integrates an increasing amount of voltage (purple trace) until, after six pulses, the neuron fires (light blue trace). As a result of the feedback connection to the other neuron column, it then also exhibits an increase in membrane voltage.

Figure 4 .

 4 Figure 4.25: (a) Circuit schematic of a routing tile. Two devices coloured green and red denote respectively the devices programmed in the high and low conductance states in the experiment of part (b). Rectangular pulse waveforms depicted the left-hand side indicate where the input voltage pulses were applied during this experiment. (b) Experimental results from a fabricated version of the routing tile shown in part (a). Continuous and dashed blue traces show the waveforms applied to the North and South inputs while the orange trace shows the response of the output towards the East. The Eastern output follows the Northern input resulting from the device programmed into the high conductance state in part (a).

Figure 4 . 26 :

 426 Figure 4.26: Schematic for a routing tile composed of eight routing columns. The routing tile receives eight inputs, comprising two pulse channels per direction, labelled as < 0 > or < 1 >, from the neighbouring tiles to the north (N), south (S), east (E) and west (W), and provides complimentary outputs. An example is shown of an input pulse arriving to the common gate of the fourth row of memory. Devices are coloured green or red to denote whether they are in the HCS or LCS. It it shown that, due to this input pulse, output pulses are produced by the routing columns containing the (green) devices programmed in the HCS. V bot denotes the DC bias of the same name in Fig. 4.22 and open rectangles are used to denote the current comparator circuit at the end of the routing column.

Figure 4 .

 4 Figure 4.27: (a) The neuromorphic memory mosaic. Green squares correspond to neuron tiles and blue squares to routing tiles. The bridges drawn between tiles correspond to the north, south, east, west signal buses carrying the V in and V out voltage pulses. (b) An example graph resulting from the random programming of devices in each of the tiles in the mosaic pictured in part (a). The green circles correspond to neurons which exist in the neuron tiles and the blue edges are defined by the resulting paths that are formed between neuron tiles through the routing tiles. The arrows on the end of each of the edges denote their direction.

Figure 4 . 28 :Figure 4 . 29 :

 428429 Figure 4.28: Heatmaps of a two-dimensional sweep of p n and p r White sections of the heatmaps correspond to graphs where there existed isolated, unconnected neurons and therefore it was not possible to calculate a small-world coefficient. (a) Heatmap of a sigma coefficient that describes the small-worldness of the resulting network. A vertical blue line shows the point at which point (p r = 0.12) the graph stops being small-world. (b) Heatmap showing the number of edges that exist in a graph.

Fig 4 .

 4 [START_REF] Hassenstein | Systemtheoretische analyse der zeit-, reihenfolgenund vorzeichenauswertung bei der bewegungsperzeption des r üsselk äfers chlorophanus[END_REF] it can be seen that as the synapse circuit model injects proportional synaptic currents onto the membrane capacitor of the neuron circuit model, V mem increases until it reaches the threshold voltage of the neuron and triggers an output spike.

Figure 4 . 30 :

 430 Figure 4.30: Schematic and corresponding SPICE waveforms of a delta-modulator circuit interfaced to a neuron column. Insets show (red) the input sinusoidal waveform and the resulting 'chopped' waveform at node V x , (green) the resulting UP and DN events at nodes V up and V dn and (blue) the membrane and spiking output voltage resulting from the events generated by the input sinusoid.

Figure 4 .

 4 Figure 4.31: (left) A delta-modulated electrocardiogram of a heartbeat. (left) The black waveform shows the original ECG waveform from a single channel of a healthy heartbeat. Superimposed green and orange vertical lines show the UP and DN that have been extracted by the delta-modulator circuit. (right) The resulting timestamped UP and DN events are transduced as voltage pulses in UP and DN channels in the circuit of Fig. 4.30.

Figure 5 . 1 :

 51 Figure 5.1: (left) A neuromorphic memory mosaic architecture composed of four (green) neuron tiles and twenty five (blue) routing tiles. Each of the neuron tiles is labelled as A, B, C and D. (centre) A generic directed graph topology with neurons (open green circles) A, B, C and D interconnected by directed synapses (blue arrows). (right) Each neuron in this network corresponds to a single mosaic neuron tile. The tile has a number of rows equal to the number of pre-synaptic connections and a column for each of the sampled spiking neuron models. The blue dashed box drawn along the top row of the tile shows the synaptic distribution for the synapse labelled as g 0 and the green box around the individual neuron models, the activation distribution of neuron D.

Figure 5 . 2 :

 52 Figure 5.2: (left) Four labels, corresponding to the four axes of the envisaged Bayesian neuromorphic computing approach, and the dependencies between them (red arrows). (right) A written list summarising the reasoning behind each red arrow. The number in the list corresponds to the number denoted alongside each arrow.

Figure 6 . 1 :Figure 6 . 2 :Figure 6 . 7 :Figure 6 . 8 :Figure 6 . 9 :Figure 6 .Figure 6 . 11 :

 61626768696611 Figure 6.1: (a) The inverter unit cell used throughout the design and layout. PMOS (M 1 ) width=1.2µm, length=250nm. NMOS (M 2 ) width=650nm, length=250nm. These transistor sizes are assumed to be constant for each of the circuits unless otherwise specified. b) The inverter unit cell layout. The cell rectangular 5.05µm in height and 1.82µm across.

Figure 6 .Figure 6 . 13 :

 6613 Figure 6.12: (a) Layout of the series one-transistor-one-resistor structure where the selector transistor is an NMOS that is used for all resistive memory points in the paper. The resistive memory device corresponds to the small red, open rectangle on the upper right of the layout. The structure uses a thickoxide transistor with dimensions: width=2.72µm, length=500nm. The transistor bulk tap is connected to the common gnd. (b) Layout of the series one-transistor-one-resistor structure where the selector transistor is an PMOS. Although initially design to be used as RRAM devices incorporated into the leak bias of the hybrid synapse circuit it was eventually not used due to issues with biasing its bulk at vdd=4.8V while existing in a vdd=1.2V circuit. The resistive memory device corresponds to the small red, open rectangle on the upper right of the layout. The structure uses a thick-oxide transistor with dimensions: width=2.05µm, length=500nm. The transistor bulk tap is connected to vdd=4.8V.

Figure 6 . 14 :

 614 Figure 6.14: The hybrid synapse layout containing three NMOS 1T1R structures in the top right corner of the images and three rows of gate, top and bottom electrode multiplexers. The transistors of the synapse circuit itself are found in the centre of the top 1.2V row of the circuit and constitutes a relatively tiny area compared with the area consumed by the multiplexing and level shifting circuits. The MOS capacitor circuits are not shown.

Figure 6 . 19 :Figure 6 . 20 :Figure 6 . 21 :Figure 6 . 22 :

 619620621622 Figure 6.19: The raster plots of all of the sensory neurons, corresponding to those nested below the hairs, due an 'easy' to detect simulated attack with low background air intensity and a large fast attacker. From left to right the columns show the activity of the hairs oriented to 45°, 135°, 225°and 315°. The top row correspond to the hairs on the left cerci and the bottom row to the hairs on the right cerci. (a) As blue points, the spike-times of all of the slow (low frequency) hair sensory neurons are plotted. (b) As orange points, the spike-times of all of the fast (high frequency) hair sensory neurons are plotted.

Figure 6 . 23 :

 623 Figure 6.23: The change in the random variable of a single OxRAM device as it undergoes RE-SET/SET cycling. Examples of two devices are shown whereby the conductance in the HCS, for sets of one hundred sequential cycles are fit with a normal distribution. The colour of the normal distributionprogresses through the colours in the order of the visible spectrum -from red to violet. Therefore the random variable of the first one hundred cycles is red and that of the last one hundred cycles is violet. (a) An example of a device which does not change significantly as it is cycled and all of the fitted normal distributions more or less overlap. (b) An example of a device where the physical normal random variable is seen to shift significantly as it is cycled. The normal distribution fitted from the first one hundred cycles (red) is centred around 260µS and has a low standard deviation. However, as the device is cycled, the median of the fitted normal moves, progressively, towards a lower conductance and its standard deviation increases.

Figure 6 . 26 :

 626 Figure 6.26: he receiver operating characteristics (ROC) curves resulting from the model where the parameter values have been re-sampled from a normal distribution centred on the optimal value over a range of percentage standard deviations.

Figure 6 .

 6 Figure 6.27: A scheme for performing row averaging, whereby multiple rows of resistive memory are used together. The current corresponding to the dot-product between the input voltage vector and each of the conductance vectors sums at the common output node. This summed current is then divided by N to compute the average, where N is equal to the number of rows being averaged over.

Figure 6 .

 6 Figure 6.28: (a) The receiver operating characteristics (ROC) curves resulting from the model where the parameter values have been re-sampled from a normal distribution centred on the optimal value over a range of percentage standard deviations but averaging over N = 16 rows. (b) The tolerated false positive rate as a function of the standard deviation of the normal distributions that are used to re-sample the parameter values. Each point has been averaged over 10 re-sampling iterations.

Figure 6 . 29 :

 629 Figure 6.29: Experimental measurements of fabricated CMOS neuron circuits using two different sets of bias conditions. (a) A neuron integrates rectangular current pulses and eventually spikes. After it spikes it enters a refractory period during which incoming current pulses are no longer integrated. Note that when the membrane voltage approaches the spiking threshold the output almost switches twice. This is an issue present in this circuit since it does not incorporate a positive feedback connection onto the membrane capacitor which helps the device switch as it approaches the threshold. In addition it can be seen that the resting potential of the membrane is not zero but some tens of micro volts. (b) The same circuit as in part (a) but with a different bias configuration. Namely, the bias current governing the input time constant has been reduced resulting a greater input time constant and therefore more output spikes.

Table 3 .

 3 1: Table showing the number of operations required in each step of the MCMC sampling process for the Cartpole reinforcement learning task in addition to the energy required in each of these steps for 130nm and 28nm technology nodes. A final column shows the total energy required to train the full model with each technology.

			2 (Model accep-tance/rejection)	Step 3 (RRAM programming)	Total
	Number of repetitions	500 × 10 × 512	10 × 512	10 × 512
	Total
	energy	5.8µJ	120nJ	1.1µJ	6.9µJ
	(130nm)
	Total
	energy	2.5µJ	34nJ	1.1µJ	3.6µJ
	(28nm)

  in this work is that shown in Fig.4.30. For an input voltage waveform, V in , which in Fig.4.30 is a sinusoid, capacitor C1 acts as a high-pass filter and removes the DC component of the waveform. The AC component of the waveform is superimposed upon a bias voltage V ref that charges the non-inverting terminal of the OPAMP A 1 , whenever transistor M 1 is switched on. Since A 1 is connected in a negative feedback configuration, the OPAMP acts to buffer the voltage waveform at the non-inverting terminal to its output V x while unloading the input. Two OPAMPS, A 2 and A 3 , configured as open-loop voltage comparators compare this buffered voltage with the DC thresholds V hi and V lo . Whenever V x is greater than V hi an 'UP' event voltage pulse is generated at V up and the contrary for V dn . These UP and DN pulses feed back, on the left-hand side of the schematic in Fig.4.30, to the input of a logical OR gate that, whenever an event is generated, discharges capacitor C 2 and thereby switches the inverter that turns on transistor M 1 and resets the voltage at the non-inverting terminal of the OPAMP to V ref . This brings to an end the output event pulse. The non-inverting terminal of A 1 remains pinned to V ref for as long as it takes capacitor C 2 to charge beyond the switching voltage of the inverter through the transistor that is biased with the DC voltage V f req at its gate. The voltage V f req therefore controls the maximum rate of event generation -somewhat similar to the refractory period in a neuron circuit model. The layout of this delta-modulator circuit can be seen in Appendix 6.11a.

	4.29d), therein
	a reservoir computing model, to engineer an input feature space for a perceptron that is trained to detect

arrhythmic heartbeats. Since the neuron columns require voltage pulses as inputs, we propose to 'deltamodulate' the two-channel electrocardiogram recordings of recorded heartbeats (see Appendix 6.6) into a series of 'UP' and 'DN' voltage pulses using a delta-modulator circuit

[START_REF] Corradi | A neuromorphic event-based neural recording system for smart brain-machine-interfaces[END_REF][START_REF] Reid R Harrison | The design of integrated circuits to observe brain activity[END_REF]

. The delta-modulation circuit designed

Acknowledgements

Impact of device variability

The properties of two OxRAM random variables, like any property of resistive memory devices, comes with of course an inherent variability. In this section, we investigate the contribution of two variability sources -the cycle-to-cycle variability of the HRS log-normal random variable and the device-to-device variability in the SET Bernoulli random variable. As plotted in Fig. 4.3b, cycle-to-cycle variability in the HRS after a RESET operation follows a log-normal distribution. One way of quantifying this variability is through the standard deviation of the 'underlying' normal distribution of the log-normal. The underlying normal is simply obtained by applying a logarithm function to the log-normal distribution. Across the devices measured in this study the underlying normal standard deviation of HRS log-normal random variable was approximately between 0.4 and 0.5. The device-to-device variation in the SET Bernoulli random variable is modelled by sampling the sigmoidal shift parameter from the measured normal distribution plotted in Fig. 4.9d. The sigmoidal slope was considered to be constant. We quantify the effects of these two variability sources using the defined performance metrics annotated in Figs. [START_REF] Schwartz | Green ai[END_REF].17b and 4.17c as; time to convergence (T), standard deviation amongst firing rates after convergence (B) and count of SET/RESET cycles after convergence (C)) averaged over three independent runs from randomly initialised parameters.

We first examine the impact of the cycle-to-cycle HRS variability on the network in Figs. [START_REF] Schwartz | Green ai[END_REF].18a and 4.18b.

In Fig. 4.18a, it is seen for low values of standard deviation of the cycle-to-cycle HRS PDF the network struggles to settle to a mean precisely equal to the target -although there is a low standard deviation amongst firing rates and a low count of SET/RESET cycles after converging. This is likely linked to the fact that the unnaturally narrow log-normal distribution does not allow for a sufficient search of the parameter space. This is supported by the result in Fig. 4.18b, whereby the convergence time drops for a higher cycle-to-cycle HRS standard deviation up to 0.5. Due to the more progressive convergence, the neuron likely stops re-sampling parameters when it enters the fringes of its tolerated firing-rate difference error. In contrast however, for values larger than 0.5, the convergence time is then seen to increase again. By the same logic, this should be due to the fact that consecutive states of the Markov Chain are too distal and not sufficiently correlated. Within the range of experimentally measured cycle-to-cycle variability therefore the hyper-parameters of the algorithm have allowed it to find a sweet spot and converge faster than values higher or lower cycle-to-cycle variability. This leads to the conclusion that the intrinsic cycle-to-cycle HRS variability has a positive impact on performance. In Figs. [START_REF] Schwartz | Green ai[END_REF].18c and 4.18d the same metrics are plotted but for the case of device-to-device SET variability. Here we sample undesired horizontal shifts in the probability-error sigmoid from a normal distribution for each device. Therefore each has a permanent offset from the desired value which impacts the effective tolerance. The time to convergence in Fig. 4.18d increases with greater standard deviation of the normally distributed shifts in the probability-error sigmoid.

However, the standard deviation amongst the neurons firing rates, the mean distance from the target firing rate and count of SET/RESET cycles appear to be largely unaffected. This result is encouraging since it appears that, even in the presence of significant device-to-device variability, the IP algorithm allows the network to self-organise and find a configuration which can compensate for the non-ideal devices and fire around the target rate -at the expense of a longer period of self-organisation. example we plot the cumulative energy consumption of the two networks in Fig. 4.17b (one employing IP and the other firing at its natural rate) in Fig. 4.19. Since the target firing rate (120Hz) is significantly lower than the natural rate (200Hz) the energy consumed, in spite of the cost of initial organisation considering the SET/RESET cycling, is reduced by half. This demonstrates the opportunities in energy management of the algorithm in applications in which the system is not connected to a reliable source of power.

Neuromorphic event routing architecture

Due to the recent success of connectionist approaches to artificial intelligence [START_REF] Minsky | Perceptrons: An introduction to computational geometry[END_REF], namely various feats in deep learning [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Silver | Mastering the game of go with deep neural networks and tree search[END_REF][START_REF] Vaswani | Attention is all you need[END_REF] whereby large general purpose models are parameterised through backpropagation [START_REF] Linnainmaa | Taylor expansion of the accumulated rounding error[END_REF][START_REF] David E Rumelhart | Learning representations by backpropagating errors[END_REF], the capabilities of graphical models -a collection of nodes (neurons) networked together by edges (synapses) -has been well recognised. The fact that animal nervous systems, given hundreds of millions of years of evolution to figure out an effective strategy, have also opted to compute using graphs provides a strong argument for the potential of the approach. Aspects of this were seen in chapter 2 which concerned itself with the development of bio-inspired neural network models. A variety of hardware systems have been proposed in order to run and parameterise different types of neural networks. Graphical [START_REF]NVIDIA Launches the World's First Graphics Processing Unit: GeForce 256[END_REF][START_REF] Lindholm | Nvidia tesla: A unified graphics and computing architecture[END_REF] and tensor [START_REF] Norman P Jouppi | In-datacenter performance analysis of a tensor processing unit[END_REF] processing units (GPU/TPUs), which are application-specific-integrated-circuits (ASICs) that are optimised to perform large-scale matrix operations, act as the predominant substrate for various types of large-scale deep learning models. Neuromorphic processors, based on analogue [START_REF] Varkey Benjamin | Neurogrid: A mixed-analog-digital multichip system for large-scale neural simulations[END_REF][START_REF] Qiao | A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128k synapses[END_REF][START_REF] Moradi | A scalable multicore architecture with heterogeneous memory structures for dynamic neuromorphic asynchronous processors (dynaps)[END_REF] or digital [START_REF] Filipp Akopyan | Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip[END_REF][START_REF] Davies | Loihi: A neuromorphic manycore processor with on-chip learning[END_REF] circuit models of biologically-plausible neurons and synapses, facilitate the implementation of neural networks with bio-inspired dynamics and topologies: reservoir computing [START_REF] Jaeger | The "echo state" approach to analysing and training recurrent neural networks-with an erratum note[END_REF][START_REF] Maass | Real-time computing without stable states: A new framework for neural computation based on perturbations[END_REF] and winner-take-all [START_REF] Lazzaro | Winner-take-all networks of o (n) complexity[END_REF][START_REF] Maass | On the computational power of winner-take-all[END_REF][START_REF] Indiveri | The importance of space and time for signal processing in neuromorphic agents: the challenge of developing low-power, autonomous agents that interact with the environment[END_REF] models for example. Neuromorphic processors are composed of multiple neuron 'cores', that feature arrays of synapse and neuron circuit models like those considered in the first section of this chapter. Various implementations of an 'event routing' protocol called address event representation (AER) are used as means of realising specific neural network topologies on neuromorphic processors [START_REF] Kwabena | Point-to-point connectivity between neuromorphic chips using address events[END_REF]. In AER, each neuron is assigned an address and also has some form of memory that stores the address the neurons which pre-synaptic to it. Whenever a neuron spikes, its ad- . At the time of writing, the wafers containing the circuits have only just completed fabrication and so only a subset have undergone basic verification that could could be presented in this thesis . However, the test of the rest of these structures will be an important step on the path to realising a fully-integrated neuromorphic memory mosaic.

Something that became apparent during the layout of these circuits was the large area consumed by the transistors used to realise the circuits in programming path of the RRAM devices relative to the area consumed by the rest of the analogue circuits. Resistive memory technologies, including the OxRAM technology considered in this thesis, require large programming voltages -well in excess of the voltages tolerated by standard transistors in most scaled CMOS nodes. The 130nm CMOS node used to design the circuits in this work has a nominal supply voltage of 1.2V for example. However, the voltages required in the SET, RESET and, in particular, the forming programming operations exceed this limit. As a result of this two types of transistors are required to in order to arrive at a reliable design. One standard 'thin' gate oxide transistor and another high-voltage transistor with a 'thick' gate oxide. The thick gate oxide transistor exhibits less ideal characteristics and also requires larger dimensions to satisfy a more constrained set of layout design rules. The thin oxide transistor, which tolerates a nominal supply voltage of 1.2V, was used to design all of the analogue circuits such as the neurons, synapses, OPAMPS and logic blocks (see Appendix 6.1). The thick-oxide transistor, with a nominal supply voltage of 4.8V, was used to design all of the circuits in the programming path of the devices such as the the 1T1R structures themselves and the multiplexers, transmission gates and level-shifters which were used to deliver programming signals to the devices (see Appendix 6.1).

Two circuits showing such additional multiplexing and level-shifter circuits, required in each of the columns and around each of the 1T1R structures in the hybrid neurons and synapses, are shown respectively in Figs. [START_REF] Schwartz | Green ai[END_REF].33a and 4.33b. A column requires a multiplexer at each end of the parallel set of 1T1R structures in order to switch the common top and bottom nodes between programming waveforms and the functional nodes of the analogue circuit. To select devices for programming, a digital control signal, S, is used to multiplex the nodes between these two types of signal. Furthermore, in order to interface the output 1.2V voltage pulses, originating from the output of the neuron and routing columns, additional level-shift circuits are required in the column circuit such that a 4.8V pulse can fully turn on the thick oxide selector transistors as in Fig. 4.33a (see Appendix 6.1).

For example in Fig. 4.33a, the devices see V x = V top and V bot on their respective top and bottom electrodes and receive 4.8V pulses at their gates as a function of the their pre-synaptic activity. However, when S is high, each device sees V src and V bit at these same terminals (in reference to the names 'source' and 'bit' lines that are normally given to these lines in a standard RRAM array) and V word < n > at the gate of the n t h device. Similarly, for the 1T1R structures incorporated into the hybrid neuron and synapse circuits (Fig. 4.33b), a multiplexer is required at each node. These multiplexers share a common selection signal S and each RRAM device requires an independent select signal. Each of the selector transistor gates are connected to the 4.8V supply voltage such that, when not selected for programming, the resistance of the 1T1R is determined predominantly by conductance state of the RRAM device.

From inspection of the layout of the column circuits, pictured in Appendix 6.13a, while the resistive memory devices and the analogue circuit models respectively consume 0.05% and 15% of the silicon area per column (Fig. 4.33a), the selector transistors, level-shifters and multiplexers consume 4%, 36% and 45% of the total area respectively. Similarly, in the case of the hybrid neuron and synapse circuit layouts (see Appendices 6.15a and 6.14a), where three multiplexers are required per 1T1R structure as in Fig. 4.33b, 80% and 4% of the total circuit area is consumed by the multiplexers and selector transistors.

While there is likely room for considerable reduction in these ratios through transistor sizing and layout optimisation, what is clear is that high device density, normally listed as one of the principal benefits of RRAM technologies, is not possible due to two practical issues. First, the large minimum dimensions imposed by the layout design rules pertaining to the thick oxide transistors that are required, in this technology, to be used in the RRAM programming path. Second, due to the degraded properties of these thick oxide transistors, they are required to sized with large width over length ratios (or be composed of multiple fingers) to allow for the series voltage drop over the multiplexers, relative to the minimum resistance of the RRAM devices in the HCS (on the order of a few kΩ) to be minimal. Therefore, the most straightforward path to reducing the ratio between the area of the circuit consumed by the functional and programming elements, would be through device level engineering in two respects. First a reduction in the programming voltages to allow the use of thin-oxide transistors in the programming path and, secondly, for the minimum resistance (conductance) of devices to be increased (decreased) which would ease the constraints on the series resistance in the design of the multiplexing circuits.

In spite of the poor device density resulting from these practical considerations, this chapter presented two key ideas as to how a truly non-von Neumann neuromorphic computing fabric could be realised by leveraging the overarching advantage of resistive memory devices -conductance non-volatility.

Firstly, it was demonstrated and discussed how resistive memory devices can be integrated within chapters; what can a full Bayesian neuromorphic computing system, integrating the presented elements, look like?

A vision of a Bayesian neuromorphic computing system

A future vision of a Bayesian neuromorphic computing system based on resistive memory is depicted in Fig. 5.1. It is based on an evolution of the mosaic architecture of chapter 4, where four neuron tiles, labelled as A, B, C and D, are shown. While in the third chapter the neurons in each mosaic tile were considered as single entities, here we go Bayesian, and, in the same fashion as the hardware proposed section 3.3, each of the neuron tiles in the mosaic becomes a single 'Bayesian spiking neuron'. Each of the neuron columns in a tile correspond then to different samples of that neuron. An example of a directed Bayesian neural network is shown in the middle of Fig. 5.1. Such a topology could be realised by programming the appropriate set of interconnections in the intermediate routing tiles -which, in addition to routing events, sends information pertaining to the sample from which the event originated. Since this example contains only four neurons, the network that is drawn is largely arbitrary. However, as discussed previously, one could imagine a larger bio-inspired, small-world, neural network architecture amenable to a mosaic-based implementation featuring a greater number of Bayesian spiking neurons. The inset on the right-hand side of Fig. 5.1 shows how the Bayesian spiking neuron D is realised using a neuron tile.

Flipped with respect to the arrays presented in chapter 3, each row of the tile corresponds to one presynaptic distribution. The synapse g 0 is labelled in a blue rectangle as an example. Each of the columns is one sample of the spiking neuron D. The joint activation of all of the spiking neurons in the tile is the output distribution of each tile -indicated by the green rectangular box in Fig. 5.1. The output distribution of each tile-based Bayesian spiking neuron, p(D), could correspond to a distribution of spike-times, average rates or some other collective property of the circuit models in that neuron tile. By using the mosaic here as a substrate for a spiking Bayesian neural network, the sequential row-wise reading strategy required to perform inference using the mutli-layer Bayesian neural network hardware presented in section 3.3 can be avoided. Instead the stored samples will be read in an asynchronous, parallelised, fashion -potentially reducing inference latency and energy. This proposed Bayesian neuromorphic computing system can be equipped with circuits, like those designed in section 3.2, to train the conductance parameters of each column through in-situ RRAM-based MCMC sampling. This would therefore enable it to adapt and learn locally, on-chip. The interpretability of the bio-inspired architecture and the uncertainty encapsulated in its output prediction distributions, that of neuron D in Fig. 5.1 for example, also facilitates the responsible and ethical application of such a system to safety-critical edge learning applications.

The interactions and co-dependencies our these four key elements -resistive memory, Bayesian machine learning, bio-inspiration and spiking neurons -are summarised in the directed graph depicted in Fig. 5.2. Each arrow denotes how one aspect relies on the other. At the point of entry to this graph, there is the core principle that runs thought this thesis that RRAM arrays could allow dot-product operations to be implemented with a high efficiency, owing to device non-volatility. Secondly, if we then want to use an RRAM-based hardware to learn at the edge, then to do so we must resort to Bayesian machine learning; because it is the only framework based on the manipulation of random variables that can be used Chapter 6 Appendices

Appendix -Additional circuits and layouts

The circuit schematics and layouts for the other circuits and layouts not discussed or presented in the main text. The thin-oxide transistors have the dimensions: PMOS width=1.2µm, length=250nm. NMOS width=650nm, length=250nm unless otherwise stated. The thick-oxide transistors have the dimensions: PMOS width=3.3µm, length=500nm. NMOS width=2.07µm, length=500nm unless otherwise stated. All transistor bulks connect to vdd is they are a PMOS and gnd if they are an NMOS through taps at the top and bottom of each structure. A bar symbol and ground symbol are used to refer to vdd and gnd which are connected to 1.2V and 0V supply voltages respectively unless otherwise specified for the thin-oxide transistors and 4.8V and 0V for the thick-oxide transistors. Each layout figure is the layout image as it exists in the Cadence Layout XL that was used to perform the layout. Layers pertaining to polysilicon, metal, transistor active regions, nwells, and n+ and p+ doped regions are shown with different colours.

Starved inverter

Operational amplifier

Half OR

V out

Figure 6.6: (a) The circuit for the half OR circuit that is used at the input of the DPI-synapse circuit as well as in the delta-modulator circuit. It is called 'half' OR since there is no PMOS complement to the logic of the NMOS transistors. This is because, for an OR gate of N inputs, it would be required to stack N PMOS in series -and they would therefore not be saturated for a large number of devices. Instead a single PMOS (M 1 ) is placed between the inverter input noe and vdd and has its gate tied to gnd -such that it is always conducting. Each of the NMOS transistors is designed with fingers each such that, when an input signal arrives, they are able to sink more current than the PMOS can source (even though they are both turned on) and pull the input node of the inverter down to 0V. (b) The layout of the half OR circuit.

Hybrid neuron layout

(a)

Figure 6.15: The hybrid neuron circuit containing two NMOS 1T1R devices and two rows of multiplexing circuits to switch these structures in and out of the neuron circuit which is found in the top left corner of the layout. The MOS capacitor circuits are not shown.)

Appendix -Note on MOS capacitors

The hybrid CMOS-RRAM process used to design and layout the circuits of this thesis consists of a standard 130nm CMOS layer which is processed as normal up to metal3, finished off by depositing resistive memory device stacks between metal4 and metal5. Normally the process should have capacitors available between metal5a and metal6 but in this case they are not added. Therefore, in order to obtain capacitors it is required to use either metal-insulator-metal capacitors, realised with large overlapping metal layers, or MOS capacitors. MOS capacitors are more practical since they can be realised by shorting the source, drain and bulk of a MOS transistors. The transistor can then be sized to give a desired capacitance. However, MOS capacitance is highly non-linear as a function of the applied bias, V gs . For a V gs lower than the threshold voltage of the transistor, the top and bottom capacitor plates are the gate electrode and the bulk of the transistor. However, when V gs exceeds this threshold, the channel of the transistor begins to conduct and acts as the new dominant capacitor plate -reducing the effective plate separation distance considerably. Therefore as V gs is increased, and channel conduction becomes more dominant, the capacitor increases non-linearly as seen in the green curve Fig. 6.16a obtained through SPICE simulation of an NMOS capacitor. As V gs is made increasingly negative a similar effect is also seen in the orange curve of Fig. 6.16a, most likely due to the accumulation of minority carriers in the channel. A number of curves are shown as the MOS area is increased. The effect of MOS capacitor area on the capacitance value is summarised in Fig. 6.16b under two V gs biases. The relationship between MOSCAP area and the resulting capacitance value under two bias conditions (green V gs = 0.25V and blue V gs = 1.2V) for an NMOS device.

The SPICE simulations consisted of applying a DC bias (V gs ) over a NMOS transistor with a shorted drain, source and bulk and then injected a current pulse of known magnitude onto the capacitor. Using the equation; can also run C++ programs that allow the system to act as a computer-in-the-loop with the 16k device array as depicted in Fig. 6.17a.

Appendix -Resistive memory experiments

Before either chip can be used, it is required to form all the devices in the array. In the forming process oxygen vacancies are introduced into the HfO 2 thin-film through a voltage-induced dielectric breakdown. This is achieved by selecting devices in the array one at a time, in raster scan fashion, and applying a voltage between the source and bit lines. At the same time, the current is limited to the order of µAs by simultaneously applying an appropriate V WL (transistor gate) voltage. A form operation consists of the following conditions; 4k device array -V SL =4V, V BL =0V, V WL =0.85V, 16k device array -V SL =4V, V BL =0V, V WL =1.3V. After the devices have been formed, they are conditioned by cycling each device in the array between the LCS and the HCS one hundred times. Unless otherwise specified, the standard RESET conditions throughout the paper were; 4k device array -V SL =0V, V BL =2.5V, V WL =3V, 16k device array -V SL = 0V, V BL = 2.6V, V WL =4.0V. Unless otherwise specified, the standard SET conditions used were; 4k device array -V SL =2V, V BL =0V, V WL =1.2V, 16k device array -V SL =2.0V, V BL =0V, V WL =1.6V.

The device conductances are determined by measuring the voltage drop over a known low-side shunt resistance connected in series with the selected SL in a read operation. Devices are read according to the following conditions; 4k device array -V SL =0.1V, V BL =0V, V WL =4.8V, 16k device array -V SL =0.4V, V BL =0V, V WL =4.0V. All off-chip generated voltage pulses for programming and reading have a pulse-width of 1µs. In this fashion, we can implement RRAM-based algorithms, whereby programming operations can be determined based on the feedback of the result of the previous programming steps. In the case of the RRAM-based MCMC sampling algorithm implemented in the paper, after reading programmed device conductance values, the computer is then responsible for calculating the dot-product between the data points and the conductance model, evaluating the likelihood of the conductance model and finally performing the acceptance ratio calculation. The ultimate aim of such a system is to experimentally verify RRAM-based algorithms, before a full system can be designed and integrated on a standalone chip.

Appendix -Implementation of RRAM-based MCMC simulator

To characterise properties of the HfO 2 physical random variable, as plotted in the second chapter, the 4k device array chip was used. This allows use of a larger selector transistor which offers a greater range of the SET programming currents. A 4k device array was formed, conditioned and then RESET/SET cycled

Appendix -Pre-processing of the electrocardiogram dataset

For certain demonstrations in the second and third chapter addressing arrhythmic heartbeat detection, the MIT-BIH heart arrhythmia database [START_REF] George | The impact of the mit-bih arrhythmia database[END_REF], composed of half hour dual-channel electrocardiogram recordings from 48 subjects, was downloaded. Single heartbeats were isolated in each recording into a 700 ms time-series and centred on the R-wave peak of the beat. The fast Fourier transform function from NumPy was then used to generate a frequency spectrum of each time-series. In the first section of chapter two, the ten lowest frequency components were used as the input features for the RRAM-based Bayesian neural network that was trained experimentally using RRAM-based MCMC sampling. Each heartbeat data-point was then either labelled as either a normal, healthy, heartbeat or as a heartbeat exhibiting signs of arrhythmia. The dataset contains observations of tens of arrhythmia types. A subset of 250 data-points were taken randomly from 47 of the subjects and then used to train the models. Then, to test the models, all of the data-points from one, previously unseen, subject were used -subject 200.

In the second section of chapter two, the dataset was used again to demonstrate that an ex-situ trained RRAM-based Bayesian neural network could say 'I don't know'. In this case, normal heartbeats and the four most classes of arrhythmia were used -left and right blocked branches, atrial premature beats and premature ventricular contractions [START_REF] George | The impact of the mit-bih arrhythmia database[END_REF]. The sixty-four 'best' frequency components over the full dataset were selected using the Chi2 best-K feature selection algorithm [START_REF] Liu | Chi2: feature selection and discretization of numeric attributes[END_REF]. Four hundred examples of each heartbeat were randomly sampled from all forty-eight patients allowing a test and training set of one thousand data points each.

In the final section of chapter three, delta-modulated time-series were generated using all of the heart-