A Comparative Study Between E-Neurons Mathematical model and Circuit model

The basic concepts and techniques involved in the development and analysis of mathematical models for individual neurons are reviewed. A spiking neuron model uses differential equations to represent various neuronal activities that have more compatibility with circuit criteria and are chosen for developing a comparative study with circuit models. For this comparison, a new fully differential neuron that uses the fully differential aspects to reach more balanced differential equations to mathematical model is presented. This comparative study of the circuit model and a neuron mathematical model provides a quantitative understanding of the challenges between mathematical models and microelectronic circuit design criteria.


| INTRODUCTION
Spiking neurons are the most plausible models of biological neurons because they accurately mimic the natural mechanisms of information processing and learning. Recently, extensive research towards novel realizations of neuronal models and computing paradigms as a complementary architecture to Von Neumann systems has been done using electronic techniques such as CMOS chips [1][2][3][4]. These publications have demonstrated that this technology is capable of impressive levels of interconnectivity and spike communication in neural-inspired circuits.
The key challenge in achieving a complete neuronal network similar to the human body is to implement a variety of neurons in electronic circuits to explore new paradigms for neuromorphic sensors and cortex neurons that are involved in brain-sensory perception. Most biologists agree with the classification of cortex neurons in six most fundamental classes of firing patterns observed in the mammalian neocortex [5]. The immediate applications of such neurons are an artificial vision [6] and audition [7] by mimicking the retina and the cochlea, respectively. Many efforts have been made to design and implement different types of neurons. However, fast-spiking (FS) [8,9], low thresholdspiking (LTS) [10] neurons have been made so far.
Choosing a special spiking neuron model for implementing different types of neurons has a significant impact on increasing the design speed like the works have been done in references [8,10]. This appropriate mathematical model that is capable of creating different kinds of neurons should have the flexibility to produce different waveforms by only some simple variation to be compatible with circuit design criteria. A spiking neuron model uses differential equations to represent various neuronal activities. Some of these activities can lead to the generation of an action potential, which is the charge in electrical potential (voltage) associated with a neuron. When a neuron reaches a certain threshold, it spikes, and the potential of the neuron resets. A popular simple neuron model is proposed by reference [5]; a hybrid spiking neuron model is introduced in reference [11], and a number of spiking neuron models are discussed in reference [12]. A spiking model based on logistic function using an analytical approach is presented in reference [13]. All of these models are developed for softwarebased computation and hardly could be used as a guideline for circuit design.
The mathematical model of reference [5] is supposed to be the base of the current comparative study. This selection is related to the ability of this model to produce different types of This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. neurons with only changing the values of some constants. This feature can provide the right conditions to be used as a reference for circuit design. Axon-Hillock (AH) neuron is considered as the base of neuron circuit implementation because of more similarity differential circuit equations with the equations of reference [5]. This comparative study between the circuit analytical model and the mathematical model of reference [5] provides a quantitative understanding of the challenges between mathematical models and microelectronic circuit design criteria.
To have a more symmetrical circuit equation to ones mentioned in reference [5] a refined new fully differential AH electronic neuron (e-neuron) is presented for the first time. Circuit equations are extracted to be used in a comparative study with a mathematical model.
This new implementation of AH e-neuron could double the output spikes with the same power budget and increase energy efficiency too. Doubling the output swing and mitigating the effects of temperature changes enables the power supply to be minimized and more power reduction could be achieved.
This paper is organized as follows. In Section 2 the required information of neurons models such as the developed mathematical model of reference [5] is explained. In Section 3 the novel AH neuron model, reasons for choosing AH structure, and fully differential structure are explained. The proposed fully differential AH neuron and its simulation results are shown in this section. The last part of this section is focussed on the new proposed mathematical model of the fully differential neuron model. An adaptive comparison between neuron mathematical model of reference [5] and the proposed model is presented in Section 4. A comparative study between the mathematical model and circuit model and certain issues related to developing a mathematical model compatible with circuit design are given in Section 5. Finally, conclusions are drawn in Section 6.

| BACKGROUND
A biological neuron model, also known as a spiking neuron model, is a mathematical description of the properties of certain cells in the nervous system that generate sharp electrical potentials across their cell membrane, roughly one millisecond in duration. Here a brief overview on biological neuron models is presented. According to our comparative study, a suitable mathematical model is explained too.

| Biological neurons
Two factors for characterization of each spiking neuron model are so critical: 1. Biologically plausible: this means a neuron spiking model can produce a set of firing patterns or behaviours exhibited by real biological neurons or not; 2. Computational efficiency: this factor shows the complexity of a neuron model and can be classified into five categories Very Low, Low, Medium, High and Very High. This factor is related to the number of floating-point operations needed to accomplish one millisecond (ms) of model simulation and the number of variables used in order to represent the neuron model (activation function).
Various mathematical models for biological neurons have been developed to represent their biological activities. As it is generally believed that neurons communicate with each other via action potentials, these models basically represent neuronal behaviour in terms of membrane potential and action potential. Some most popular models are Hodgkin-Huxley (HH), integrate-and-fire (I&F), FitzHugh-Nagumo (FHN), Morris-Lecar (ML), Wilson, Izhikevich, Hindmarsh-Rose (HR) [14].
These neuron models represent some or all of the characteristics of the responses of real neurons. The exact description of all these models is beyond the scope of this article, but in a simple comparison (Table 1), the Integrate-and-Fire model is the lowest model in consumption of computational power; which it could be used in a simple simulation that accuracy is not an important manner. While the HH model exhibit all neural behaviours, which could be used in applications where every single detail is needed, but this model requires very huge computational power. Izhikevich model exhibits most of the neural behaviours and does not require huge computational power, which it is the best model that could be used in any simulation or implementation of spiking neural networks, for example hippocampus simulation, classification or solving engineering problems [14].
According to the given explanations, choosing a simple mathematical model that can use the simplicity of Integrateand-Fire model and the accuracy of HH model at the same time will certainly help to implement the circuit of an e-neuron.

| Mathematical model
In reference [5], a model is presented that reproduces spiking and bursting behaviour of known types of cortical neurons. The model combines the biological plausibility of HH-type dynamics and the computational efficiency of integrate-and-fire neurons. Depending on four parameters, the model reproduces spiking and bursting behaviour of known types of cortical neurons. The model represented by two differential Equations (1) and (2), where Equation (3) is used to adjust membrane voltage v and the recovery variable u, as following: The v represents the membrane potential of the neuron, and u represents a membrane recovery variable, which accounts for the activation of K + ionic currents and inactivation of Na + ionic currents, and it provides negative feedback to v. After the spike reaches its apex (+30 mV), the membrane voltage and the recovery variable are reset according to Equation (3). Synaptic currents or injected DC currents are delivered via the variable I.
The parameters a, b, c and d describe the time scale of the recovery variable u, sensitivity of the recovery variable u to the subthreshold fluctuations of the membrane potential v, afterspike reset value of the v and after-spike reset of the recovery variable u, respectively.

| NOVEL E-NEURON MODEL
If a mathematical model requires to be used in circuit design, its different parts and mechanisms for making output waveform should be clarified. This clarification could help circuit designers to design a special circuit to implement its equilibrium mechanism in mathematics. Chosen mathematical models in Equations (1)-(3) have two main variables and two independent mechanisms for producing the different kinds of neuron spikes.
Equations (1) and (2) represent the first mechanisms. From circuit point of view variables v and u are the output voltage and its control voltage respectively. Solving these two equations leads to rising part of the output voltage until its apex point.
Equation (3) implements the jumping down mechanism which is completely independent of the first part Equations (1) and (2). Effective parameters in Equation (3) are c and d which determine the jumping steps. Figure 1 shows the output and controlling voltages. As it can be seen these two voltages are in phase in the rising part and only in the jumping part, they jump in the opposite direction. According to Figure 1, two points need to be considered in order to create a comparable circuit model.
1. Select a specific circuit implementation that has the same model voltages as the signals obtained from the mathematical equations.Design a circuit which has in phase output and controlling voltage.

| AH neuron
In order to choose the proper circuit implementation of the neuron, which has more similar voltages to ones shown in Figure 1, different types of implemented state-of-the-art eneurons analysed mathematically [8,10,[15][16][17]. Because AHneuron circuit has two main node voltages which determines the circuit behaviour and two critical differential equations, AH neuron has more compatibility with mathematical modelling of Equations (1)-(3) among all other e-neuron circuit implementations.
The original AH circuit is based on a simple voltage amplifier which most of the time implemented using two inverters cascaded in series and uses two capacitances [18,19]. Membrane capacitance and feedback capacitance.
In reference [9], for achieving extremely low DC power consumption and energy efficiency, a refined AH architecture was presented (see Figure 2). F I G U R E 1 Mathematical model waveforms and systematic parameter description [5] F I G U R E 2 Refined single ended AH artificial neuron [9] DALIRI ET AL.
-177 3.2 | Proposed fully differential AH circuit If in the refined AH architecture of Figure 2, V mem and V out are chosen as the output voltage and controlling voltage respectively, these two voltages could not be in-phase because V out derive the pull-down transistor MN 3 and V out should go high to pull down the V mem . Therefore, a single ended AH neuron could not be used for this comparative study. This problem could be solved by implementing the AH neuron differentially. Figure 3 shows the proposed fully differential AH artificial neuron circuit. The membrane potential, referred to ground, is denoted by V m1 and its differential counterpart is V m2 . Controlling voltages of the spike voltages V m1 and V m2 are represented as V ct1 and V ct2 , respectively.
The two inputs excitatory currents to the circuit are I ex1 and I ex2 which are directly coupled to the controlling voltages V ct1 and V ct2 , respectively. Feedback capacitances are completely symmetrical and are shown in Figure 3 by C f .
When excitatory (DC) current I ex1 and I ex2 are applied, parasitic capacitances of nodes ct1 and ct2 and feedback capacitances C f1 and C f2 are charged and discharged, respectively. When the magnitude of I ex is sufficient, the increment of V ct1 and decrement of V ct2 reach the switching voltage of NMOS transistor M n2 and PMOS transistor M p1 and both output voltages V m1 and V m2 states are changed. V m1 rises towards V dd , and vice versa, V m2 goes towards -V dd . Meanwhile, positive feedback occurs through C f , pulling up the control voltage V ct2 to a positive value enough for turning off the PMOS transistor M p1 , similarly on the opposite side M p3 pulling down the V ct1 and turning off the NMOS transistor M n2 . M n1 and M p2 are turning on quickly and the circuit goes back to its first state again. This procedure keeps repeating until excitatory currents are in the appropriate range which could charge and discharged the ct1 and ct2 equivalent capacitances respectively in comparison to the counterpart transistor.
V m1 (V m2 ) magnitude is limited by the voltage, which develops between drain and source of M p1 (M n2 ). Voltage waveforms V m1 , V ct1 of the proposed circuit are shown in Figure 4. Circuit parameters of FS neurons are presented in Table 2, C f = 5fF, I ex = 25pA. The FS neuron is designed using the BiCMOS SiGe 55 nm technology using low-power low-threshold voltage transistors to enable weak inversion (WI) region bias. The variation of spike frequency is plotted as a function of excitation current (I ex1 = I ex2 ) in Figure 5

| Differential circuit mathematical model
To reduce the power consumption of implemented e-neuron WI region is chosen for transistors. WI model of saturated MOS transistor is obtained as [20]: where Φ t is the thermal voltage kT/q, η is the slope factor 1 + C d /C ox (i.e. depletion C d and oxide C ox capacitance ratio) and I z is the specific current which has the following equation: where μ is the mobility and V Th is the threshold voltage of the transistor. According to the proposed schematic of the neuron (see Figure 3), Kirchhoff's circuit law (KCL) could be used for four main nodes of the circuits and four equivalent differential F I G U R E 3 The proposed fully differential AH artificial neuron 1.08 µm/55 nm equations are obtained. The proposed circuit consists of two sections whose outputs are 180°out of phase; therefore two output voltages are supposed to be completely differential. It leads to V m1 = −V m2 = V m and V ct1 = −V ct2 = V ct and I ex1 = I ex2 = I ex . Four differential equations are reduced to two.
These two equations are as follows: In the above equations, the symbols are explained as follows: C m is the membrane capacitance which equals to the parasitic components corresponds to the V m1 node capacitance, I ex is the excitatory current (I ex1 = I ex2 = I ex ), C p is parasitic capacitance of V ct1 node.
Substituting dV ct /dt from Equation (7) in Equation (6) the explicit differential equation of V m could be obtained as follows: |ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl {zffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl } Minimum excitation current value can be set by adjusting transistor M p3 dimensions, for more reduction of power consumption, less excitatory current and low aspect ratio of M p3 and M n3 are needed. According to mentioned explanation, I ex and M p3 currents are negligible in comparison to other parts of Equation (8) and it could be simplified as presented in: Now Equations (7) and (9) are the final differential equations modelling the proposed fully differential neuron. Because these two exponential differential equations don't have any intuitive responses, exponential responses are approximated by their equivalent Taylor expansions which third and higher order sentences are neglected. These assumptions simplify the differential equations as mentioned in Equations (10) and (11).
Coefficients A 1 -A 4 and B 1 -B 5 formulas are presented in Table 3. To check the validity of the above assumptions, the voltages V m and V ct are represented in differential Equations (10) and (11) are drawn in Figure 6. Comparing the voltages obtained from Equation (6) to Equation (7) and from Equation (10) to Equation (11) and the exact voltages obtained from simulations (bold line) in 55 nm CMOS technology, it is clear that exponential equations faithfully represent the actual neuron voltages and the only difference is the frequency of spiking which has less than 10% difference. Polynomial approximated equations have more differences with ones obtained from simulations, especially in pulling down phase, which will be discussed completely in the next section. Hence, we will use Equations (10) and (11) for the systematic analysis of the circuit in the next sections.

Symbol Expression
ct − B 5 · V ct · V m |ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl {zffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl ffl } Jumping mechanism # ð10Þ dV ct dt Solving both Equations (10) and (11) for two variables, V m , and V ct , can be found. However, solving these two strongly non-linear equations simultaneously in terms of circuit parameters is very complicated and the results would not convey any useful qualitative information about the behaviour of the output voltages. To overcome this problem, a comparative study with reference [5] is made in the next section to show how a systematic model of pure mathematical environment could be implemented in the electronic word and its challenges.

| E-NEURON AND MATHEMATICAL MODEL COMPARISON
For more compatibility between the presented model of reference [5] and the differential equations of the circuit, Equations (10) and (11) are rewritten as follows: According to Equations (12) and (13), V m , V ct , β and γ could be supposed to be instead of v, u, a and b in the mathematical model respectively (see Equations (1) and (2)). Jumping mechanism in Equation (12) is shown in Equation (10) and Table 2. Spike voltage could be divided in two phases: 1. rising phase until the spike reaches its apex, and 2. reset phase which the membrane voltage jumps down to its resting potential.
The first part of Equation (12) models the first phase and th e jumping mechanism is approximately mimicking the jumping down process. Substituting Equation (10) in Equation (11), coefficients β and γ could be obtained as below: δ in Equation (13) shows the other parts of differential equation and includes the I ex and other non-linear parts.
Based on the equations obtained in the previous sections we compare the Equation (12) with the mathematical model. A two-dimensional sweep for NMOS and PMOS widths are performed and the variation of αA 1 -αA 4 are computed and illustrated in Figure 7. As can be seen, all the coefficients are in the range of 10 −9 with different dimensions. Except for αA 3 , the absolute values of all other ones are increased as the size of PMOS increases. In the case of αA 1 increasing the PMOS sized cause to sign change and change it from a positive value to a negative one.
Equation (2) models the variations of variable u from peak value to the moment of jump upward, but in the proposed circuit a completely different process is followed to create the equivalent of this portion of the signal.
Referring to Figure 6, it is clear that, increasing V ct from its minimum value until the beginning of the jump process F I G U R E 6 Simulations results versus exponential and polynomial approximation of circuit differential equations. (a)V m (b) V ct is done at a constant rate, which is dependent on the ratio of the excitation current to the equivalent capacitance of the node. Going down of V ct is also done by M p3 , which is relatively faster than the variable u in Equation (2). Using interpolation techniques can certainly find a single function for modelling this part of the signal. Therefore, coefficient a and b are not the main factors in determining the V ct behaviour.

| DISCUSSION
Comparing circuit behaviour and mathematical models in Equations (1)-(3), it becomes clear that there are some serious challenges for implementing the mathematical equations in the electronics domain. These challenges have been attempted to be presented below in order to provide an overview, improving the compatibility of mathematical models and implementation criteria. Certainly, when implementation challenges are also taken into account, proposed models could be more applicable and usable in circuit design.
The model presented in reference [5] consists of two completely separate and independent sections: 1. Spike initiation dynamics are modelled by two ordinary differential equations. 2. Auxiliary after spike resetting or jumping system in Equation (3).
In reality, two independent systems for one waveform are impossible and surely, they will influence each other. As can be seen in Table 2, B 1 -B 4 are directly dependent on the size of NMOS and PMOS transistors. It shows that in each mathematical model the relation of these two parts should be clarified.
Sharp jumping down could be obtained only in the WI regime because of the exponential I-V characteristic of transistors. This difference between polynomial approximation and exponential relation is shown in Figure 6a. Polynomial approximation could model the rising part of waveform very well but the second part of jumping down is not as precise as exponential one. Due to power consumption limitation and mimicking the biological system, using WI transistors are inevitable. Using WI transistors cause the coefficients of the differential equation to be in the nanoscale because they have a direct relationship with the size of the transistor. It shows that in spike dynamic modelling, the range of coefficients should be considered.

| CONCLUSION
A comparative study between the circuit model and the mathematical model of reference [5] is developed. To find a more compatible mathematical neuron model with circuit design, this paper provides the right conditions to be used as a reference for circuit design. For this purpose, a new topology for a differential AH artificial neuron is designed, which doubles output spikes using fully differential structures. By comparing circuit behaviour and mathematical models, challenges for implementing the mathematical equations in the electronics domain were addressed.