Introduction

• In deep learning, the artificial neurons communicate each other using analog signals. But the neurons in brain using eletric spike.
• The back propagation (BP) gives excellent performance to deep learning. However, so far there is no evidence that the BP mechanism exists in the biological brain.
• In deep learning, a large number of labeled data and training are of great importance. However, for biological brain, we didn't need so much data when we learn to classfy cats and dogs.
• When inference, a huge amount of energy is needed by deep learning. However, the power of a human brain is only equivalent to a 20W light bulb.

What is SNN? A brief view

The basic conception

Spiking neural network is evolved from the potential action characteristics of biological neurons. Inspired by brain circuits, each neuron in an SNN model updates the membrane potential based on its memorized state and current inputs, and fires a spike when the membrane potential crosses a threshold. The spiking neurons communicate with each other using binary spike events rather than continuous activations in ANN. It is particularly noteworthy that SNN introduces a spatiotemporal mechanism, and only special neurons such as RNN and LSTM can introduce the time dimension in ANN [1]. The binary spike communication mode and rich spatiotemporal dynamics of SNN constitute the most basic difference from the current ANN [2]. Basic neuron model in (a) ANN and (b) SNN. Image source

The spike neuron models

Just as we learn wings from birds, but not feathers. The researchers simplified the simulation of biological neurons, and initialization and propagation of action potentials were preserved. Nowadays in the SNN community, leaky integrate and fire (LIF) neurons are widely used for computing. LIF model achieves a balance between preserving biological neuron characteristics and simplifying computation. In addition, there are a series of neuronal models, such as Izhikevich model, Hindmarsh–Rose model and Morris–Lecar model, which trade off biological characteristics and computational costs to different degrees.

How does SNN works? The training of SNN

Direct training

The direct training methods, such as spike timing dependent plasticity (STDP) and Hebbian theory, come from the conceptual modeling of biological mechanisms [4]. During STDP, if the input spike of a neuron tends to appear immediately before the output spike of the neuron on average, the specific input will become stronger. If, on average, the input spike tends to appear immediately after the output spike, the specific input will be slightly weaker. As for Hebbian, a widely recognized summary is "Cells that fire together wire together." These learning mechanisms, which can explain biological concepts, have many praiseworthy advantages except that neural networks trained with these learning methods perform poorly.

• Image source*

Helps from deep learning

Another idea is to improve SNN, such as approxiamate spike, so that BP can be applied. The reason why BP is difficult to apply is that binary spike is non differentiable. So that, what about just use an approxiamate differentiable spike? When we approximate the derivative of spike activity, the BP could be used and it's convenient thanks to the automatic derivation of Pytorch. The deep learning based training methods achieve a balance between ease of use and performance [6]. Derivative approximation of the non-differentiable spike activity. (a) Step activation function. (b) Several typical curves to approximate the derivative of spike activity. Image source

Why choose SNN? Discussion of advantages and disadvantages

Advantages

• The rich spatial-temporal feature. SNN has a time dimension, so it is conducive to handling dynamic tasks such as SLAM, speech and dynamic vision tasks.
• Hardware friendly. Since the neurons communicate with each other by binary spikes, the multiplication operation degenerates into addition operation. For example, in ANN model, a feedforward of neurons could be write as : $$Y=w_1\ast x_1+w_2\ast x_2+...+w_n\ast x_n$$$$Y=w_1\ast x_1+w_2\ast x_2+...+w_n\ast x_n$$Y=w_(1)**x_(1)+w_(2)**x_(2)+...+w_(n)**x_(n)Y = w_1*x_1+w_2*x_2+ ...+ w_n*x_n in which $Y$$Y$YY is the output, $w$$w$ww is the synaptic weight and $x$$x$xx is the input. However, in SNN model, $x$$x$xx here are binary spikes, so that in one case, the feedforward could be transfered into: $$Y=w_1+w_2+...+w_n$$$$Y=w_1+w_2+...+w_n$$Y=w_(1)+w_(2)+...+w_(n)Y = w_1+w_2+...+w_nAs we all know, the addition operation is much more hardware friendly than the multiplication. - Energy conservation. Only neurons whose membrane potential reaches the threshold will work and issue spikes, so only a small part of neurons in the SNN model work at any time. Therefore, the energy consumption is low. But in ANN models, all the neurons work together at anytime. Imagine that if neurons in your brain work like ANN all the time, then you can eat and drink all day without gaining weight.

Disadvantages

Write behind

Reference