Wei Lu earns 2024 Best Paper Award for work in spiking neural networks

In September 2023, an interdisciplinary and international group of nine researchers came together to provide a tutorial on applying the decades of existing research in deep learning and related fields to spiking neural networks, with the ultimate goal of building more efficient neural networks. As the energy costs of today’s AI models skyrocket with no end in sight, the authors are looking to find new ways to improve AI performance with a dramatic reduction in energy consumption.

A year later, the paper, “Training Spiking Neural Networks Using Lessons From Deep Learning,” earned the 2024 Best Paper Award from the journal that published the work, Proceedings of the IEEE.

According to the authors, “The brain is the perfect place to look for inspiration to develop more efficient neural networks. The inner workings of our synapses and neurons provide a glimpse at what the future of deep learning might look like. This article serves as a tutorial and perspective showing how to apply the lessons learned from several decades of research in deep learning, gradient descent, backpropagation, and neuroscience to biologically plausible spiking neural networks (SNNs).” Companion tutorials are available using the Python package snnTorch.

First author on the paper is Jason Eshraghian, who worked on the paper as a postdoctoral researcher under the guidance of Wei Lu, the James R. Mellor Professor of Engineering. Eshraghian is now an Assistant Professor at the University of California, Santa Cruz.

“I want to emphasize that the work outlined in this paper is primarily the effort of Jason,” said Lu. “He’s truly becoming a star.”

We asked Wei Lu to describe his perspective on the research, which was conducted primarily during the COVID years.

What inspired this research?

Wei Lu: 

We all know that when people talk about AI, they really talk about deep neural networks (or machine learning, ML). In these deep neural networks, only the basic idea or the shape of the network is inspired by the brain––but nothing else is very brain-like.  

But there’s another type of AI that is more inspired by the brain, and that’s called spiking neural networks.  Spiking neural networks offer a couple important advantages. They can be very fast, and they consume a lot less energy when performing certain tasks because the data is just represented by simple 0s and 1s. 

In spiking neural networks, the data is also very sparse. To make an analogy with the brain: when a neuron becomes active, it fires a spike. But neurons don’t fire very often, otherwise our brain will just overheat and explode. The reason we can survive with all these tens of billions of neurons is because they fire very infrequently. So we have a very small amount of data in a given time––that’s called sparsity.

Data is also represented in time in these systems. As a result, spiking neural networks are very good at processing so-called spatial-temporal information, and time becomes very important. Every input is treated as having some correlation with the previous input. 

On the contrary, conventional AI, like deep neural networks, deals with static inputs. Every input completely resets the neural network. 

In spiking neural networks, we want to incorporate more of the biological features we learn from neuroscience. But the problem is that we still don’t really understand the brain. So when people are developing these bio-inspired neural networks, their performance is actually not good. They may have very high energy efficiency, but when you try to use them in a real task, they perform much worse than the conventional AI.

This paper is trying to develop a better theory and better training methods to train the spiking neural networks by borrowing some of the techniques, and even theory, from deep neural networks.

But right now, the two communities don’t really talk to each other. 

We can actually learn from each other and borrow some of the AI techniques and apply them to the spiking neural networks and vice versa. 

The paper includes a lot of proofs and shows why some of the approaches work better, and that’s really important. It explains how some of the machine learning theory actually aligns with the neuroscience theory, while previously people have thought that the two contradict each other. 

How does this relate to your current work?

In my current work, we do two things. 

We have one effort where we just run the AI algorithms, the deep neural networks, using new hardware designs such as in-memory computing. That’s what my startup MemryX does. At MemryX, we are implementing existing deep learning algorithms on new, very efficient hardware.

Another part of my group is interested in developing better spiking neural networks, including the fundamental theory and their implementations. As discussed earlier, these networks can be even more efficient and faster, but they are less well understood than deep neural networks.  

The main idea is that we can come up with better overall theory and implementations by combining knowledge from both the fields of neuroscience and deep neural networks.

Almetric Attention Score for:

“Training Spiking Neural Networks Using Lessons From Deep Learning,” by Jason K. Eshraghian, Max Ward, Emre O. Neftci, Xinxin Wang, Gregor Lenz, Girish Dwivedi, Mohammed Bennamoun, Doo Seok Jeong, and Wei D. Lu. Proceedings of the IEEE, vol. 111, no. 9, pp. 1016-1054, Sept. 2023, doi: 10.1109/JPROC.2023.3308088