Guest post by Peter U. Diehl from ETH Zurich. Peter’s research is focused on bringing together the fields of neuromorphic computing, machine learning and computational neuroscience.
At the Telluride Neuromorphic Cognition Engineering Workshop 2015 a team from IBM Research (Rodrigo Alvarez-Icaza, John Arthur, Andrew Cassidy, and Paul Merolla) brought their newly developed low-power neuromorphic TrueNorth chip to introduce this platform to a broader research community. Among the other participants were Guido Zarella, principal research scientist at the MITRE Corporation and an expert in natural language processing (NLP), Bruno Pedroni, PhD student at UCSD and previous intern at IBM Research Almaden, Emre Neftci, professor at UC Irvine and a pioneer on using deep learning with spiking neural networks, and myself. Together we pursued the ambitious goal of bringing deep learning based NLP to neuromorphic systems.
Driven by the ever increasing amounts of natural language text available on the world wide web and by the necessity to make sense of it, the field of NLP showed dramatic progress in recent years. Simultaneously, the field of neuromorphic computing has started to emerge. Neuromorphic systems are modeled after the brain, which leads to hardware that consumes orders of magnitude less power than its conventional counterpart. However, such a new architecture requires new algorithms since most of the existing ones are designed for von-Neumann architectures and usually cannot be mapped directly.
At Telluride, myself and the group mentioned above were eager to fill the existing algorithmic gap of NLP for neuromorphic computing by mapping existing state-of-the-art NLP systems for von-Neumann architectures to TrueNorth. Achieving this goal enables a range of highly attractive technologies like high-quality analysis of user input on mobile devices with negligible battery drain, or data-centers for understanding queries that consume orders of magnitude less power than conventional high-performance computers. During the workshop we focused on two tasks.
- The first task is sentiment analysis on TrueNorth, that is, predicting the “happiness” associated with the given words. Our system, called “TrueHappiness”, uses a fully-connected feedfoward neural network which is trained using backpropagation, and that is converted to a TrueNorth compatible network after the training finished.
- The second task is question classification, where we identify what kind of answer the user is looking for in a given question. Similarly to the design of TrueHappiness, we start by using deep learning techniques, that is, we train a recurrent neural network using backpropagation and convert it afterwards to a spiking neural network suitable for TrueNorth.
For both tasks we managed to implement end-to-end systems where the user can type words that are mentioned in Wikipedia that are then used for sentiment analysis or (in case of a question) are analyzed with regards to the desired content. Demos and details about the training and conversion of both systems will be soon available at peterudiehl.com.
Although the designed algorithms are not yet viable for commercial scale applications because we are just getting started, they provide an important first step and a generally applicable framework for the mapping of traditional deep learning systems to neuromorphic platforms and thereby opening up neuromorphic computing and deep learning for entirely new applications. This is also the vision at the IBM TrueNorth Boot Camp at IBM Research in San Jose, where I am at the moment. Together with over 60 other participants we are diving into TrueNorth programming, creating new neuromorphic algorithms and mapping existing von-Neumann architecture algorithms to TrueNorth to advance the state-of-the-art in low-power computing.