Archives for August 2015

Guest post by Peter U. Diehl from ETH Zurich. Peter’s research is focused on bringing together the fields of neuromorphic computing, machine learning and computational neuroscience.

At the Telluride Neuromorphic Cognition Engineering Workshop 2015 a team from IBM Research (Rodrigo Alvarez-Icaza, John Arthur, Andrew Cassidy, and Paul Merolla) brought their newly developed low-power neuromorphic TrueNorth chip to introduce this platform to a broader research community. Among the other participants were Guido Zarella, principal research scientist at the MITRE Corporation and an expert in natural language processing (NLP), Bruno Pedroni, PhD student at UCSD and previous intern at IBM Research Almaden, Emre Neftci, professor at UC Irvine and a pioneer on using deep learning with spiking neural networks, and myself. Together we pursued the ambitious goal of bringing deep learning based NLP to neuromorphic systems.

Driven by the ever increasing amounts of natural language text available on the world wide web and by the necessity to make sense of it, the field of NLP showed dramatic progress in recent years. Simultaneously, the field of neuromorphic computing has started to emerge. Neuromorphic systems are modeled after the brain, which leads to hardware that consumes orders of magnitude less power than its conventional counterpart. However, such a new architecture requires new algorithms since most of the existing ones are designed for von-Neumann architectures and usually cannot be mapped directly.

At Telluride, myself and the group mentioned above were eager to fill the existing algorithmic gap of NLP for neuromorphic computing by mapping existing state-of-the-art NLP systems for von-Neumann architectures to TrueNorth. Achieving this goal enables a range of highly attractive technologies like high-quality analysis of user input on mobile devices with negligible battery drain, or data-centers for understanding queries that consume orders of magnitude less power than conventional high-performance computers. During the workshop we focused on two tasks.

• The first task is sentiment analysis on TrueNorth, that is, predicting the “happiness” associated with the given words. Our system, called “TrueHappiness”, uses a fully-connected feedfoward neural network which is trained using backpropagation, and that is converted to a TrueNorth compatible network after the training finished.
• The second task is question classification, where we identify what kind of answer the user is looking for in a given question. Similarly to the design of TrueHappiness, we start by using deep learning techniques, that is, we train a recurrent neural network using backpropagation and convert it afterwards to a spiking neural network suitable for TrueNorth.

For both tasks we managed to implement end-to-end systems where the user can type words that are mentioned in Wikipedia that are then used for sentiment analysis or (in case of a question) are analyzed with regards to the desired content. Demos and details about the training and conversion of both systems will be soon available at peterudiehl.com.

Although the designed algorithms are not yet viable for commercial scale applications because we are just getting started, they provide an important first step and a generally applicable framework for the mapping of traditional deep learning systems to neuromorphic platforms and thereby opening up neuromorphic computing and deep learning for entirely new applications. This is also the vision at the IBM TrueNorth Boot Camp at IBM Research in San Jose, where I am at the moment. Together with over 60 other participants we are diving into TrueNorth programming, creating new neuromorphic algorithms and mapping existing von-Neumann architecture algorithms to TrueNorth to advance the state-of-the-art in low-power computing.

Guest Post by Michael Debole with contributions by Rodrigo Alvarez-Icaza, Filipp Akopyan, Nabil Imam, Bryan Jackson, William Risk, and Jun Sawada.

Today, we end our first week of a 3-week TrueNorth bootcamp, where we began to unveil and teach the ecosystem for TrueNorth. We take a look behind the scenes and present some of the effort the team has put forth when preparing the hardware for participants to learn on, play with, and eventually call their own.

The goal was simple: Assemble, test, and configure the first batch of 100 single-chip, index card-sized TrueNorth development boards (dubbed NS1e for Neurosynaptic System Evaluation Platform), 48 of which will be used as the playground for the Boot Camp participants. First to learn on, but eventually to take home and continue to develop their own algorithms and apply to their own applications. While seemingly straightforward, several items had to come together to allow us to provide users with an experience that has become expected of a development platform. While there was much to do, boards from the factory were received almost exactly one month to the day (actually less), leaving very little room for errors. Here is a timeline that summarizes our experiences as we worked to deliver the hardware in time for bootcamp:

07/01/15-07/06/15: NS1e boards are in-route from the factory, but there is little time to waste. We begin to create manufacturing and test scripts responsible for exercising board components as well as tracking boards as they execute the test procedures.

07/08/15: Boards are here! Less than one month away from bootcamp, we received NS1e boards in Almaden for assembly, test, and staging.

07/09/15: It was all hands on deck as we began to assemble and test the boards, attaching feet and heat sinks, everyone got involved in the fun! After boards were assembled, they were run through a round of testing which exercised TrueNorth and other discrete components to ensure functionality.

Bryan Jackson assembling and testing NS1e True North development boards.

7/9/15: Due to the hardwork of the team and lab members, all boards were tested and configured by the end of the day!

Assembled NS1e Boards.

07/10/15-07/24/15: With two weeks before we begin infrastructure testing the team still needed to decide how the NS1e boards would be finally configured. One important choice was which operating system would be loaded on the embedded Zynq platform: whether to use the strongly tested Petalinux or to push to provide Ubuntu. The team collectively decided to ensure the boards could run Ubuntu, which enables the NS1e platforms to pick up applications from the Linux community at large and expands the platforms versatility. To enable this, the team worked to resolve a few compatibility issues, but finally managed to get a working and tested image in time for infrastructure testing.

07/24/15-7/30/15: Infrastructure testing week! Due to the number of participants we split our 48 boards across 4 gateway servers. Testing now consists of ensuring each board is accessible by the servers and each has the latest version of the image containing FPGA firmware, OS, and the TrueNorth control software. S ome load testing is also performed to see what happens when many users are requesting to use boards simultaneously. Things mostly worked, and we fixed what didn’t.

Staging area for the 48 NS1e Development Boards the Participants will use while at Boot Camp.

07/31/2015: It is the weekend before bootcamp, so the team builds the final home of the NS1e development boards before they are finally transported to the classroom. Each of the NS1e platforms are placed into individual cases and inserted into a 6×8 array. The final rack contains the same number of neurons as a small mouse (48 Million Neurons and 12 Billion Synapses)! Now off to bootcamp!

Today, August 3, 2015 at 9am PST, we launched a three-week-long Boot Camp. Read my thoughts here. Representing five continents, sixty+ participants from the following institutions are attending including IBMers from IBM Research – Australia, IBM Research – Tokyo Lab, IBM Brazil, and IBM UK:

• Air Force Research Lab, Rome, NY
• Argonne National Lab
• Arizona State University
• Army Research Lab
• California Institute of Technology
• Cornell University
• Johns Hopkins University
• Imperial College, London
• Institute of Neuroinformatics, ETH Zurich
• Lawrence Berkeley National Lab
• Lawrence Livermore National Lab
• National University of Singapore
• Naval Research Lab
• Pennsylvania State University
• Riverside Research
• Rensselaer Polytechnic Institute
• SRC
• Syracuse University
• Technology Services Corporation
• University of California, Davis
• University of California, Los Angeles
• University of California, San Diego
• University of California, Santa Cruz
• University of Dayton
• University of Pittsburgh
• University of Tennessee, Knoxville
• University of Western Ontario
• University of Wisconsin-Madison

In addition, in attendance are research interns at IBM Research – Almaden from:

• MIT
• Pennsylvania State University
• University of California, Irvine
• University of Geneva
• University of Heidelberg
• University of California, San Diego
• University of Ulm

Full House!

Here is a rough transcript of my opening remarks.

History

When history looks back and asks “When did Brain-inspired Computing reach critical mass?”,
I believe the answer will be 9am PST on August 3, 2015.

Welcome

Thank you for investing three precious weeks of your life, we will strive to make it worth it.
Special welcome to Telluride participants who have come back for another 3 weeks.
All of you are a very special group of people —
incredibly talented, excited, and engaged researchers who have the right background.
Because of the space limitations, many could not be admitted.
Today is the first time that IBM is opening the SyNAPSE Ecosystem.
In fact, you are getting access to these boards before our own team.
According to the dictionary, to pioneer is the one to “develop or be the first to use or apply (a new method, area of knowledge, or activity)”.
So all of you are pioneers.
It takes two hands to clap and and you are our partner in this quest to bring brain-inspired computers to the society.

Where are we?

With 1 million neurons,
256 million synapses,
4096 cores interconnected via a network-on-chip
at less than 100mW of power consumption,
TrueNorth is literally a supercomputer the size of a postage stamp consuming the power of a hearing aid battery.

The chip has an entirely novel parallel, distributed, modular, scalable, fault-tolerant, event-driven architecture that breaks path with nearly 70 year dominance of the von Neumann blueprint.
Today, the effort is much broader than the chip.
It is an end-to-end ecosystem consisting of development boards;
a simulator;
a programming language;
an integrated programming environment;
a library of algorithms as well as applications;
firmware;
deep learning tools;
a teaching curriculum;
and cloud enablement.

We got here via a 10 year sustained effort with cumulative effort of 250 person years.

Where we are headed is ever more useable and useful ecosystem
and what is forthcoming is a sequence of ever-denser and energy-efficient chips with the end goal to literally build a brain-in-a-shoebox.

What is the best possible outcome?

The best possible outcome
is to map the entirety of existing cache of neural network algorithms and applications to this energy-efficient substrate.
And, to invent entirely new algorithms that were hereto before impossible to imagine. Then, to compose these algorithms to exhibit mind-like behavior on top of the brain-like substrate.

What is the challenge?

You have all heard about deep learning.
Deep learning and neuromorphic computing are two sides of the same coin.
Deep Learning is about capability in terms of application performance and accuracy.
Brain-inspired Computing Is about energy-efficiency, volume-efficiency, speed-efficiency, and scalability.
Deep Learning is used to program neuromorphic computing.
Neuromorphic computing used to deliver deep learning.
The goal is to bring together these two complimentary revolutions.

To exploit the full potential of the substrate, the challenge is to shift the thinking.
Energy-efficiency was realized by architecture shift to
spiking neurons,
limited precision synapses, and
connectivity within cores and between cores.

The reason it is possible to shift the thinking is because
brain itself is able to achieve its remarkable function with similar constraints and
because neural networks are remarkably forgiving structures.

The first thought of nearly everyone is to train a classical network, and as an afterthought map it to hardware.
This approach leads to inefficient and inelegant solutions.
Thinking has to shift from “train and constraint”
to a mindset of “constraint and train”.
The challenge is to design efficient algorithms from the start.
As a motivational example in historical context, let us go back to “sorting algorithms” for today’s computers, where
quadratic-time algorithms were upturned by Tony Hoare’s beautiful and efficient quicksort.
Tony Hoare was awarded the Turing Prize for his many contributions
and perhaps one of you, someday, might receive the same honor, perhaps
for Brain-inspired Computing.
Neural networks are remarkably versatile structures —
there are often multiple paths to the solution.
Amongst these, the challenge is to seek the most efficient and most elegant and most beautiful algorithms.
The opportunity for innovation is huge and unprecedented.
Supervised learning,
unsupervised learning,
reinforcement learning as well as
feedforward networks,
recurrent networks all are up for grabs.
The frontier is open.
And, it is yours to capitalize.
I predict that by December 2016 there will be at least one major breakthrough —
inventing a fresh new algorithm for neuromorphic computing — from one of you.
My hope is that each of you will be part of at least one such breakthrough.

What are we teaching at the Boot Camp?

We are teaching an overview of the ecosystem.
We are teaching end-to-end examples with simple datasets so as to completely and
comprehensively touch on the entire tool set. In particular, we are teaching how
we are leveraging Caffe (a deep learning tool) to produce native-TrueNorth programs.

In the final week, we will provide clinics and help from both infrastructure and
algorithmic perspectives to ensure that you are successful after the bootcamp which is really the whole point of this exercise.

How are we teaching it?

The approach is to teach the building blocks.
We are looking forward to seeing what you produce with them.
The possibilities are endless
because permutations and combinations of the underlying build blocks are limited only by your imagination and creativity,
which I know is infinite.

Boot Camp is about teaching the techniques of fishing, and not about catching one big, fat fish right away.
it is about bricks, and not about buildings.
It is end-to-end integration, and not fragmentation.
It is simple, and not complex.
It is practical, and not theoretical.
It is about breadth across tools and techniques, and not about depth in a particular application.
It is about ingredients and recipes and not tasty dishes.
It is atomic and not molecular.
It is sequential, where each step leads to the next level.
It is slow, and not fast.
It is flexible, and not a priori cast in stone.

Now, our team has worked very hard but much is to be done.
You are experiencing a living and growing set of tools and not something that is polished and dead.
We welcome all feedback, and will humbly accept it.

Conclusion

Sixty years ago, IBM released FORTRAN to the world.
At that time, there were only a few hundred or at best low thousands of programmers.
Today, we a team of 30 people,
feel that our ecosystem has tripled in size instantaneously by adding 60 new partners.
The hope is that each of you will become an ambassador for the project.
A seed carrier.
And, together we will draw in ever greater number of collaborators to create a spiral of value co-creation.

On our part, we are committed
to continue to provide systems and tools
to enable you to build more algorithms and applications.

This is a historic moment. Let us partner and together we can bring brain-inspired computers to society.

In conclusion, I invite the whole Brain-inspired Computing Team to join with me in welcoming our friends.

Finally, for all our guests, please join me in thanking the Brain-inspired Computing Team
for countless nights, weekends, and their blood, sweat, and tears in making this event a reality.

Thank you.