Dharmendra S. Modha

My Work and Thoughts.

Archives for 2016

Demo at Conference on Computer Vision and Pattern Recognition

By

Guest Blog by Arnon Amir, Brian Taba, and Timothy Melano

The Conference on Computer Vision and Pattern Recognition (CVPR) is widely considered as the preeminent conference for computer vision. This year the IBM Brain Inspired Computing team had the pleasure of demonstrating our latest technology at the CVPR 2016 Industry Expo, held in the air-conditioned conference halls of Caesars Palace, Las Vegas. The expo was co-located with academic poster presentations, which created an excellent opportunity for us to not only meet very interesting academics, but also to see the latest demos from other amazing companies, both large and small.

Demo booth

We too were excited to demonstrate our new Runtime API for TrueNorth. To showcase it, we connected an event-based vision sensor, the DVS128 (made by iniLabs), over USB to our NS1e board.

Hardware flow

We used our Eedn framework to train a convolutional neural network on hand and arm gestures collected from our team, including air-drums and air-guitar! This Eedn network was used to configure the TrueNorth chip on the NS1e board. Overall, the system received asynchronous pixel events from the DVS128 sensor and passed them to TrueNorth. A new classification was produced every one millisecond, or at 1000 classifications per second.

The reaction to the real-time gesture classifications was very positive and drew large crowds (and other hardware vendors ;). People were blown away by that fact that we were running a convnet in real-time at 1000 classifications per second while consuming only milliwatts of power. We invited anyone who was interested to come behind our table to play with the gesture recognition. With a little bit of adjustment, people were able to interact with TrueNorth and have their gestures recognized. To many in the audience, the entire concept of neuromorphic engineering was new. Their visit to our booth was a great opportunity to introduce them to the DVS128, a spiking sensor inspired by the human retina, and TrueNorth, a spiking neural network chip inspired by the human brain!

Gesture icons
A video can be seen here.

Previously, we have demonstrated that TrueNorth can perform greater than 1000 classifications per second on benchmark datasets. Therefore, the new Runtime API opens the interface to the NS1e board and the TrueNorth chip for many exciting real-time applications, processing complex data at very fast rates, yet consuming very low power.

We give special thanks to our teammates David Berg, Carmelo di Nolfo and Michael Debole for leading efforts to develop the Runtime API, to Jeff Mckinstry for performing the Eedn training, to Guillaume Garreau for his help with data preparation, and to the entire Brain Inspired Computing team for volunteering to create the training data set!

Gearing Up for 2016 Telluride Neuromorphic Cognition Engineering Workshop

By

Guest Blog by Andrew Cassidy and Rodrigo Alvarez-Icaza

Gearing up. We are preparing for the 2016 Telluride Neuromorphic Cognition Engineering Workshop,
in the Colorado mountain town. Beginning Sunday Jun 26th, this annual workshop brings together nearly 100 researchers from all around the world to investigate brain-inspired solutions to topics such as:

  • Decoding Multi-Modal Effects on Auditory Cognition
  • Spike-Based Cognition in Active Neuromorphic Systems
  • Neuromorphic Path Planning for Robots in a Disaster Response Scenario
  • Neuromorphic Tactile Sensing
  • Computational Neuroscience

IBM’s Brain-Inspired Computing Group is sending two researchers with an end-to-end hardware/software ecosystem for training neural networks to run, in realtime, on the 4096 core TrueNorth neurosynaptic processor. The Eedn (Energy-efficient deep neuromorphic network) training algorithm enables near state-of-the-art accuracy on a wide range of visual, auditory, and other sensory datasets. When run on TrueNorth, these networks can be run at between 25 and 275mW, achieving >6000 FPS/W performance.

We are bringing (Figures 1-3):

  • 16 NS1e boards (each with 1 TrueNorth neurosynaptic processor)
  • 1 server (with 4 Titan X GPUs) for training deep neuromorphic networks
  • and a bucket of cables.

Building on the successes from last year’s workshop, and leveraging the training material from Bootcamp,
our goal is to enable train, build, and run for workshop participants. Combined with real-time runtime infrastructure to connect input sensors and output actuators to/from the NS1e board, we have all of the tools in place to build low-power end-to-end mobile and embedded systems, to solve real-world cognitive problems.

NS1e

Figure 1. Sixteen NS1e Boards

Training Server

Figure 2. Training Server and Gear

Prep Station

Figure 3. Prep Station

Photo Credits: Rodrigo Alvarez-Icaza

PREPRINT: Structured Convolution Matrices for Energy-efficient Deep learning

By

Guest Blog by Rathinakumar Appuswamy

To seek feedback from fellow scientists, my colleagues and I are very excited to share a preprint with the community.

Title: Structured Convolution Matrices for Energy-efficient Deep learning

Authors: Rathinakumar Appuswamy, Tapan Nayak, John Arthur, Steven Esser, Paul Merolla, Jeffrey Mckinstry, Timothy Melano, Myron Flickner, Dharmendra S. Modha 

Extended Abstract: We derive a relationship between network representation in energy-efficient neuromorphic architectures and block Toplitz convolutional matrices. Inspired by this connection, we develop deep convolutional networks using a family of structured convolutional matrices and achieve state-of-the-art trade-off between energy efficiency and classification accuracy for well-known image recognition tasks. We also put forward a novel method to train binary convolutional networks by utilising an existing connection between noisy-rectified linear units and binary activations. We report a novel approach to train deep convolutional networks with structured kernels. Specifically, all the convolution kernels are generated by the commutative pairs of elements from the Symmetric group S4. This particular structure is inspired by the TrueNorth architecture and we use it to achieve an improved accuracy vs energy tradeoff than we had previously reported. Our work builds on the growing body of literature devoted to developing convolutional networks for low-precision hardware toward energy-efficient deep learning.

Link: http://arxiv.org/abs/1606.02407

PREPRINT: Deep neural networks are robust to weight binarization and other non-linear distortions

By

Guest Blog by Paul A. Merolla

To seek feedback from fellow scientists, my colleagues and I are very excited to share a preprint with the community.

Title: Deep neural networks are robust to weight binarization and other non-linear distortions

Authors: Paul A. Merolla, Rathinakumar Appuswamy, John V. Arthur, Steve K. Esser, Dharmendra S. Modha 

Abstract: Recent results show that deep neural networks achieve excellent performance even when, during training, weights are quantized and projected to a binary representation. Here, we show that this is just the tip of the iceberg: these same networks, during testing, also exhibit a remarkable robustness to distortions beyond quantization, including additive and multiplicative noise, and a class of non-linear projections where binarization is just a special case. To quantify this robustness, we show that one such network achieves 11% test error on CIFAR-10 even with 0.68 effective bits per weight. Furthermore, we find that a common training heuristic–namely, projecting quantized weights during backpropagation–can be altered (or even removed) and networks still achieve a base level of robustness during testing. Specifically, training with weight projections other than quantization also works, as does simply clipping the weights, both of which have never been reported before. We confirm our results for CIFAR-10 and ImageNet datasets. Finally, drawing from these ideas, we propose a stochastic projection rule that leads to a new state of the art network with 7.64% test error on CIFAR-10 using no data augmentation.

Link: http://arxiv.org/abs/1606.01981

May 23-26, 2016: Boot Camp Reunion

By

Last year, from August 3 to August 20, 2015, IBM Brain-inspired Computing Team held a 3-week Boot Camp. Now, almost 9 months after, we held a Boot Camp Reunion from May 23 to 26, 2016 that brought together 64 attendees.

It was incredible to see the results from attendees and gratifying to see them productive on the next-gen Ecosystem, and achieve state-of-the-art results, within a matter of hours.

The following are three perspectives from my colleagues, Ben G. Shaw, Hartmut E. Penner, Jeffrey L. Mckinstry, and Timothy Melano.Don’t miss the attendee comments at the bottom of this blog entry!

Developer Workshop
Photo Credit: William Risk

Line Separator

Guest Blog by Ben G. Shaw on Attendees.

We carefully selected and vetted all attendees.

The following institutions are return attendees:

  • Air Force Research Lab, Rome, NY and Dayton, Ohio
  • Arizona State University
  • Army Research Lab
  • Georgia Institute of Technology
  • Lawrence Berkeley National Lab
  • Lawrence Livermore National Lab
  • National University of Singapore
  • Naval Research Lab
  • Pennsylvania State University
  • Riverside Research
  • Rensselaer Polytechnic Institute
  • SRC
  • Syracuse University
  • Technology Services Corporation
  • University of California, Davis
  • University of California, Los Angeles
  • University of California, San Diego
  • University of California, Santa Cruz
  • University of Dayton
  • University of Pittsburgh
  • University of Tennessee, Knoxville
  • University of Ulm
  • University of Western Ontario
  • University of Wisconsin-Madison.

In addition, the following institutions are new attendees:

  • Department of Defense
  • Johns Hopkins University, Applied Physics Laboratory
  • Mathworks
  • MITRE Corporation
  • Oak Ridge National Laboratory
  • Pacific Northwest National Lab
  • RWTH Aachen & FZ Juelich / JARA
  • Sandia National Laboratories
  • Technical University of Munich
  • University of Florida
  • University of Notre Dame

Line Separator


Guest Blog by Hartmut E. Penner on Docker Infrastructure.

For the Bootcamp Reunion, we needed to create an environment where participants could use our latest release. From past experience, it was clear that even with the best possible installation instructions, it would not be possible to cope with the multitude of systems of the participants in the given time we had and we wanted to spend precious Reunion time exploring the new Eedn programming environment. And beside, the training required server software with specific high end GPUs to do efficient training in the short time.

To be able to support all this and minimize the installation time and effort necessary we decided to use the IBM SoftLayer Cloud to provide the system with the GPUs and Docker as a way to package the software. The SoftLayer Cloud provides so called Bare Metal Server with GPUs which gives the user full hardware access to the system all the way to a hardware console over a web interface. This Bare Metal Server we ordered came pre-installed with Ubuntu Linux 14.04.

Docker as a container technology allows to package an application with all it dependencies like runtime, system tools, system libraries and the code itself. Unlike virtualization technology, all instances share the same kernel and therefore have much lower resource consumption while still providing full isolation between them. Details are here.

For the setup of our software, we needed to have access to the GPU in a shared fashions to limit the amount of hardware. Fortunately, Nvidia provided a full version of a dockerized environment which could be used as a nucleus to build a specialized Docker container with all our Software and access to the GPUs from multiple container instances. The container image was based on the CentOS 7 version or the cuda:7.5-devel image, added with a VNC server, MATLAB, matconvnet and our TrueNorth software release. Each participant got an instance created with exactly the same software with only differences being: an user specific persistent storage, ssh port and public authorized SSH key. With that each user was able to connect to their instance and upload data, from any kind of Windows, MacOS or Linux laptop. VNC traffic was tunneled over SSH, ensuring authentication and encryption of all traffic between the local user and the Cloud instance. For the uploading of a model and running test data on NS1e we chose HTTPS protocol between the cloud instances and the local gateway server, ensuring that this traffic also could not be compromised.

The installation we had finally was built on 7 Bare Metal Servers, each with two NVIDIA K80 GPUs and 16 NS1e boards. On those systems we had up to 80 simultaneous users accessing the instances over WIFI, with parallel training and testing on the NS1e board. The only problem with this installation was the public / private key handling with tunneling, due to its different handling between Windows and Linux/MacOS laptops. After passing this hurdle the installation ran very smoothly without any major issues and provided an environment to concentrate on the task of learning the new software and not dealing with installation and incompatibilities.

It was beautiful to see 80 people simultaneously using the infrastructure!

Line Separator


Guest Blog by Jeffrey L. Mckinstry and Timothy Melano on Developer Workshop.

On first day, May 23, IBM Team presented new results on convolution networks, NS1e16 system, and NS16e system.  This was followed by exciting new results from many of the participants — there are now 16 publications from Boot Campers!

From May 24-26, we ran a hands-on Developer’s Workshop in which the attendees were able to use for the first time our Energy efficient Deep Neuromorphic (Eedn) networks on their own data and run these networks on TrueNorth. The reunion had a variety of sessions ranging from theoretical deep dives into the mathematical abstractions of how we have mapped convolutional networks and backpropagation onto TrueNorth, to hands-on tutorial sessions. After a few hours of guided Eedn exercises, we took the training
wheels off and let the students use new datasets to create state-of-art neural networks. By the end of the day students had learned how to use the tools from our latest software release and were launching overnight training runs.

The next morning, May 25, students were excited to share very competitive scores on their datasets. The energy was great. Our guests from the Air Force Research Lab were our first success of many. On a aerial radar dataset their scores were comparable to the performance they were getting on other unconstrained networks, the difference in this case was that they were classifying image at 1000 frames per second, while minuscule amount power!

Later in the day our friend, Diego Cantor from the University of Western Ontario, was getting better results on an Eedn network than on his unconstrained Caffe networks! His data story is actually quite entertaining because his original image sizes were too big to be efficiently fed into a single TrueNorth chip. He was quite skeptical when we asked him to downsample his images to 32×32, but was later shocked to see that with smaller images and a sparse Eedn Convnet he was able to beat his Caffe convnet, a hat tip to brain-inspired computing indeed. His data is of ultrasound images of the human spinal column.

Developer Workshop

There were lots of other successes such as Tobi Brosch achieving 87 percent on an 14 category action recognition dataset and Garrick Orchard achieving 98.7 percent accuracy on spiking MNIST dataset;
but the real success was that all of the researches in attendance were training Eedn networks and visualizing high accuracy results very quickly.

Developer Workshop

Line Separator


Some Attendee Comments.

“I can make networks in a day now :)”

“I truly appreciate all of the effort from the IBM staff to put these events together; they are immesurably beneficial for this community. Not only do we gain technical insight, but we have the opportunity to reconnect as a community. I thought the use of docker was inspired. It really simplified the process of baselining all participants without the need for complex, time consuming installations. I plan to emulate this process for my lab server. I loved having the bootcamp reunion at Almaden. What a truly beautiful and inspiring location to work and gather as a community.
…
What I like most of all is how welcoming IBM is to all of us. I think I speak for this entire group when I say that you always make us feel like we are all part of something important.”

“The docker setup that provided an existing installation was genius. That helped us focus our time and attending on learning Eedn instead of the installation.”

“The reunion has been great – as was last year’s BootCamp. An excellent hands on tutorial for training and deploying CNNs on TN. I feel that I could train my own networks, and this is huge for me because I do not come from a formal machine learning background. I applaud the IBM team for putting together another great workshop.”

“The technology is impressive, from our perspective its ideal for robotic applications. We are excited to get this back to our lab and put this on our robotics platform to see how well this operates in our environments. … The reunion, the format, and the hosting has all been absolutely outstanding. It’s nice to have a lot of IBM folks around that have been responsive to all of our questions and concerns.  You all clearly spent a lot of time preparing tutorials, documentation, etc. and this preparation has helped and is well appprecaiated.”

“The fact that participants were able to get get up and running so quickly (and complained so little) reflects very positively on the tools and preparation.”

“I’ve really enjoyed the reunion. I wasn’t at the bootcamp so this is the first chance I’ve had to use the tools and found them very well constructed.”

“I found this training to be very helpful in understanding how to implement neural networks on the TrueNorth hardware. Contained in only 4 days I think the training had a good balance of theory, applications and hands-on projects. The use of Docker containers to implement MatConvNet was very easy to access and deploy and the background information sent prior to the bootcamp made it easy to get up to speed quickly. What other groups have done with the chip in such a short time was quite impressive.”

“Overall the presentations were layed out very well and were conducive to good information flow. The responsiveness of the team to help field any and all questions was well-receivied and invaluable; if someone did not know how to answer a question they actively sought someone on the team who did.”

“For a 3.5-day crash course on TrueNorth (compared to a 3-week long boot camp), this is a success and probably the best one can organize for a such short duration workshop. I would like to thank everyone at IBM for making it possible. I was able to work through the tutorials at my own pace, be immersed right away, and customize without starting from scratch.”

“It has been interesting and inspiring to see the breadth and extent of work that has been done by BootCamp participants, and to see how the tools have evolved since last summer.”

“Excited about the new convolutional network capabilties.”

“I want to start by saying thank you to all that put forth so much time and effort into this workshop.
I wish I could have been here last year at the first bootcamp, but you all have made it very easy to get up to speed. I like the structure and format at this workshop, it was well conceived and executed.”