Author: snikolenko

Neuroplasticity is another part of this issue. Scientists conducted experiments demonstrating how different areas of the brain can easily learn to do things for which they’re seemingly not designed. Neurons are the same everywhere, but there are areas in the brain responsible for different things. There’s the Broca area responsible for speech, an area responsible for vision (actually, a lot of areas — vision is very important for humans), and so forth. Nevertheless, we can break down these notional biological borders.

This man is learning to see with his tongue. He attaches electrodes to his tongue, puts a camera on his forehead, and the camera streams an image on the electrodes pricking his tongue. People stick that thing on them and walk around with it for a few days, with their eyes open, naturally. The part of the brain that receives signals from the tongue starts to figure out what’s going on — this feels a lot like something that comes from my eyes. If you abuse somebody like that for a week and then blindfold him, he’ll actually be able to see with his tongue! He is now able recognize simple forms and doesn’t bump into walls.

The man in this photo has turned into a bat. He’s walking around blindfolded, using an ultrasonic scope whose signals reach his tactile neurons through his skin. With a sonar like this, a human being can develop echolocation abilities within a few days of training. We do not have a special organ that can discern ultrasound, so you have to attach a scope to your body. However, we can relatively easily learn to process this information, meaning that we can walk in the dark and not bump into any walls.

All of this shows that the brain can adapt to a very large number of different data sources. Hence, the brain probably has a “common algorithm” that can extract meaning from whatever it takes in. This common algorithm is the Holy Grail of modern artificial intelligence (a recent popular book on machine learning by Pedro Domingos was called The Master Algorithm). It appears as though deep learning is the closest we have come to the master algorithm of all the things done in the field up until now.

Naturally, one has to be cautious when making claims about whether all of this is like what the brain does. “Could a neuroscientist understand a microprocessor?”, a recent noteworthy article, tries to elucidate how effective current approaches in neurobiology are at analyzing a very simple “brain”, like a basic Apple I processor or Space Invaders on Atari. We will return to this game soon enough and won’t go into much detail about the results but we do recommend reading the paper. Spoiler alert: modern neurobiology couldn’t figure out a single thing about Space Invaders.

Feature extraction

Unstructured information (texts, pictures, music) is processed in the following way: there is raw input, then features that bear content take shape, and then classifiers are built based on those features. The most complicated part of this process is understanding how to pick good features out of unstructured input. Up until recently, systems for processing unstructured information have worked as follows: people have attempted to select good features manually and then assess the quality of relatively simple regressors and classifiers based on these features.

Take Mel Frequency Cepstral Coefficients (MFCC), which had been commonly used as features in speech recognition systems, for example. In 2000, the European Telecommunications Standards Institute defined a standardized MFCC algorithm to be used in mobile phones; all of these algorithms were laid out by hand. Up until a certain point, manually-extracted features dominated machine learning. For instance, SIFT (Scale Invariant Feature Transform), which enables one to detect and describe local features in images based on Gabor filters and the like, was commonly used in computer vision.

Overall, people have come up with many approaches to feature extraction but still cannot duplicate the brain’s incredible success. Moreover, the brain has no biological predetermination, meaning that there are no neurons genetically created only for producing speech, remembering people’s faces, etc. It looks like any area of the brain can learn to do anything. Regardless of the brain’s activity, naturally, we would like to learn to select features automatically to create complex AI and large models containing neurons linked to one another for transmitting signals containing all sorts of different information. Most likely, humans lack the resources necessary to develop the best possible features for images or speech manually.

Artificial neural networks

When Frank Rosenblatt introduced his perceptron, everyone started imagining that machines would become truly smart any day now. His network learned to recognize letters on photographs, which was very cool for the late 1950s. Very soon after, neural networks made up of many perceptrons were developed; they could learn with backpropagation (a method used to calculate the gradient descent called the backward propagation of errors). Basically, backpropagation is a method used to calculate gradients or error functions.

The idea of automatic differentiation was floating around back in the 1960s even, but Geoffrey Hinton, a British-Canadian computer scientist who has been one of the leading researchers on deep learning, rediscovered backpropagation and expanded its scope. Incidentally, George Boole, one of the founders of mathematical logic, was Hinton’s great-great-grandfather.

Multi-layer neural networks were developed in the second half of the 1970s. There weren’t any technical barriers in place at that time. All you had to do was take a network with one layer of neurons, then add a hidden layer of neurons, and then another. That got you a deep network, and, formally speaking, backpropagation works in exactly the same way on it. Later on, researchers started using these networks for speech and image recognition systems. Then recurrent neural networks (RNN), time delay neural networks (TDNN), and others followed; however, by the end of the 1980s it became evident that there were several significant problems with neural network learning.

First off, let us touch upon a technical problem. A neural network needs good hardware to learn to act intelligently. In the late eighties and early nineties, research on speech recognition using neural networks looked something like this: tweak a hyperparameter, let the network train for a week, look at the outcome, tweak the hyperparameters, wait another week, rinse, repeat. Of course, these were very romantic times, but since tuning the parameters in neural networks is nearly as important as the architecture itself, too much time or too powerful hardware was needed to achieve a good outcome for each specific task.

As for the core problem, backpropagation does work formally, but not always in practice. For a long time, researchers weren’t able to efficiently train neural networks with more than two hidden layers due to the vanishing gradients problem: when you compute a gradient with backpropagation, it may decrease exponentially as it progresses from the output to input neurons. The opposite problem — exploding gradients — would crop up in recurrent networks; if one starts to unravel a recurrent network, the gradient may spin out of control and start growing exponentially.

Eventually, these problems led to the “second winter” of neural networks, which lasted through the 1990s and early 2000s. As John Denker, a neural networks researcher, wrote in 1994, “neural networks are the second best way of doing just about anything” (the second half of this quote isn’t as well-known:.”…and genetic algorithms are the third”). Nonetheless, a true revolution in machine learning occurred ten years ago. In the mid-2000s, Geoffrey Hinton and his research group discovered a method of training deep neural networks. Initially, they did this for deep belief networks based on Boltzmann machines, and then they extended this approach to traditional neural networks.

What was Hinton’s idea? We have a deep network that we want to train. As we know, layers close to the network’s output can learn well using backpropagation. How can we train what’s close to the input, though? At first, we will train the first layer by unsupervised learning. After that, the first layer will already be extracting some features, looking for what the input data points have in common. After doing that, we pre-train the second layer, using results of the first one as inputs, and then the third. Eventually, once we’ve trained all the layers, we’ll use the system as a first approximation and then fine-tune the resulting deep network to our specific task by using backpropagation. This is an excellent approach… and, of course, it was first introduced back in the seventies and eighties. However, much like regular backpropagation, it worked poorly. Yann LeCun’s team achieved great success in the early 1990s in computer vision with autoencoders, but, generally speaking, their method didn’t work better than solutions based on manually-designed features. In short, Hinton can take credit for making this approach work for deep neural networks (and it would be too long and complicated to explain exactly what he did).

However, researchers had sufficient computational capabilities to apply this method by the end of the 2000s. The main technological revolution occurred when Ruslan Salakhutdinov (also advised by Hinton) managed to shift the training of deep networks to GPUs. One can view this training as a large number of relatively independent and relatively undemanding computational processes, which is perfect for the highly parallel GPU architectures, so everything started working much faster. By now, you simply have to use GPUs to train deep learning models efficiently, and for GPU manufacturers like NVIDIA deep learning has become a primary application that carries the same weight as modern games. Take a look at CEO NVIDIA’s pitch here:

Sergey Nikolenko,
Chief Research Officer, Neuromation

Sergey’s a researcher in the field of machine learning (deep learning, Bayesian methods, natural language processing and more) and analysis of algorithms (network algorithms, competitive analysis). He has authored more than 120 research papers, several books, courses “Machine learning”, “Deep learning”, and others. Extensive experience with industrial projects (Neuromation, SolidOpinion, Surfingbird, Deloitte Analytics Institute).

This article compares neurons to machines and explores the human brain’s capabilities and limitations.

Despite decades of steady advances, in many fields the human brain is still more capable than computers. For instance, we handle natural language better — we can read, understand, and parse content from a book. We are pretty good at learning in a broader sense too. So, what does the human brain do and how does it manage to achieve such remarkable results? How do neurons in your brain work differently than transistors in a processor? Naturally, this topic is inexhaustible, but let us try to begin with a few examples.

As you may know, every neuron occasionally sends electrical impulses, otherwise known as spikes, along axons. Neurons never stop and keep sending signals as long as they’re still alive; however, when neurons are “turned off” they rarely send signals. When neurons are triggered, or “turned on”, spikes occur much more frequently.

Neurons function stochastically, meaning they produce electric signals at random intervals. The patterns of these signals can be pretty accurately approximated with a Poisson process. Computers contain logic gates that send signals back and forth, but their synchronization frequency is fixed and by no means random. This frequency is called a computer’s “clock rate”, which has been measured in gigahertz for quite a while now. On every tick, gates on a certain layer send signals up to the next layer. Although this is done a few billion times a second, it’s performed simultaneously, as though the gate were following a strict order.

Actually, it is very easy to see that neurons can synchronize well and count tiny time intervals very precisely. Stereo sound is the simplest and most illustrative example. When you move from one end of the room to the other, you can easily tell, based solely on the sound coming from the television, where you’re going (being able to tell where sound was coming from was crucial for surviving in prehistoric times). You can tell where you’re going by noticing that the sound reaches your left and right ears at different times.

Your inner ears aren’t all that far apart (about 20 cm), and if you divide that by the speed of sound (340 m/s) you get a very short interval — hundredth of milliseconds — between when the sound waves reach each ear. Nevertheless, your neurons pick up on this minor difference excellently, which enables you to figure out precisely where you’re headed. In other words, your brain could process frequency — in kilohertz — just like a computer. Considering the extensive parallel processing performed by your brain, its architecture could generate rather intelligent computational capabilities…but, for some reason, your brain doesn’t do that.

Let us go back to parallel processing for a second. We recognize people’s faces within a few hundred milliseconds, and connections between different neurons are activated within tens of milliseconds, which means that only a few neurons — probably fewer than a dozen — form a serial circuit in the full facial recognition cycle.

On the one hand, the human brain contains an incredible number of neurons, while, on the other hand, it doesn’t have as many layers as a regular processor. Processors have very long serial circuits, while the brain has short and highly parallel circuits. And while a processor core basically works on one thing at a time (but can switch between different tasks with lightning speed, so it appears to you that everything is working at once), the brain can work on a lot of tasks simultaneously, since neurons light up in many areas of the brain when they start recognizing someone’s face or doing some other equally exciting thing.

The illustration above shows how the brain processes a visual signal in time. The light reaches the retina, where it transforms into electrical impulses and then the image is transmitted 20–40 milliseconds later. The first stage takes 10–20 milliseconds (the image shows the cumulative time, i.e. a total of 140–190 milliseconds passes by the time a motor command is issued).

During the second stage, 20–30 milliseconds later, the signal reaches the neurons that recognize simple visual forms. Then there’s another stage, and another, and only during the fourth stage do we see intermediate forms — there are neurons that “light up” when seeing squares, color gradients or other similar objects. Then the brain goes through a few more stages, and neurons capable of discerning high level object descriptions light up 100 milliseconds after the process began. For instance, when you meet someone new a neuron responsible for recognizing her face appears (this is a terrible simplification and we can’t verify this claim but it appears that there is some truth to it). Most likely, a neuron or a group of neurons responsible for this person in general and lighting up whenever you come into contact with her, including when you interact with her not face-to-face, appear. If you see her face again (and the neuron didn’t unlearn or forget this earlier information) that same neuron will be activated ~100 milliseconds later.

Why does the brain work like that? Answering that question with a simple “evolution did it” doesn’t really explain anything. The human brain evolved to a certain point, and that was sufficient to solve problems as we evolved. The rationalist community says that living organisms are not fitness-maximizers who optimize some survival objective function, but rather adaptation executors, who execute “relatively solid” decisions that were chosen randomly at some point. Well, rigid synchronization with a built-in chronometer never took place; however, we can’t tell you exactly why it played out that way.

Actually, in this case, it seems as though asking “why” isn’t all that relevant. It’s better, more interesting, and more productive to ask “how”. How exactly does the brain work? We don’t know for sure but now we can describe the processes going on inside our heads quite well, at least in terms of individual neurons or, in certain instances, groups of neurons.

What can we learn from the brain? First, feature extraction. The brain can learn to make excellent generalizations based on a very, very limited sample size. If you show a young child a table and tell her it’s a table then the child starts calling other tables tables, although they seemingly don’t have anything in common — they could be round, square, have one leg or four. It’s evident that a child doesn’t learn to do this by supervised learning; she obviously lack the training set necessary to do so. One can assume the child created a cluster of “objects with legs on which people place things”. Her brain had already extracted the “Plato’s eidos” and then, when she heard the word for it, she simply attached a label to a ready-made idea.

Naturally, this process can go in the opposite direction, too. Although the neurons (and other things) of many linguists start twitching nervously when they hear the names Sapir and Whorf, one must admit that many ideas, especially abstract ones, are mostly socio-cultural constructs. For instance, every culture has a word similar in meaning to the concept of “love”; however, the sentiment may be very different. American “love” has little in common with that of ancient Japan. Since, generally, all people have the same physiological traits, the abstract idea of “being drawn towards another person” is not simply labeled in language but rather its adjusted and constructed by the texts and cultural data that define it for a person. But let us return to the main point of the article…to be continued next week.

Sergey Nikolenko
Chief Research Officer, Neuromation

Sergey Nikolenko, Chief Research Officer at Neuromation, visits Samsung R&D Office in Seoul — Follow our news!