Category: Neuromation

[translated from the Russian version by Andrey V. Polyakov]

Recently, a new round of conversations about self-moving carriages and other potential achievements of artificial intelligence has again posed one of the classic questions of humanity: who will be marginalized by the monorail track of the next technological revolution? Some studies argue that artificial intelligence in the near future would lead to a surge of unemployment comparable to the Great Depression. Today we will also talk about who can be replaced by computers in the near future, and who can, without fear and even with some degree of self-satisfaction, expect the arrival of our silicon overlords.

Luddites: senseless English riot or reasonable economic behavior?

Some people believe labor-saving technological change is bad for the workers because it throws them out of work. This is the Luddite fallacy, one of the silliest ideas to ever come along in the long tradition of silly ideas in economics.William Easterly. The Elusive Quest for Growth: Economists’ Adventures and Misadventures in the Tropics

Such a text could hardly do without recalling the most famous opponents of technological progress: the Luddites. Moods against technological progress were strong among English textile workers as far back as the 18th century, but Nottingham manufacturers did not receive letters on behalf of Ned Ludd until 1811. The trigger for the transition to active actions was the introduction of stocking machines, which made the skills of skilled weavers unnecessary: now stockings could be sewn of separate parts without special skills. The resulting product was, by the way, much worse, with stockings quickly bursting at the seams, but they were so much cheaper and they still enjoyed immense popularity. Fearing to lose work, the weavers began attacking factories and smashing newfangled machines.

Luddites were well organized. They understood that conspiracy was vital for them; they brought terrible oaths of loyalty, acted under the cover of night and were rarely apprehended. By the way, most probably General Ludd has never existed: it was a folklore “bigger than life” figure like Paul Banyan. The fight against Luddites was taken seriously: in 1812, “machine breaking” was proclaimed a capital crime, and there were more British soldiers suppressing uprisings than there were fighting Napoleon in the same years! Yet, the Luddites managed to significantly reduce automation in textile production, the prices for products increased, and the goals of the movement were partially achieved. But honestly, when did you last wear any hand-woven stockings, socks or pantyhose?…

We have placed a quote from the book of economist William Easterly as epigraph to this section. Easterly explains that the Luddite movement has spawned (no longer among English weavers, but in quite intellectual circles) the erroneous idea, which he calls the Luddite fallacy, and which still, from time to time, occurs to quite real economists. The idea is that the development of automation is supposed to inevitably lead to a reduction in employment, because fewer people are needed to maintain the same level of production. However, both theory and practice show that quite another outcome is no less likely: simply the same, or even a larger, number of people would produce more goods! And the progress is usually on the side of workers, increasing their productivity and, consequently, their income. Yes, the last consequence is not always that obvious, but there is no apparent way to increase the welfare of each individual worker.

Nevertheless, these arguments do not refute the Luddites themselves. Regardless of the rhetoric, the Luddites were afraid not of a bright future, in which every stockinger could make a hundred pairs of stockings a day, but of immediate tomorrow, when they were on the street and no one would need the only skill they possessed. Moreover, even Easterly does not deny that progress can lead to unemployment and decline of prosperity for certain workers, even if the average well-being of the people grows. Let us take a closer look at this argument.

Whom did the robots replace recently

…we set up this room with girls in it. Each one had a Marchant: one was the multiplier, another was the adder. This one cubed — all she did was cube a number on an index card and send it to the next girl… The speed at which we were able to do it was a hell of a lot faster… We got speed… that was the predicted speed for the IBM machine. The only difference is that the IBM machines didn’t get tired and could work three shifts. But the girls got tired after a while.Richard Feynman. Surely You’re Joking, Mr. Feynman!

An economist from Harvard James Bessen conducted an interesting study. He took a list of major 270 occupations used in the 1950 U.S. Census, and then checked how many had been completely automated so far. It turned out, that out of 270 occupations:

• 232 are active so far;
• 32 were eliminated due to decline of demand and changes in market structure (Bessen refers to, e.g., boardinghouse keepers, however, how would you call people occupied in Airbnb…);
• five became obsolete due to technological advances, but were never automated (for example, a telegraph operator);
• and only one occupation was really fully automated.

Those, willing to think, will have time until the next paragraph.

The only fully automated occupation in Bessen’s research is elevator operator. At this point, our 80+ readers who grew up in the U.S. can certainly sigh and complain that earlier warm colored boys closed behind you double elevator doors, which could not close themselves, and pressed warm lamp buttons… but was it so good for you, and for the colored boys themselves? Maybe they should go to school after all?…

We can add another interesting career path to Bessen’s research. For quite a long time computers have completely replaced the occupation of… computer. Oddly enough, not everyone knows about the existence of this occupation, although it seems obvious that before the advent of computers it was necessary to calculate manually. Many great mathematicians were also outstanding computists: for example, Leonhard Euler could carry out complex calculations in his mind and often amused himself with some complicated exercise when his wife managed to get him to the theater. But how were the calculations made, for example, in the Manhattan project, where no physicist could have managed them on his own, “on paper”?

Well, they really did them manually. The main idea is described in the epigraph: when people perform operations sequentially, as if on a conveyor, it turns out much faster. Of course, humans are prone to error, and the results of the calculations had to be rechecked, but for this, special algorithms can also be developed. In the 19th century, human computers compiled mathematical tables (for example, sinuses or logarithms), and in the twentieth century they worked for military needs, including the Manhattan project. It is computists, by the way, who are responsible for the fact that the occupation of programmer was at first considered female: the computists were mostly girls (for purely sexist reasons — it was believed that women are better suited for monotonous tasks), and the first programmers were often recruited from them.

In his book When Computers Were Human, David Alan Grier describes the atmosphere of the work of living calculators, citing Dickens (another unexpected affinity to real Luddites): “A stern room, with a deadly statistical clock in it, which measured every second with a beat like a rap upon a coffin lid.” Should we be nostalgic for this patriarchal, but by no means bucolic picture?

Bessen distinguishes between complete and partial automation. Indeed, a hereditary elevator operator could have a hard time in a brave new world (there is only one problem: hereditary elevator operators apparently did not have enough time to grow). However, if only part of what you are doing is automated, the demand for your occupation can even increase. And again one can return to the Luddites: during the 19th century, 98% of the labor required to weave a yard of cloth was automated. Theoretically, it was possible to dismiss forty-nine weavers out of fifty and produce the same amount of cloth. However, the final effect was opposite: the demand for cheap cloth grew dramatically, resulting in demand for weavers, and, therefore, the number of jobs increased significantly. And in the 1990’s, the widespread deployment of ATMs did not reduced, but even increased the demand for bank tellers: ATMs allowed banks to operate branch offices at lower cost; this prompted them to open many more branches.

Despite numerous technological innovations, complete occupation automation in the last 50–100 years did not have a noticeable effect in the society. Do you know anyone whose grandfather was an elevator operator and grandmother a computer, who have lost jobs and sunk to the bottom of society because of the goddamned automation?

But partial automation over these years has radically changed the content of work for the vast majority of us. Computers and especially the Internet have made many professions many times more productive — can you imagine how long it would take me to collect data for this article in the 1950’s? And globalization and automation of economy tangibly raised the standard of living — for all, not just the elite. It would be trivial to say that we now live better than medieval kings did — but try to compare our standard of living with the way our grandparents lived. I will not give any examples or statistics, every reader has their own experience thereof: just recall your life before such trifles as mobile phones, Google, microwave ovens, dishwashers (and even washing machines were recently not in all households)…

Therefore, it seems that progress and automation have so far only helped people, and the Luddites’ fears were greatly exaggerated. But, maybe, “the fifth industrial revolution” will be absolutely different? In the second part of the article we will try to dream on this topic.

“The threat of automation”

How far has gone “progrès”? Laboring is in recess,
And mechanical, for sure, will the mental one replace.
Troubles are forgotten, no need for quest,
Toiling are the robots, humans take a rest.

Yuri Entin. From the movie Adventures of Electronic

Elevator operators, computists and weavers perfectly symbolize those activities that have been automated so far: technical work, where the output result is expected to maximally match certain parameters, and creativity is not only difficult, but forbidden and, in fact, harmful. Of course, an elevator operator could smile to his passengers, or the computer girl could, having scorned potential damage to her reputation, get acquainted with Richard Feynman himself. But their function was to accurately perform clear, algorithmically defined actions.

I will allow myself a little emotion: these are exactly the kinds of activities that must be automated further! There is nothing human in following a fixed algorithm. Monotonous work with a predetermined input and output is always an extreme measure, the forced oppression of the human spirit in order to achieve certain practical goal. And if the goal can be achieved without wasting human time, that is the way to proceed.

However, now it comes to the fact that computers start gradually replacing people in areas that were so far considered as human domain. For example, since 2014, Facebook is able to recognize faces on human performance level, and computer vision technologies are only improving.

Sergey Nikolenko
Chief Research Officer, Neuromation

Why do we need deep neural networks with dozens of hidden layers? Why can’t we just train neural networks with one hidden layer? In 1991, Kurt Hornik proved the universal approximation theorem, which states that for every continuous function there exists a neural network with linear output that approximates this function with a given accuracy. In other words, a neural network with one hidden layer can approximate any given function as accurately as we want. However, as it often happens, the network will be exponentially large, and even if efficiency isn’t a concern, it still isn’t clear how to make a transition from that network existing somewhere in the space of all possible networks to one we can train in real life.

Actually, with a deeper representation you can approximate the same function and solve the same task more compactly, or solve more tasks with the same resources. For instance, traditional computer science studies Boolean circuits that implement or Boolean functions, and it turns out that many functions can be expressed much more efficiently if you allow for circuits of depth 3, some more functions, of depth 4, and so on, even though circuits of depth 2 can obviously express everything by reducing to a DNF or CNF. Something similar happens in machine learning. Picture a space of test points we want to sort into two groups. If we have only “one layer” then we can divide them with a (hyper)plane. If there are two layers then it comes down to linear separating surfaces composed of several hyperplanes (that’s roughly how boosting works — even very simple models become much more powerful if you compose them in the right way). On the third layer, yet more complicated structures made out of these separating surfaces take shape; it’s not all that easy to visualize them any more; and the same goes for subsequent layers. Below is a simple example from Goodfellow, Bengio, Courville “Deep Learning”. This illustration shows that if you combine even the simplest linear classifiers, each of which divides a plane into two half-planes, then, consequently, you can define regions of a much more complicated shape:

Thus, in deep learning we’re trying to build deeper architectures. We have described the main idea already — we pre-train the lower layers, one-by-one, and then finish training the whole network by finetuning with backpropagation. Therefore, let’s cover pretraining in more detail.

One of the simplest ideas is to train a neural network to copy its input to its output through a hidden layer. If the hidden layer is smaller, then it appears that the neural network should learn to extract some significant features from the data, which enables the hidden layer to restore the input. This type of neural network architecture is called an autoencoder. The simplest autoencoders are feedforward neural networks that are most similar to the perceptron and contain an input layer, hidden layer, and output layer. Unlike the perceptron, an autoencoder’s output layer has to contain as many neurons as its input layer.

The training objective for this network is to bring theoutput vector x’ as close as possible to the input vector x

The main principle of (working with and) training an autoenconder network is to achieve a response on the output layer that is as close as possible to the input. Generally speaking, autoencoders are understood to be shallow networks, although there are exceptions.

Early autoencoders were basically doing dimensionality reduction. If you take much fewer hidden neurons than the inner and outer layers have, you wind up making the network compress the input into a compact representation, while at the same time ensuring that you’ll be able to decompress it later on. That’s what undercomplete autoencoders, where the hidden layer has lower dimension than the input and output layers, do. Now, however, overcomplete autoencoders are used much more often; they have a hidden layer of higher, sometimes much higher, dimension than that of the input layer. On the one hand, this is good, because you can extract more features. On the other hand, the network may learn to simply copy input to output with perfect reconstruction and zero error. To keep this from happening, you have to introduce regularization, not simply optimize the error.

Returning the points to the dataset manifold

Classical regularization aims to avoid overfitting by using a prior distribution, like in a linear regression. But this won’t work in this case. Generally, autoencoders use more sophisticated methods of regularization that change inputs and outputs. One classical approach is a denoising autoencoder, which adds artificial noise to the input pattern and then asks the network to restore the initial pattern. In this case, you can make the hidden layer larger than that of the input (and output) layers: the network, however large, still has an interesting and nontrivial task to learn.

Denoising autoencoder

Incidentally, “noise” can be a rather radical change of the input. For instance, if it’s a binary pixel-based image then you can simply remove some of the pixels and replace them with zeroes (people often remove up to half of them!). However, the objective function you’re reconstructing is still the correct image. So, you make the autoencoder reconstruct part of the input based on another part. Basically, the autoencoder has to learn to understand how all the inputs are designed and understand the structure of the aforementioned manifold in a space with a ridiculous number of dimensions.

How deep learning works

Suppose you want to find faces on pictures. Then one input data point is an image of a predefined size. Essentially, it’s a point in a multi-dimensional space, and the function you’re trying to find should take on the value of 0 or 1 depending on whether or not there’s a face on the picture. For example, mathematically speaking, a 4Mpix photo is a point in the space of dimension about 12 million: 4 million pixels times three color values per pixel. It’s glaringly obvious that only a small percentage of all possible images (points in this huge space) contain faces, and these images appear as tiny islands of 1s in an ocean of 0s. If you “walk” from one face to another in this multi-dimensional space of pixel-based pictures you’ll come across nonsensical images in the center; however, if you pre-train a neural network — by using a deep autoencoder, for instance — then closer to the hidden layer the original space of images turns into a space of features where peaks of the objective function are clustered much closer to one another, and a “walk” in the feature space will look much more meaningful.

Generally, the aforementioned approach does work; however, people can’t always interpret the extracted features. Moreover, many features are extracted by a conglomeration of neurons, which complicates matters. Naturally, one can’t categorically assert this is bad, but our modern notions of how the brain works tell us that different processes happen in the brain. We practically don’t have any dense layers, which means that only a small percentage of neurons responsible for extracting the features in question take part in solving each specific task.

So how do you get each neuron to learn some useful feature? This brings us back to regularization; in this case, to dropout. As we already mentioned, a neural network usually trains by stochastic gradient descent, randomly choosing one object at a time from the training sample. Dropout regularization involves a change in the network structure: each node in the network is removed from training with a certain probability. By throwing away, say, half of the neurons, you get a new network architecture on every step of the training.

Having trained the other half of the neurons, we get a very interesting result. Now every neuron has to train to extract some feature by itself. It can’t count on teaming up with other neurons, because they may have been dropped out.

Dropout basically averages a huge number of different architectures. You build a new model on each test case. You take one model from this gigantic ensemble and perform one step of training, then you take another model for the next case and perform one step of training, and, eventually, you average all of these at the end, on the output layer. It is a very simple idea on the surface, but when dropout was introduced it led to great boons for practically all deep learning models.

We will go off on one more tangent that will link what’s happening now to the beginning of the article. What does a neuron do under dropout? It has a value, usually a number from 0 to 1 or from -1 to 1. A neuron sends this value further, but only 50% of the time rather than always. But what if we do it the other way around? Suppose a neuron always sends a signal of the same value, -namely ½, but sends it with probability equal to its value. The average output wouldn’t change, but now you get stochastic neurons that randomly send out signals. The intensity with which they do so depends on their output value. The larger the output the more the neuron is activated and the more often it sends signals. Does it remind you of anything? We wrote about that at the beginning of the article. That’s how neurons in the brain work. Like in the brain, neurons don’t transmit the spike’s amplitude, but rather they transmit one bit, the spike itself. It’s quite possible that our stochastic neurons in the brain perform the function of a regularizer, and it’s possible that, thanks to this, we can differentiate between tables, chairs, cats, and hieroglyphics.

After a few more tricks for training deep neural networks were added to dropout, it turned out that unsupervised pre-training wasn’t all that necessary, actually. In other words, the problem of vanishing gradients has mostly been solved, at least for regular neural networks (recurrent networks are a bit more complicated). Moreover, dropout itself has been all but replaced by new techniques such as batch normalization, but that would be a different story.

Sergey Nikolenko
Chief Research Officer, Neuromation

Block Tribune asked Sergey Nikolenko to comment on the hype around knowledge mining idea.

Earnings from mining crypto currencies are shrinking — the computations complexity grows, while the energy is not getting any cheaper. Many people are looking for alternative applications for expensive hardware purchased during the mining boom. It is quite possible that a substantial part of video cards would be used by scientists or start-ups for complex computing.

With the energy price at RUR 4.5 per kilowatt-hour, there already are not so many wishing to engage in cryptocurrencies mining in Russia. As soon as in winter and spring of this year, investments in new video cards and ASIC chips for a medium-sized cryptocurrency farm paid off completely within few months. Now, in order to earn on mining of many popular crypto currencies, you must first be a millionaire — the format of “makeshift video card” does not work at all, as large farms with well-tuned hardware are required.

That is why, when power plants started to offer their excess capacities on sites with complete infrastructure for rent, miners were immediately interested — with the price of just two rubles per kilowatt-hour. Considering that they are into mining of “light” crypto currencies, like ethereum, Zcash and Monero, earning less than $6 per day, this was a significant support.

Due to energy prices, approximately half of cryptocurrencies are mined in several regions of China. However, the computing complexity will continue to grow, causing a gradual fall of profits. Hence the interest to alternative sources of income.

Since many farms have, in fact, huge computing capacity, they can be primarily used for science purposes.

Usually, for such tasks, supercomputers capacities are rented, e.g “Lomonosov” in Moscow State University being one of the most powerful. These are the machines that will compete with the mining capacities. .

The rental market for mining capacities is already emerging. For example, Neuromation has created a distributed synthetic data platform for neural network applications. Their first commercial product was making store shelves smart. For this, large well-labeled datasets for all the SKUs are created. Algorithms trained on these are able to analyze shelf layout accuracy, percentage of the shelf, and customers’ interaction. The system is capable, actually, to predict customers’ behavior.

The platform requires more than a billion of labeled images of merchandise. Manual labeling of photographs is a painstaking, and very costly, task. For example, on the Amazon Mechanical Turk crowdsourcing service, the manual labeling of a billion pictures would cost about $120 million.

Neuromation entered the market with a new concept of using synthetic data for training neural networks. They generate all the necessary images in 3D generator, similar to a computer game for artificial intelligence. It is partly for this generator that they need large-scale computing capacities, which, if rented from Amazon or Microsoft, would cost tens of millions of dollars. On the other hand, there are thousands of the most advanced video cards available, and they are engaged in ever less profitable ethereum mining.

The founder of Neuromation, Maxim Prasolov, decided, instead of renting capacity for millions of dollars, to lease these mining farms for useful computing, and the company is already using a pool of 1000 video cards. “This is a serious savings for our research process and is beneficial for the miners: the farm services cost 5–10 times cheaper than renting cloud servers, and the miners can earn more by solving fundamental problems instead of mining crypto currency,” he calculates.

It is, of course, to remember that Google has a search for images and Facebook has facial recognition technology for photos, that they managed to develop with their own cloud services without using mining farms. However, the task for Neuromation was substantially different. “First, searching by pictures is a completely different task, and there are specially developed methods for face recognition. Second, Google and Facebook do not need to rent computing power from Amazon Web Services — they have more than enough of their own clusters. But the course of action for a small start-up in this situation is not so obvious,” explains Sergey Nikolenko, Chief Research Officer in Neuromation.

Potentially, miners will gain in average 10–20% more on knowledge mining compared to crypto mining.

Moreover, with tangible benefits for society. “Basically, mining is milling the wind. To generate a “nice” hash, dozens of system operation hours are needed. On the other hand, if we are talking about the search for a drug formula, such a use of capacities from around the world, the application of combined work of computers for the common good would be comparable to the results of research at the Large Hadron Collider,” says Petr Kutyrev, editor of the noosfera.su portal.

The tasks solvable with the mining hardware will be limited due to its customization level. For example, ASIC hardware would be difficult to adapt for scientific tasks, as it is designed exclusively for hashing. The video cards, however, can cope with various scientific tasks.

True, special hardware configuration and sometimes software would be required for such computing. “Video cards for mining can be used for video recognition and rendering, biological experiments. However, for efficient computing, direct access to the computer hardware is essential. If the computation task may be delayed, or the accuracy of machine learning of neural networks is not critical, then, of course, standard tools can be used. Otherwise, you need to develop your own hardware and software infrastructure,” Evgeny Glariantov believes.

Thus, using the farms in science will require some time to set them up and develop special allocation protocols. Yet, considering a more profitable segment, miners will switch to useful computing, and platforms for such tasks may appear in the near future together with the first operating system based on EOS blockchain, believe in the BitMoney Information and Consulting Center. Miners will be able, from time to time, to switch from cryptocoin mining to processing scientific or commercial data, thereby increasing their profitability. Profits from the times of cryptocurrency rush will no longer exist, but business will be more meaningful and stable: unlike volatile crypto currencies, there is always a demand for knowledge.

Source: http://blocktribune.com/making-money-redundant-mining-hardware-opinion/

Neuroplasticity is another part of this issue. Scientists conducted experiments demonstrating how different areas of the brain can easily learn to do things for which they’re seemingly not designed. Neurons are the same everywhere, but there are areas in the brain responsible for different things. There’s the Broca area responsible for speech, an area responsible for vision (actually, a lot of areas — vision is very important for humans), and so forth. Nevertheless, we can break down these notional biological borders.

This man is learning to see with his tongue. He attaches electrodes to his tongue, puts a camera on his forehead, and the camera streams an image on the electrodes pricking his tongue. People stick that thing on them and walk around with it for a few days, with their eyes open, naturally. The part of the brain that receives signals from the tongue starts to figure out what’s going on — this feels a lot like something that comes from my eyes. If you abuse somebody like that for a week and then blindfold him, he’ll actually be able to see with his tongue! He is now able recognize simple forms and doesn’t bump into walls.

The man in this photo has turned into a bat. He’s walking around blindfolded, using an ultrasonic scope whose signals reach his tactile neurons through his skin. With a sonar like this, a human being can develop echolocation abilities within a few days of training. We do not have a special organ that can discern ultrasound, so you have to attach a scope to your body. However, we can relatively easily learn to process this information, meaning that we can walk in the dark and not bump into any walls.

All of this shows that the brain can adapt to a very large number of different data sources. Hence, the brain probably has a “common algorithm” that can extract meaning from whatever it takes in. This common algorithm is the Holy Grail of modern artificial intelligence (a recent popular book on machine learning by Pedro Domingos was called The Master Algorithm). It appears as though deep learning is the closest we have come to the master algorithm of all the things done in the field up until now.

Naturally, one has to be cautious when making claims about whether all of this is like what the brain does. “Could a neuroscientist understand a microprocessor?”, a recent noteworthy article, tries to elucidate how effective current approaches in neurobiology are at analyzing a very simple “brain”, like a basic Apple I processor or Space Invaders on Atari. We will return to this game soon enough and won’t go into much detail about the results but we do recommend reading the paper. Spoiler alert: modern neurobiology couldn’t figure out a single thing about Space Invaders.

Feature extraction

Unstructured information (texts, pictures, music) is processed in the following way: there is raw input, then features that bear content take shape, and then classifiers are built based on those features. The most complicated part of this process is understanding how to pick good features out of unstructured input. Up until recently, systems for processing unstructured information have worked as follows: people have attempted to select good features manually and then assess the quality of relatively simple regressors and classifiers based on these features.

Take Mel Frequency Cepstral Coefficients (MFCC), which had been commonly used as features in speech recognition systems, for example. In 2000, the European Telecommunications Standards Institute defined a standardized MFCC algorithm to be used in mobile phones; all of these algorithms were laid out by hand. Up until a certain point, manually-extracted features dominated machine learning. For instance, SIFT (Scale Invariant Feature Transform), which enables one to detect and describe local features in images based on Gabor filters and the like, was commonly used in computer vision.

Overall, people have come up with many approaches to feature extraction but still cannot duplicate the brain’s incredible success. Moreover, the brain has no biological predetermination, meaning that there are no neurons genetically created only for producing speech, remembering people’s faces, etc. It looks like any area of the brain can learn to do anything. Regardless of the brain’s activity, naturally, we would like to learn to select features automatically to create complex AI and large models containing neurons linked to one another for transmitting signals containing all sorts of different information. Most likely, humans lack the resources necessary to develop the best possible features for images or speech manually.

Artificial neural networks

When Frank Rosenblatt introduced his perceptron, everyone started imagining that machines would become truly smart any day now. His network learned to recognize letters on photographs, which was very cool for the late 1950s. Very soon after, neural networks made up of many perceptrons were developed; they could learn with backpropagation (a method used to calculate the gradient descent called the backward propagation of errors). Basically, backpropagation is a method used to calculate gradients or error functions.

The idea of automatic differentiation was floating around back in the 1960s even, but Geoffrey Hinton, a British-Canadian computer scientist who has been one of the leading researchers on deep learning, rediscovered backpropagation and expanded its scope. Incidentally, George Boole, one of the founders of mathematical logic, was Hinton’s great-great-grandfather.

Multi-layer neural networks were developed in the second half of the 1970s. There weren’t any technical barriers in place at that time. All you had to do was take a network with one layer of neurons, then add a hidden layer of neurons, and then another. That got you a deep network, and, formally speaking, backpropagation works in exactly the same way on it. Later on, researchers started using these networks for speech and image recognition systems. Then recurrent neural networks (RNN), time delay neural networks (TDNN), and others followed; however, by the end of the 1980s it became evident that there were several significant problems with neural network learning.

First off, let us touch upon a technical problem. A neural network needs good hardware to learn to act intelligently. In the late eighties and early nineties, research on speech recognition using neural networks looked something like this: tweak a hyperparameter, let the network train for a week, look at the outcome, tweak the hyperparameters, wait another week, rinse, repeat. Of course, these were very romantic times, but since tuning the parameters in neural networks is nearly as important as the architecture itself, too much time or too powerful hardware was needed to achieve a good outcome for each specific task.

As for the core problem, backpropagation does work formally, but not always in practice. For a long time, researchers weren’t able to efficiently train neural networks with more than two hidden layers due to the vanishing gradients problem: when you compute a gradient with backpropagation, it may decrease exponentially as it progresses from the output to input neurons. The opposite problem — exploding gradients — would crop up in recurrent networks; if one starts to unravel a recurrent network, the gradient may spin out of control and start growing exponentially.

Eventually, these problems led to the “second winter” of neural networks, which lasted through the 1990s and early 2000s. As John Denker, a neural networks researcher, wrote in 1994, “neural networks are the second best way of doing just about anything” (the second half of this quote isn’t as well-known:.”…and genetic algorithms are the third”). Nonetheless, a true revolution in machine learning occurred ten years ago. In the mid-2000s, Geoffrey Hinton and his research group discovered a method of training deep neural networks. Initially, they did this for deep belief networks based on Boltzmann machines, and then they extended this approach to traditional neural networks.

What was Hinton’s idea? We have a deep network that we want to train. As we know, layers close to the network’s output can learn well using backpropagation. How can we train what’s close to the input, though? At first, we will train the first layer by unsupervised learning. After that, the first layer will already be extracting some features, looking for what the input data points have in common. After doing that, we pre-train the second layer, using results of the first one as inputs, and then the third. Eventually, once we’ve trained all the layers, we’ll use the system as a first approximation and then fine-tune the resulting deep network to our specific task by using backpropagation. This is an excellent approach… and, of course, it was first introduced back in the seventies and eighties. However, much like regular backpropagation, it worked poorly. Yann LeCun’s team achieved great success in the early 1990s in computer vision with autoencoders, but, generally speaking, their method didn’t work better than solutions based on manually-designed features. In short, Hinton can take credit for making this approach work for deep neural networks (and it would be too long and complicated to explain exactly what he did).

However, researchers had sufficient computational capabilities to apply this method by the end of the 2000s. The main technological revolution occurred when Ruslan Salakhutdinov (also advised by Hinton) managed to shift the training of deep networks to GPUs. One can view this training as a large number of relatively independent and relatively undemanding computational processes, which is perfect for the highly parallel GPU architectures, so everything started working much faster. By now, you simply have to use GPUs to train deep learning models efficiently, and for GPU manufacturers like NVIDIA deep learning has become a primary application that carries the same weight as modern games. Take a look at CEO NVIDIA’s pitch here:

Sergey Nikolenko,
Chief Research Officer, Neuromation

Sergey’s a researcher in the field of machine learning (deep learning, Bayesian methods, natural language processing and more) and analysis of algorithms (network algorithms, competitive analysis). He has authored more than 120 research papers, several books, courses “Machine learning”, “Deep learning”, and others. Extensive experience with industrial projects (Neuromation, SolidOpinion, Surfingbird, Deloitte Analytics Institute).

This article compares neurons to machines and explores the human brain’s capabilities and limitations.

Despite decades of steady advances, in many fields the human brain is still more capable than computers. For instance, we handle natural language better — we can read, understand, and parse content from a book. We are pretty good at learning in a broader sense too. So, what does the human brain do and how does it manage to achieve such remarkable results? How do neurons in your brain work differently than transistors in a processor? Naturally, this topic is inexhaustible, but let us try to begin with a few examples.

As you may know, every neuron occasionally sends electrical impulses, otherwise known as spikes, along axons. Neurons never stop and keep sending signals as long as they’re still alive; however, when neurons are “turned off” they rarely send signals. When neurons are triggered, or “turned on”, spikes occur much more frequently.

Neurons function stochastically, meaning they produce electric signals at random intervals. The patterns of these signals can be pretty accurately approximated with a Poisson process. Computers contain logic gates that send signals back and forth, but their synchronization frequency is fixed and by no means random. This frequency is called a computer’s “clock rate”, which has been measured in gigahertz for quite a while now. On every tick, gates on a certain layer send signals up to the next layer. Although this is done a few billion times a second, it’s performed simultaneously, as though the gate were following a strict order.

Actually, it is very easy to see that neurons can synchronize well and count tiny time intervals very precisely. Stereo sound is the simplest and most illustrative example. When you move from one end of the room to the other, you can easily tell, based solely on the sound coming from the television, where you’re going (being able to tell where sound was coming from was crucial for surviving in prehistoric times). You can tell where you’re going by noticing that the sound reaches your left and right ears at different times.

Your inner ears aren’t all that far apart (about 20 cm), and if you divide that by the speed of sound (340 m/s) you get a very short interval — hundredth of milliseconds — between when the sound waves reach each ear. Nevertheless, your neurons pick up on this minor difference excellently, which enables you to figure out precisely where you’re headed. In other words, your brain could process frequency — in kilohertz — just like a computer. Considering the extensive parallel processing performed by your brain, its architecture could generate rather intelligent computational capabilities…but, for some reason, your brain doesn’t do that.

Let us go back to parallel processing for a second. We recognize people’s faces within a few hundred milliseconds, and connections between different neurons are activated within tens of milliseconds, which means that only a few neurons — probably fewer than a dozen — form a serial circuit in the full facial recognition cycle.

On the one hand, the human brain contains an incredible number of neurons, while, on the other hand, it doesn’t have as many layers as a regular processor. Processors have very long serial circuits, while the brain has short and highly parallel circuits. And while a processor core basically works on one thing at a time (but can switch between different tasks with lightning speed, so it appears to you that everything is working at once), the brain can work on a lot of tasks simultaneously, since neurons light up in many areas of the brain when they start recognizing someone’s face or doing some other equally exciting thing.

The illustration above shows how the brain processes a visual signal in time. The light reaches the retina, where it transforms into electrical impulses and then the image is transmitted 20–40 milliseconds later. The first stage takes 10–20 milliseconds (the image shows the cumulative time, i.e. a total of 140–190 milliseconds passes by the time a motor command is issued).

During the second stage, 20–30 milliseconds later, the signal reaches the neurons that recognize simple visual forms. Then there’s another stage, and another, and only during the fourth stage do we see intermediate forms — there are neurons that “light up” when seeing squares, color gradients or other similar objects. Then the brain goes through a few more stages, and neurons capable of discerning high level object descriptions light up 100 milliseconds after the process began. For instance, when you meet someone new a neuron responsible for recognizing her face appears (this is a terrible simplification and we can’t verify this claim but it appears that there is some truth to it). Most likely, a neuron or a group of neurons responsible for this person in general and lighting up whenever you come into contact with her, including when you interact with her not face-to-face, appear. If you see her face again (and the neuron didn’t unlearn or forget this earlier information) that same neuron will be activated ~100 milliseconds later.

Why does the brain work like that? Answering that question with a simple “evolution did it” doesn’t really explain anything. The human brain evolved to a certain point, and that was sufficient to solve problems as we evolved. The rationalist community says that living organisms are not fitness-maximizers who optimize some survival objective function, but rather adaptation executors, who execute “relatively solid” decisions that were chosen randomly at some point. Well, rigid synchronization with a built-in chronometer never took place; however, we can’t tell you exactly why it played out that way.

Actually, in this case, it seems as though asking “why” isn’t all that relevant. It’s better, more interesting, and more productive to ask “how”. How exactly does the brain work? We don’t know for sure but now we can describe the processes going on inside our heads quite well, at least in terms of individual neurons or, in certain instances, groups of neurons.

What can we learn from the brain? First, feature extraction. The brain can learn to make excellent generalizations based on a very, very limited sample size. If you show a young child a table and tell her it’s a table then the child starts calling other tables tables, although they seemingly don’t have anything in common — they could be round, square, have one leg or four. It’s evident that a child doesn’t learn to do this by supervised learning; she obviously lack the training set necessary to do so. One can assume the child created a cluster of “objects with legs on which people place things”. Her brain had already extracted the “Plato’s eidos” and then, when she heard the word for it, she simply attached a label to a ready-made idea.

Naturally, this process can go in the opposite direction, too. Although the neurons (and other things) of many linguists start twitching nervously when they hear the names Sapir and Whorf, one must admit that many ideas, especially abstract ones, are mostly socio-cultural constructs. For instance, every culture has a word similar in meaning to the concept of “love”; however, the sentiment may be very different. American “love” has little in common with that of ancient Japan. Since, generally, all people have the same physiological traits, the abstract idea of “being drawn towards another person” is not simply labeled in language but rather its adjusted and constructed by the texts and cultural data that define it for a person. But let us return to the main point of the article…to be continued next week.

Sergey Nikolenko
Chief Research Officer, Neuromation

Sergey Nikolenko, Chief Research Officer at Neuromation, visits Samsung R&D Office in Seoul — Follow our news!