Today we continue the NeuroNuggets series with a new installment. This is the first time when a post written by one of our deep learning researchers was so long that we had to break it up into two parts. In the first part, we discussed the notion of a computational graph and what functionality should a deep learning framework have; we found out that they are basically automated differentiation libraries and understood the distinction between static and dynamic computational graphs. Today, meet again Oktai Tatanov, our junior researcher in St. Petersburg, who will be presenting a brief survey of different deep learning frameworks, highlighting their differences and explaining our choice:

Comparative popularity

Last time, we finished with this graph published by the famous deep learning researcher Andrej Karpathy; it shows comparative popularity of deep learning frameworks in the academic community (mentions in research papers):

Unique mentions of deep learning frameworks in arXiv papers (full text) over time, based on 43K ML papers over last 6 years. Source

We see that the top 4 general-purpose deep learning frameworks right now are TensorFlow, Caffe, Keras, and PyTorch. Today, we will discuss the similarities and differences between them and help you make the right choice of a framework.

Tensorflow

TensorFlow is probably the most famous deep learning framework; it is being developed and maintained by Google. It is written in C++/Python and provides Python, Java, Go and JavaScript API. TensorFlow uses static computational graphs, although a recently released TensorFlow Fold library has added support for dynamic graphs as well. Also, since version 1.7 TensorFlow took a different step towards dynamic execution and implemented eager execution that can evaluate Python code immediately, without building graphs.

At present, TensorFlow has gathered the largest deep learning community around it, so there are a lot of videos, online courses, tutorials, and so on. It offers support for running models on multiple GPUs and can even split a single computational graph over multiple machines in a computational cluster.

Apart from purely computational features, TensorFlow provides an awesome extension called TensorBoard that can visualize the computational graph, plot quantitative metrics about the execution of model training or inference, and basically provide all sorts of information necessary to debug and fine-tune a deep neural network in an easier way.

Plenty of data scientists consider TensorFlow to be the primary software tool of deep learning, but there are also some problems. Despite the big community, learning is still difficult for beginners, and many experts agree that other mainstream frameworks are faster than TensorFlow.

As an example of implementing а simple neural network, look at the following:

import numpy as np
import tensorflow as tf

data_size = 10

input_size = 28 * 28
hidden1_output = 200
output_size = 1

data = tf.placeholder(tf.float32, shape=(data_size, input_size))
target = tf.placeholder(tf.float32, shape=(data_size, output_size))

h1_w1 = tf.Variable(tf.random_uniform((input_size, hidden1_output)))
h2_w1 = tf.Variable(tf.random_uniform((hidden1_output, output_size)))

hidden1_out = tf.maximum(tf.matmul(data, h1_w1), 0)
target_ = tf.matmul(hidden1_out, h2_w1)
loss = tf.losses.mean_squared_error(target_, target)

opt = tf.train.GradientDescentOptimizer(1e-3)
upd = opt.minimize(loss)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    feed = {data: np.random.randn(data_size, input_size), target: np.random.randn(data_size, output_size)}

    for step in range(100):
        loss_val, _ = sess.run([loss, upd], feed_dict=feed)

It’s not so elementary for beginners, but it shows the main concepts in TensorFlow, so let us try to focus on the code structure only first. We begin by defining the computational graph: placeholders, variables, operations (maximum, matmul) and the loss function at the end. Then we assign an optimizer that defines what and how we want to optimize. And finally, we train our graph over and over in a special execution environment called a session.

Unfortunately, if you want to improve the network’s architecture with conditionals or loops (this is especially useful, even essential for recurrent neural networks), you cannot simply use python keywords. As you already know, a static graph is constructed and compiled once, so to add nodes to the graph you should use special control flow or higher order operations.

For instance, to add a simple conditional to our previous example, we have to modify the previous code like this:

# update for https://gist.github.com/Oktai15/4b6617b916c0fa4feecab35be09c1bd6 
a = tf.constant(10)

data = tf.placeholder(tf.float32, shape=(data_size, input_size))
h1_w1 = tf.placeholder(tf.float32, shape=(input_size, hidden1_output))
h2_w1 = tf.placeholder(tf.float32, shape=(input_size, hidden1_output))
 
def first(): return tf.matmul(data, h1_w1)
def second(): return tf.matmul(data, h2_w1)
 
hidden1_out = tf.cond(tf.greater(a, 0), first, second)

Caffe

The Caffe library was originally developed at UC Berkeley; it was written in C++ with a Python interface. An important distinctive feature of Caffe is that one can train and deploy models without writing any code! To define a model, you just edit configuration files or use pre-trained models from the Caffe Model Zoo, where you can find most established state-of-the-art architectures. Then, to train a model you just run a simple script. Easy!

To show how it works (at least approximately), check out the following code:

name: "SimpleCaffeNet"
layer {
  name: "data"
  type: "Input"
  top: "data"
  input_param { shape: { dim: 10 dim: 1 dim: 28 dim: 28 } }
}
layer {
  name: "fc1"
  type: "InnerProduct"
  bottom: "data"
  top: "fc1"
  inner_product_param {
    num_output: 784
  }
}
layer {
  name: "relu"
  type: "ReLU"
  bottom: "fc1"
  top: "fc1"
}
layer {
  name: "fc2"
  type: "InnerProduct"
  bottom: "fc1"
  top: "fc2"
  inner_product_param {
    num_output: 10
  }
}
layer {
  name: "prob"
  type: "Softmax"
  bottom: "fc2"
  top: "prob"
}

We define the neural network as a set of blocks that correspond to layers. At first, we see a data layer where we specify the input shape, then two fully connected layers with ReLU activations. At the end, we have a softmax layer where we get the probability for every class in the data, e.g., 10 classes for the MNIST dataset of handwritten digits.

In reality, Caffe is rarely used for research but is quite often used in production. However, its popularity is waning because there is a new great alternative, Caffe2 which we will touch upon a little when we talk about PyTorch.

Keras

Keras is a high-level neural network library written in Python by Francois Chollet, currently a member of the Google Brain team. It works as a wrapper over one of the low-level libraries such as TensorFlow, Microsoft Cognitive Toolkit, Theano or MXNet. Actually, for quite some time Keras has been shipped as a part of TensorFlow.

Keras is pretty simple, easy to learn and to use. Thanks to brilliant documentation, its community is big and very active, so beginners in deep learning like it. If you do not plan to do complicated research and develop new extravagant neural networks that Keras might not cover, then we heartily advise to consider Keras as your primary tool.

However, you should understand that Keras is being developed with an eye towards fast prototyping. It is not flexible enough for complicated models, and sometimes error messages are not easy to debug. We implemented on Keras the same neural network which we did on TensorFlow. Look:

import numpy as np
import tensorflow as tf

data_size = 10

input_size = 28 * 28
hidden1_output = 200
output_size = 1

data = np.random.randn(data_size, input_size)
target = np.random.randn(data_size, output_size)

model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(hidden1_output, input_shape=(input_size,), activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(output_size))

model.compile(loss=tf.keras.losses.mean_squared_error,
              optimizer=tf.keras.optimizers.SGD(lr=1e-3))

model.fit(data, target, epochs=100, batch_size=data_size)

What immediately jumps out in this example is that our code has been reduced a lot! No placeholders, no sessions, we only write concise informative constructions, but, of course, we lose some extensibility due to extra layers of abstraction.

PyTorch

PyTorch was released by Facebook’s artificial-intelligence research group for Python, based on Torch (previous Facebook’s framework for Lua). It is the main representative of dynamic graph.

PyTorch is pythonic and very developer-friendly. The memory usage in PyTorch is extremely efficient for any neural networks. It is also said to be a bit faster than TensorFlow.

It has a responsive forum where you can ask any question and extensive documentation with a lot of official tutorials and examples, however, the community is still quite smaller as opposed to TensorFlow. Sometimes you can’t find implementation of contemporary model on PyTorch, but easy to see two or three on TensorFlow. Anyway, this framework is considered as a best choice to research.

Quite surprisingly, since May of 2018, PyTorch project was merged with Caffe2, successor of Caffe which actively developed by Facebook for production exactly. It means for supporters these frameworks that bottleneck between researchers and developers will be vanished.

Now look at this code below that shows simple way to “touch” PyTorch:

import torch
import torch.nn as nn
import torch.nn.functional as fun

data_size = 10

input_size = 28 * 28
hidden1_output = 200
output_size = 1

data = torch.randn(data_size, input_size)
target = torch.randn(data_size, output_size)

model = nn.Sequential(
    nn.Linear(input_size, hidden1_output),
    nn.ReLU(),
    nn.Linear(hidden1_output, output_size)
)

opt = torch.optim.SGD(model.parameters(), lr=1e-3)

for step in range(100):
    target_ = model(data)
    loss = fun.mse_loss(target_, target)
    loss.backward()
    opt.step()
    opt.zero_grad()

Here we initialize randomly our trial data and target, then assign model and optimizer. The last block executes training: every time calculates answer from model and change weights with SGD. It looks like Keras: easy read, but we don’t lost ability to write complicated neural networks.

Thanks for dynamic graph, PyTorch are integrated in Python more than TensorFlow. So you can write conditionals and loops like as in ordinary python program.

You can see it when try to realize, for example, simple recurrent block that we represent as hi=hi-1·xi:

import torch

h0 = torch.randn(10)
x = torch.randn(5, 10)
h = [h0]

for i in range(5):
    h_i = h[-1] * x[i]
    h.append(h_i)

The Neuromation choice

Our Research Lab at St. Petersburg mostly prefers PyTorch. For instance, we have used it for computer vision models that we applied to price tag segmentation. Here is a sample result:

But sometimes, especially in cases when PyTorch does not have a ready solution for something yet, we create our models in TensorFlow. The main idea of Neuromation idea is to train neural networks on synthetic data. We are convinced that a great result on real data can be obtained with transfer learning from perfectly labeled synthetic datasets. Have a look at some of our results for the segmentation of retail items based on synthetic data:

Conclusion

There are several deep learning frameworks, and we could go into a lot more detail about which to prefer. But, of course, frameworks are just tools to help you develop neural networks, and while the differences are important they are, of course, secondary. The primary tool in developing modern machine learning solutions is the neural network in your brain: the more you know, the more you think about machine learning solutions from different angles, the better you get. Knowing several deep learning frameworks can also help broaden your horizons, especially when the top contenders are as different as Theano and PyTorch. So it pays to learn them all even if your primary tool has already been chosen for you (e.g., your team uses a specific library). Good luck with your networks!

Oktai Tatanov
Junior Researcher, Neuromation

Sergey Nikolenko
Chief Research Officer, Neuromation

Our sixth installment of the NeuroNuggets series is slightly different from previous ones. Today we touch upon an essential and, at the same time, rapidly developing area — deep learning frameworks, software libraries that modern AI researchers and practitioners use to train all these models that we have been discussing in our first five installments. In today’s post, we will discuss what a deep learning framework should be able to do and see the most important algorithm that all of them must implement.

We have quite a few very talented junior researchers in our team. Presenting this post on neural networks’ master algorithm is Oktai Tatanov, our junior researcher in St. Petersburg:

What a Deep Learning Framework Must Do

A good AI model begins with an objective function. We also begin this essay with explaining the main purpose of deep learning frameworks. What does it mean to define a model (say, a large-scale convolutional network like the ones we discussed earlier), and what should a software library actually do to convert this definition into code that trains and/or applies the model?

Actually, every modern deep learning framework should be able to do the following checklist:

• build and operate with large computational graphs;
• perform inference (forward propagation) and automatic differentiation (backpropagation) on computational graphs;
• be able to place the computational graph and perform the above operations on a GPU;
• provide a suite of standard neural network layers and other widely used primitives that the computational graph might consist of.

As you can see, every point is somehow about computational graphs… but what are those? How does it relate to neural networks? Let us explain.

Computational Graphs: What

Artificial neural networks are called neural networks for a reason: they model, in a very abstract and imprecise way, processes that happen in our brains. In particular, neural networks consist of a lot of artificial neurons (perceptrons, units); outputs of some of the neurons serve as inputs for others, and outputs of the last neurons are the outputs of the network as a whole. Mathematically speaking, a neural network is a very large and complicated composition of very simple functions.

Computational graphs reflect the structure of this composition. A computational graph is a directed graph where every node represents a mathematical operation or a variable, and edges connect these operations with their inputs. As usual with graphs, a picture is worth a thousand words — here is a computational graph for the function :

The whole idea of neural networks is based on connectionism: huge compositions of very simple functions can give rise to very complicated behaviour. This has been proven mathematically many times, and modern deep learning techniques show how to actually implement these ideas in practice.

But why are the graphs themselves useful? What problem are we trying to solve with them, and what exactly are deep learning frameworks supposed to do?

Computational Graphs: Why

The main goal of deep learning is to train a neural network in such a way that it best describes the data we have. Most often, this problem is reduced to the problem of minimizing some kind of loss function or maximizing the likelihood or posterior distribution of a model, i.e., we either want to minimize how much our model gets wrong or want to maximize how much it gets right. The frameworks are supposed to help with these optimization problems.

Modern neural networks are very complicated and non-convex, so basically the only optimization method we have for large neural networks is the simplest and most universal optimization approach: gradient descent. In gradient descent, we basically compute the derivatives of the objective function (the gradient is the vector consisting of all partial derivatives) and then go into the direction where the objective function increases or decreases, as needed. Like this:

There are, of course, many interesting improvements and modifications to this simple idea: Nesterov’s momentum, adaptive gradient descent algorithms that change the learning rate separately for every weight… Perhaps one day we will return to this discussion in NeuroNuggets. But how do we compute the gradient if we have the neural network as model? That’s where computational graphs help…

Computational Graphs: How

To compute the gradient, deep learning frameworks use an algorithm called backpropagation (bprop); it basically amounts to using the chain rule sequentially across the computational graph. Let us walk through an application of backpropagation to our previous example. We begin by computing partial derivatives of every node of the graph with respect to each of its inputs; we assume that it is easy to do, and neural networks do indeed consist of simple units for which it is easy. Like in our example:

Now we need to combine these derivatives with the chain rule. In backpropagation, we do it sequentially from the graph’s output, where the objective function is computed. There we always have

   

Next, for example, we can get

   

since we already know both factors in this formula. Backpropagation means that we go through the graph from right to left, computing partial derivatives of f with respect to every node, including the weights that we are interested in. Here is the final result for our example:

This very simple algorithm allows us to set up algorithms to train any deep neural network. This is exactly what any deep learning framework is supposed to do; they are in reality automatic differentiation libraries more than anything else. The main function of any framework is to compute and take derivatives of huge compositions of functions. Note, by the way, that to compute the function you also need to traverse the computational graph, but this time from left to right, from variables to the outputs; this process is called forward propagation (fprop).

Parallelizing the Computations

Once you have the basic functionality of fprop and bprop in your library, you want to make it as efficient as possible. Efficiency gains mostly come from parallelization: note that operations in one part of the graph are completely independent from what happens in other parts. This means, for instance, that if you have a layer in your neural network, i.e., a set of nodes that do not feed into each other but all receive inputs from previous layers, you can compute them all in parallel during both forward propagation and backpropagation.

This is exactly the insight that to a large extent fueled the deep learning revolution: this kind of parallelization can be done across hundreds or even thousands of computational cores. What kind of hardware has thousands of cores? Why, the GPUs, of course! In 2009–2010, it turned out that regular off-the-shelf GPUs designed for gamers can provide a 10x-50x speedup in training neural networks. This was the final push for many deep learning models and applications into the territory of what is actually computationally feasible. We will stop here for the moment but hopefully will discuss parallelization in deep learning in much greater detail at some future post.

There is one more interesting complication. Deep learning frameworks come with two different forms of computational graphs, static and dynamic. Let us find out what this means.

Static and Dynamic Computational Graphs

The main idea of a static computational graph is to separate the process of building the graph and executing backpropagation and forward propagation (i.e., computing the function). Your graph is immutable, i.e., you can’t add or remove nodes at runtime.

In a dynamic graph, though, you can change the structure of the graph at runtime: you can add or remove nodes, dynamically changing its structure.

Both approaches have their advantages and disadvantages. For static graphs:

• you can build a graph once and reuse it again and again;
• the framework can optimize the graph before it is executed;
• once a computational graph is built, it can be serialized and executed without the code that built the graph.

For dynamic graphs:

• each forward pass basically defines a new graph;
• debugging is easier;
• constructing conditionals and loops is easy, which makes building recurrent neural networks much easier than with static graphs.

We will see live examples of code that makes use of dynamic computational graphs in the next installment, where we will consider several deep learning frameworks in detail. And now let us finish with an overview.

Deep Learning Frameworks: An Overview

On March 10, Andrej Karpathy (Director of AI at Tesla) published a tweet with very interesting statistics about machine learning trends. Here is the graph of unique mentions of deep learning frameworks over the last four years:

Unique mentions of deep learning frameworks in arXiv papers (full text) over time, based on 43K ML papers over last 6 years. Source: https://twitter.com/karpathy/status/972295865187512320

The graph shows that the top 4 general-purpose deep learning frameworks right now are TensorFlow, Caffe, Keras, and PyTorch, while, for example, historically the first widely used framework theano has basically lost traction.

The frameworks have interesting relations between them, and it is worthwhile to consider them all, get a feeling of what the code looks like for each, and discuss their pros and cons. This post, however, is already growing long; we will come back to this discussion in the second part.

Oktai Tatanov
Junior Researcher, Neuromation

Sergey Nikolenko
Chief Research Officer, Neuromation

This week, we continue the NeuroNuggets series with the fifth installment on another important computer vision problem: pose estimation. We have already talked about segmentation; applied to humans, segmentation would mean to draw silhouettes around pictures of people. But what about the skeleton? We need pose estimation, in particular, to understand what a person is doing: running, standing, reading NeuroNuggets?

Today, we present a pose estimation model based on the so-called Part Affinity Fields (PAF), a model from this paper that we have uploaded on the NeuroPlatform as a demo. And presenting this model today is Arseny Poezzhaev, our data scientist and computer vision aficionado who has moved from Kazan to St. Petersburg to join Neuromation! We are excited to see Arseny join and welcome him to the team (actually, he joined from the start, more than a month ago, but the NeuroNuggets duty caught up only now). Welcome:

Introduction

Pose estimation is one of the long-standing problems of computer vision. It has interested researchers over the last several decades because not only is pose estimation an important class of problems itself, but it also serves as a preprocessing step for many even more interesting problems. If we know the pose of a human, we can further train machine learning models to automatically infer relative positions of the limbs and generate a pose model that can be used to perform smart surveillance with abnormal behaviour detection, analyze pathologies in medical practices, control 3D model motion in realistic animations, and a lot more.

Moreover, not only humans can have limbs or a pose! Basically, pose estimation can deal with any composition of rigidly moving parts connected to each other at certain joints, and the problem is to recover a representative layout of body parts from image features. We at Neuromation, for example, have been doing pose estimation for synthetic images of pigs (cf. our Piglet’s Big Brother project):

Traditionally, pose estimation used to be done by retrieving motion patterns from optical markers attached to the limbs. Naturally, pose estimation would work much better if we could afford to put special markers on every human on the picture; alas, our problem is a bit harder. The next point of distinction between different approaches is the hardware one can use: can we use multiple cameras? 3D cameras that estimate depth? infrared? Kinect? is there a video stream available or only still images? Again, each additional source of data can only make the problem easier, but in this post we concentrate on a single standard monocular camera. Basically, we want to be able recognize the poses on any old photo.

Top-Down and Bottom-Up

Pose estimation from a single image is a very under-constrained problem precisely due to the lack of hints from other channels, different viewpoints from multiple cameras, or motion patterns from video. The same pose can produce quite different appearances from different viewpoints and, even worse, human body has many degrees of freedom, which means that the solution space has high dimension (always a bad thing, trust me). Occlusions are another big problem: partially occluded limbs cannot be reliably recognized, and it’s hard to teach a model to realize that a hand is simply nowhere to be seen. Nevertheless, single person pose estimation methods show quite good results nowadays.

When you move from a single person to multiple people, pose estimation becomes even harder: humans occlude and interact with other humans. In this case, it is common practice to use a so-called top-down approach: apply a separately trained human detector (based on object detection techniques such as the ones we discussed before), find each person, and then run pose estimation on every detection. It sounds reasonable but actually the difficulties are almost insurmountable: if the detector fails to detect a person, or if limbs from several people appear in a single bounding box (which is almost guaranteed to happen in case of close interactions or crowded scenes), the whole algorithm will fail. Moreover, the computation time needed for this approach grows linearly with the number of people on the image, and that can be a big problem for real-time analysis of groups of people.

In contrast, bottom-up approaches recognize human poses from pixel-level image evidence directly. They can solve both problems above: when you have information from the entire picture you can distinguish between the people, and you can also decouple the runtime from the number of people on the frame… at least theoretically. However, you are still supposed to be able to analyze a crowded scene with lots of people, assigning body parts to different people, and even this task by itself could be NP-hard in the worst case.

Still, it can work; let us show which pose estimation model we chose for the Neuromation platform.

Part Affinity Fields

In the demo, we use the method based on the “Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields” paper done by researchers from The Robotics Institute at Carnegie Mellon University (Cao et al., 2017). Here is it in live action:

https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FpW6nZXeWlGM%3Ffeature%3Doembed&display_name=YouTube&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DpW6nZXeWlGM&image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FpW6nZXeWlGM%2Fhqdefault.jpg&key=a19fcc184b9711e1b4764040d3dc5c07&type=text%2Fhtml&schema=youtube

It is a bottom-up approach, and it uses the so-called Part Affinity Fields (PAFs) together with estimation of body-part confidence maps. PAFs are the main new idea we introduce today, so let us discuss them in a bit more detail. A PAF is a set of 2D vector fields that encode the location and orientation of the limbs. Vector fields? Sounds mathy… but wait, it’s not that bad.

Suppose you have already detected all body parts (hands, elbows, feet, ankles etc.); how do you now generate poses from them? First, you must find out how to connect two points to form a limb. For each body part, there are several candidates to form a limb: there are multiple people on the image, and there also can be lots of false positives. We need some confidence measure for the association between each body part detection. Cao et al. propose a novel feature representation called Part Affinity Fields that contains information about location as well as orientation across the region of support of the limb.

In essence a PAF is a set of vectors that encodes the direction from one part of the limb to the other; each limb is considered as an affinity field between body parts. Here is a forehand:

If a point lies on the limb then its value in the PAF is a unit vector pointing from starting joint point to ending joint point of this limb; the value is zero if it is outside the limb. Thus, PAF is a vector field that contains information about one specific limb for all the people on the image, and the entire set of PAFs encodes all the limbs for every person. So how do PAFs help us for pose estimation?

Method overview

First, let us go through the overall pipeline of the algorithm.

Figure 2 above illustrates all the steps from an input image (Fig. 2a) to anatomical keypoints as an output (Fig. 2e). First, a feedforward neural network predicts a set of body part locations on the image (Fig. 2b) in the form of a confidence map and a set of PAFs that encode the degree of association between these body parts (Fig. 2c). Thus, the algorithm gets all information necessary for further matching of limbs and people (all of this stuff does sound a bit bloodthirsty, doesn’t it?). Next, confidence maps and affinity fields are parsed together (Fig 1d) to output the final positions of limbs for all people on the picture.

All of this sounds very reasonable: we now have a plan. But so far this is only a plan: we don’t know how to do any of these steps above. So now let us consider every step in detail.

Architecture

One of the core ideas of (Cao et al., 2017) is to simultaneously predict detection confidence map and affinity fields. The method uses a special feedforward network as a feature extractor. The network looks like this:

As you can see, it is split into two branches: the top branch predicts the detection confidence maps and the bottom branch is for affinity fields. Both branches are organized as an iterative prediction architecture, which refines predictions over successive stages. The improvement of accuracy of predictions is controlled by intermediate supervision at each stage. Here is how it might look on a real image:

Before passing input to this two-branch network the method uses auxiliary CNN (first 10 layers of VGG-19) to extract an input feature map F. This prediction is processed by both branches, and their predictions concatenated with initial F are used as input for the next stage (as features).

This process is repeated on every stage, and you can see the refinement process across stages on Figure 4 above.

From Body Parts to Limbs

Take a look at Figure 5 below, which again illustrates the above-mentioned refinement process:

At the end of each stage, the corresponding loss function is applied for each branch to guide the network.

Now consider the top branch; each confidence map is a 2D representation of our confidence that each pixel belongs to a particular body part (we remind that “body parts” here are “points” such as wrists and elbows, and, say, forearms are referred to as “limbs” rather than “body parts”). To get body part candidate regions, we aggregate confidence maps for different people. After that, the algorithm performs non-maximum suppression to obtain a discrete set of parts locations:

During inference, algorithm computes line integrals over all the PAFs along the line segments between pairs of detected body-parts. If the candidate limb formed by connection of certain pair of points is aligned with corresponding PAF then it’s considered as a true limb. This is exactly what the bottom branch does.

From Limbs to Full Body Models

We now see how the algorithm can find limbs on the image between two points. But we still cannot estimate poses because we need the full body model! We need to somehow connect all these limbs into people. Formally speaking, the algorithm has found body part candidates and has scored pairs of these parts (integrating over PAFs), but the final goal is to find the optimal assignment for the set of all possible connections.

Formally speaking, this problem can be viewed as a k-partite graph matching problem, where nodes of the graph are body part detections, and edges are all possible connections between them (possible limbs). Here k-partite matching means that the vertices can be partitioned into k groups of nodes with no connections inside each group (i.e., vertices corresponding to the same body part). Edges of the graph are weighted with part affinities. Like this:

A direct solution of this problem may be computationally infeasible (NP-hard), so (Cao et al., 2017) propose a relaxation where the initial k-partite graph is decomposed into a set of bipartite graphs (Fig. 7d) where the matching task is much easier to solve. The decomposition is based on the problem domain: basically, you know how body parts can connect, and a hip cannot be connected to a foot directly, therefore we can first connect hip to knee and then knee to foot.

That’s all, folks! We have considered all the steps in the algorithm that can retrieve poses from a single raw image. Let us now see how it works on our platform.

Pose Estimation Demo in Action

There are, as always, a few simple steps to run this algorithm on our image of interest:

1. Login at https://mvp.neuromation.io
2. Go to “AI models”
3. Click “Add more” and “Buy on market”:

4. Select and buy the OPENPOSE 2D demo model:

5. Launch it with “New Task” button:

6. Choose the Estimate People On Image Demo:

7. Try the demo! You can upload your own photo for pose estimation. We chose this image from the Mannequin Challenge:

8. And here you go! One can see stick models representing poses of people on the image:

And here is a picture of Neuromation leadership in Singapore:

The results are, again, pretty good:

Sergey Nikolenko
Chief Research Officer, Neuromation

Arseny Poezzhaev
Researcher, Neuromation

We’ve got great news: the very first paper with official Neuromation affiliation has appeared! This work, “3D Molecular Representations Based on the Wave Transform for Convolutional Neural Networks”, has recently appeared in a top biomedical journal, Molecular Pharmaceutics. This paper describes the work done by our friends and partners Insilico Medicine in close collaboration with Neuromation. We are starting to work together with Insilico on this and other exciting projects in the biomedical domain to both significantly accelerate drug discovery and improve the outcomes of clinical trials; by the way, I thank CEO of Insilico Medicine Alex Zhavoronkov and CEO of Insilico Taiwan Artur Kadurin for important additions to this post. Collaborations between top AI companies are becoming more and more common in the healthcare space. But wait — the American Chemical Society’s Molecular Pharmaceutics?! Doesn’t sound like a machine learning journal at all, does it? Read on…

Lead Molecules: Educating the Guess

Getting a new drug to the market is a long and tedious process; it can take many years or even decades. There are all sorts of experiments, clinical studies, and clinical trials that you have to go through. And about 90% of all clinical trials in humans fail even after the molecules have been successfully tested in animals.

But to a first approximation, the process is as follows:

• the doctors study medical literature, in particular associations between drugs, diseases, and proteins published in other papers and clinical studies, and find out what the target for the drug should be, i.e., which protein it should bind with;
• after that, they can formulate what kind of properties they want from the drug: how soluble it should be, which specific structures it should have to bind with this protein, should it treat this or that kind of cancer…
• then they sit down and think about which molecules might have these properties; there is a lot to choose from on this stage: e.g., one standard database lists 72 million molecules, complete with their formulas, some properties and everything; unfortunately, it doesn’t always say whether a given molecule cures cancer, this we have to find out for ourselves;
• then their ideas, called lead molecules, or leads, are actually sent to the lab for experimental validation;
• if the lab says that the substance works, the whole clinical trial procedure can be initiated; it is still very long and tedious, and only a small percentage of drugs actually go all the way through the funnel and reach the market, but at least there is hope.

So where is the place of AI in this process? Naturally, we can’t hope to replace the lab or, God forbid, clinical trials: we wouldn’t want to sell a drug unless we are certain that it’s safe and confident that it is effective in a large number of patients. This certainty can only come from actual live experiments. In the future it is likely that we will be able to go from in silico (in a computer) to patients immediately with the AI-driven drug discovery pipelines but today we need to do the experiments.

Note, however, the initial stage of identifying the lead molecules. At this stage, we cannot be sure of anything, but live experiments in the lab are still very slow and expensive, so we would like to find lead molecules as accurately as we can. After all, even if the goal is to treat cancer there is no hope to check the entire endless variation of small molecules in the lab (“small” are molecules that can easily get through a cell membrane, which means basically everything smaller than a nucleic acid). 72 million is just the size of a specific database, the total number of small molecules is estimated to be between 10⁶⁰ and 10²⁰⁰, and synthesizing and testing a single new molecule in the lab may cost thousands or tens of thousands of dollars. Obviously, the early guessing stage is really, really important.

By now you can see how it might be beneficial to apply latest AI techniques to drug discovery. We can use machine learning models to try and choose the molecules that are most likely to have desired properties.

But when you have 72 million of something, “choosing” ceases to look like classification and gets more into the “generation” part of the spectrum. We have to basically generate a molecule from scratch, and not just some molecule, but a promising candidate for a drug. With modern generative models, we can stop searching for a needle in a haystack and design perfect needles instead:

How do we generate something from scratch? Deep learning does have a few answers when it comes to generative models; in this case, the answer turned out to be…

Generative Adversarial Networks

We have already briefly talked about generative adversarial networks (GANs) in a previous post, but I’m sure a reminder is in order here. GANs are a class of neural networks that aim to learn to generate objects from a certain class. Previously, GANs had been mostly used to generate images: human faces as in (Karras et al., 2017), photos of birds and flowers as in StackGAN, or, somewhat suprisingly, bedroom interiors, a very popular choice for GAN papers due to a commonly used part of the standard LSUN scene understanding dataset. Generation in GANs is based on a very interesting and rather commonsense idea. They have two parts that are in competition with each other:

• the objective of the generator is to generate new objects that are supposed to pass for “true” data points;
• while the discriminator has to decipher the tricks played by the generator and distinguish between real data points and the ones produced by the generator.

Here is how the general scheme looks:

In other words, the discriminator learns to spot the generator’s counterfeit images, while the generator learns to fool the discriminator. I refer to, e.g., this post for a simple and fun introduction to GANs.

We at Neuromation are following GAN research with great interest due to many possible exciting applications. For example, conditional GANs have been used for image transformations with the explicit purpose of enhancing images; see, e.g., image de-raining recently implemented with GANs in this work. This ties in perfectly with our own ideas of using synthetic data for computer vision: with a proper conditional GAN for image enhancement, we might be able to improve synthetic (3D-rendered) images and make them more like real photos, especially in small details. In the post I referred to, we saw how NVIDIA researchers introduced a nice way to learn GANs progressively, from small images to large ones.

But wait. All of this so far makes a lot of sense for images. Maybe it also makes sense for some other relatively “continuous” kinds of data. But molecules? The atomic structure is totally not continuous, and GANs are notoriously hard to train for discrete structures. Still, GANs did prove to work for generating molecules as well. Let’s find out how.

Adversarial Autoencoders

Our recent paper on molecular representations is actually a part of a long line of research done by our good friends and partners, Insilico Medicine. It began with Insilico’s paper “The cornucopia of meaningful leads: Applying deep adversarial autoencoders for new molecule development in oncology”, whose lead author Artur Kadurin is a world-class expert on deep learning, one of Insilico Medicine’s Pharma.AI team on deep learning for molecular biology, recently appointed CEO of Insilico Taiwan… and my Ph.D. student.

In this work, Kadurin et al. presented an architecture for generating lead molecules based on a variation of the GAN idea called Adversarial Autoencoders (AAE). In AAE, the idea is to learn to generate objects from their latent representations. Generally speaking, autoencoders are neural architectures that take an object as input… and try to return the same object as output. Doesn’t sound too hard, but the idea is that in the middle of the architecture, the input must go through a middle layer that learns a latent representation, i.e., a set of features that succinctly encode the input in such a way that afterwards subsequent layers can decode the object back:

Either the middle layer is simply smaller (has lower dimension) than input and output, or the autoencoder uses special regularization techniques, but in any case it’s impossible to simply copy the input through all layers, and the autoencoder has to extract the really important stuff.

So what did Kadurin et al. do? They took a conditional adversarial autoencoder and trained it to generate fingerprints of molecules, using and serving desired properties as conditions. Here is the general model architecture from (Kadurin et al., 2017):

Looks just like the autoencoder above, but with two important additions in the middle:

• on top, there is a discriminator that tries to distinguish the distribution of latent representations from some standard distribution, e.g., a Gaussian; this is the main idea of AAE: if you can make the distribution of latent codes indistinguishable from some standard distribution, it means that you can then sample from this distribution and generate reasonable samples through the decoder;
• on the bottom, there is a condition that in this case encodes desired properties of the molecule; we train on the molecules with known properties, and the problem is then to generate molecules with desired (perhaps even never before seen) combinations of properties.

There is still that nagging question about the representations, though. How do we generate discrete structures like molecules? We will discuss molecular representations in much greater detail in the next post; here let me simply mention that this work used a standard representation of a molecule as a MACCS fingerprint, a set of binary characteristics of the molecule such as “how many oxygens is has” or “does it have a ring of size 4”.

Basically, the problem becomes to “translate” the condition, i.e., desired properties of a molecule, into more “low-level” properties of the molecular structure encoded into their MACCS fingerprints. Then a simple screening of the database can find molecules with the fingerprints most similar to generated ones.

At the time that was the first peer-reviewed paper showing that GANs can generate novel molecules. The submission was made in June 2016 and it was accepted in December 2016. In 2017 the community started to notice:

It turned out that the resulting molecules do look interesting…

Conclusion

This post is getting a bit long; let’s take it one step at a time. We will get to our recent paper in the next installment, and now let us summarize what we’ve seen so far.

Since the deep learning revolution, deep neural networks have been revolutionizing one field after another. In this post, we have seen how modern deep learning techniques are transforming molecular biology and drug discovery. Constructions such as adversarial autoencoders are designed to generate high-quality objects of various nature, and it’s not a huge wonder that such approaches work for molecular biology as well. I have no doubt that in the future, these or similar techniques will bring us closer to truly personalized medicine.

So what next? Insilico has already generated several very promising candidates for really useful drugs. Right now they are undergoing experimental validation in the lab. Who know, perhaps in the next few years we will see new drugs identified by deep learning models. Fingers crossed.

Sergey Nikolenko
Chief Research Officer, Neuromation

In the fourth installment of the NeuroNuggets series, we continue our study of basic problems in computer vision. We remind that in the NeuroNuggets series, we discuss the demos available on the recently released NeuroPlatform, concentrating not so much on the demos themselves but rather on the ideas behind each deep learning model.

In the first installment, we talked about the age and gender detection model, which is basically image classification. In the second, we presented object detection, a more complex computer vision problem where you also have to find where an object is located. In the third, we continued this with segmentation and Mask R-CNN model. Today, we turn to something even more exciting. We will consider a popular and very beautiful application of deep learning: style transfer. We will see how to make a model that can draw your portrait in the style of Monet — or Yves Klein if you’re not careful with training! I am also very happy to present the co-author of this post, one of our lead researchers Kyryl Truskovskyi:

Style Transfer: the Problem

Imagine that you can create a true artwork by yourself, turning your own photo or a familiar landscape into a painting done like Van Gogh or Picasso would do it. Sounds like a pipe dream? With the help of deep neural networks, this dream has now become reality. Neural style transfer has become a trending topic both in academic literature and industrial applications; we all know and use popular mobile apps for style transfer and image enhancement. Starting from 2014, these style transfer apps have become a huge PR point for deep neural networks, and today almost all smartphone users have tried some style transfer app for photos and/or video. By now, all of these methods work in real-time and run on mobile devices, and anyone can create artwork with a simple app, stylizing their own images and videos… but how do these apps work their magic?

From the deep learning perspective, ideas for style transfer stem from attempts to interpret the features that a deep neural network learns and understand how exactly it works. Recall that a convolutional neural network for image processing gradually learns more and more convoluted features (see, e.g., our first post in the NeuroNuggets series), starting from basic local filters and getting all the way to semantic features like “dog” or “house”… or, quite possibly, “Monet style”! The basic idea of style transfer is to try to disentangle these features, pull apart semantic features of “what is on the picture” from “how the picture looks”. Once you do it, you can try to replace the style while leaving the contents in place, and the style can come from a completely different painting.

For the reader interested in a more detailed view, we recommend “Neural Style Transfer: A Review”, a detailed survey of neural style transfer methods. E.g., here is an example from that paper where a photo of the Great Wall of China has been redone in classical Chinese painting style:

And here is an example from DeepArt, one of the first style transfer services:

You can even do it with videos:

But let’s see how it works!

Neural Style Transfer by Optimizing the Image

There are two main approaches to neural style transfer: optimization and feedforward network. Let us begin with optimization.

In this approach, we optimize not the network but the resulting image. This is a very simple but powerful idea: the image itself is also just a matrix (tensor) of numbers, just like the weights of the network. This means that we can take derivatives with respect to these weights, extending backpropagation to them too and optimizing an image for a network rather then the other way around; we have already discussed this idea in “Can a Neural Network Read Your Mind?”.

For style transfer, it works as follows: you have a content image, style image and trained neural network (on ImageNet for example). You create a newly generated image, initializing it completely at random. Then the content image and style image pass through the early and intermediate layers of the network to compute two types of loss functions: style loss and content loss (see below). Next, we optimize their losses by changing the generated image, and after a few iterations we have beautiful stylized images. The structure is a bit intimidating:

But do not worry — let’s talk about the losses. After we pass an image through a network we get a feature map from intermediate layers. This feature map captures the semantic representation of this image. And we definitely want the new image to be similar to the content image, so for content image feature maps C and generated image feature maps P we get the following content loss:

The style loss is slightly different: for the style loss we compute Gram matrices of the intermediate representations of generated images and style image:

The style loss is the Euclidean distance between Gram matrices

By directly minimizing the sum of these losses by gradient descent on our generated image, we make it more and more similar to the style image in terms of style while still keeping the content due to the content loss. We refer to the original paper, “A Neural Algorithm of Artistic Style”, for details. This approach works great, but its main disadvantage is that it takes a lot of computational effort.

Neural Style Transfer Via Image Transformation Networks

The basic idea for the next is to use feed-forward networks for image transformation tasks. Basically, we want to create an image transformation network that would directly create beautiful stylized images, with no complicated “image training” process.

The algorithm is simple. Suppose that, again, you have a content image and a style image. You feed the content image through the image transformation network and get a new generated image. After that, loss networks are used to compute style and content losses, like in the optimization method above, but after that, we optimize not the image itself but the image transformation network. This way, we get a trained image transformation network for style transfer and then can use it for stylizing as only a forward pass without any optimization at all.

Here is how it looks through the entire process with the image transformation network and the loss network:

The original paper on this approach, “Perceptual Losses for Real-Time Style Transfer and Super-Resolution”, led to the first truly popular and the first real-time implementation of style transfer and superresolution algorithms; you can find a sample code for this approach here. The image transformation network works fast, but its main disadvantage is that we need to train a completely separate new network for every style image. This is not a problem for preset instagram-like filters but does not solve the general problem.

To fix this, authors of “A Learned Representation for Artistic Style” introduced an approach called conditional instance normalization. The basic idea is to train one network for several different styles. It turns out that normalization plays a huge role in style networks to model a style, and it is sufficient to specialize scaling and shifting parameters after normalization to each specific style. In simpler words, it is enough just to tune the parameters of a simple transformation after normalization for each style, before image transformation network.

Here is a picture of how it works; for the mathematical details, we refer to the original paper:

You can find an example code for this model here.

In the platform demo, we use the Python framework PyTorch for training the neural networks, Flask + Gunicorne to serve it on our platform, and Docker + Mesos for deploying and scaling.

Try it out on the platform

Now that we know about style transfer, it is even better to see the result for ourselves. We follow the usual steps to get the model working.

1. Login at https://mvp.neuromation.io
2. Go to “AI Models”:

3. Click “Add more” and “Buy on market”:

4. Select and buy the Image Style Transfer demo model:

5. Launch it with the “New Task” button:

6. Try the demo! You can upload your own photo for style transfer. We chose this image:

Neuromation Team in Singapore

7. And here you go!

Stylized and enhanced Neuromation Team in Singapore

And here we are:

Sergey Nikolenko
Chief Research Officer, Neuromation

Kyryl Truskovskyi
Lead Researcher, Neuromation

In the third post of the NeuroNuggets series, we continue our study of basic problems in computer vision. I remind the reader that in this series we discuss the demos available on the recently released NeuroPlatform, concentrating not so much on the demos themselves but rather on the ideas behind each deep learning model. This series is also a great chance to meet the new Neuromation deep learning team that has started working at our new office in St. Petersburg, Russia.

In the first installment, we talked about the age and gender detection model, which is basically image classification. In the second, we presented object detection, a more complex computer vision problem where you also have to find where an object is located. Today, we continue with segmentation, the most detailed problem of this kind, and consider the latest addition to the family of R-CNN models, Mask R-CNN. I am also very happy to present the co-author of this post, Anastasia Gaydashenko:

Segmentation: Problem Setting

Segmentation is a logical next step of the object detection problem that we talked about in our previous post. It still stems from the same classical computer vision conundrum: even with great feature extraction, simple classification is not enough for computer vision, you also have to understand where to extract these features. Given a photo of your friends, a landscape scene, or basically any other image, can you automatically locate and separate all the objects in the picture? In object detection, we looked for bounding boxes, i.e., rectangles that enclose the objects. But what if we require more detail and label the exact silhouettes of the objects, excluding background? This problem is called segmentation. Generally speaking, we want to go from pictures in the first row to pictures in the second row on this picture:

Formally speaking, we want to label each pixel of an image with a certain class (tree, road, sky, etc) as shown in the image. The first question, of course, is why? What’s wrong with regular object detection? Actually, segmentation is applied widely: in medical imaging, сontent-based image retrieval, and so on. It avoids the big problem of regular object detection: overlapping bounding boxes for different objects. If you see three heavily overlapping bounding boxes, are these three different hypotheses for the same object (in which case you should choose one) or three different objects that happen to occupy the same rectangle (in which case you should keep all three)? Regular object detection models can’t really decide.

And if the shape of the object is far from rectangular segmentation provides much better information (this is very important for medical imaging). For instance, Google used semantic image segmentation to create the Portrait Mode in its signature Pixel 2 phone.

These pictures also illustrate another important point: the different flavours of segmentation. What you see above is called semantic segmentation; it’s the simpler version, when we simply want to classify all pixels to categories such as “person”, “airplane”, or “background”. You can see in the picture above that all people are labeled as “person”, and the silhouettes blend together into a big “cuddle pile”, to borrow the expression used in certain circles.

This leads to another, more detailed type of segmentation: instance segmentation. In this case, we would want to separate the people in the previous photo and label them as “person 1”, “person 2”, etc., as shown below:

Here all people on the photo are marked in different colors, which mean different instances. Note also that they are labeled with probabilities that reflect the model’s confidence in a particular class label; the confidence is very high in this case, but generally speaking it is also a desirable property of any AI model to know when it is not sure.

Convolutions and Convolutional Neural Networks

So how can a computer vision system parse these images in such an accurate and humanlike way? Let’s find out. The first thing we need to study is how convolution works and why do we use it. And yes, we return to CNNs in each and every post — because they are really important and we keep finding new things to say about them. But before we get to the new things, let’s briefly go through the old, concentrating on the convolutions themselves this time.

Initially, the idea of convolution has come from biology, or, more specifically, studies of the visual cortex. David Hubel and Torsten Wiesel studied the lower layers of the visual cortex in cats. Cat lovers, please don’t watch this:

https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FYoo4GWiAx94%3Ffeature%3Doembed&url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DYoo4GWiAx94&image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FYoo4GWiAx94%2Fhqdefault.jpg&key=a19fcc184b9711e1b4764040d3dc5c07&type=text%2Fhtml&schema=youtube

When Hubel and Wiesel moved a bright line across a cat’s retina, they noticed an interesting effect: the activity of neurons in the brain changed depending on the orientation of the line, and some of the neurons fired only when the line was moving in a particular direction. In simple terms, that means that different regions of the brain react to different simple shapes. To model this behaviour, researchers had to invent detectors for simple shapes, elementary components of an image. That’s how convolutions appeared.

Formally speaking, convolution is just a scalar product of two matrices taken as vectors: we multiply them componentwise and sum up the results. The first matrix is called the “kernel”; it represents a simple shape, like this one:

The second matrix is just some patch from the picture where we want to find the pattern shown in the kernel. If the convolution result is large, we decide that the target shape is indeed present in this patch. Then we can simply run the convolution through all patches in the image and detect where the pattern occurs.

For example, let us try to find the filter above in a picture of a mouse. Here you can see the part where we would expect the filter to light up:

And if we multiply the corresponding submatrix with the kernel and sum all the values, we indeed get a pretty big number:

On the other hand, a random part of the picture does not produce anything at all, which means that it is totally different from the filter’s pattern:

Filters like this let us detect some simple shapes and patterns. But how do we know which of these forms we want to detect? And how can we recognize a plane or an umbrella from these simple lines and curves?

This is exactly where the training of the neural networks comes in. The shapes defined by kernels (the filters) are actually what the CNN learns from the training set. It is usually simple lines and gradients on the first layers of the neural network, but then, with each layer of the model, these shapes are combined with one another into recognizable kernels; a set of kernels is called a map:

The other operation necessary for CNNs is pooling. It is mostly used to reduce computational costs and suppress noise. The main idea is to cover a matrix with small submatrices and leave only one number in each, thus reducing the dimension; usually, the result is just a maximum or average of the values in the small submatrix. Here is a simple example:

This kind of operations is also sometimes called downsampling as they reduce the amount of information we store.

The R-CNN models and Mask R-CNN

Above, we have seen a simplified explanation of techniques used to create convolutional neural networks. We have not touched upon learning at all, but that part is pretty standard. With these simple tools, we can teach a computer model to recognize all sorts of shapes; moreover, CNNs operate on small patches so they are perfectly parallelizable and well suited for GPUs. We recommend our previous two posts for more details on CNNs, but let us now move on to the more high-level things, that is, segmentation.

In this post, we do not discuss all modern segmentation models (there are quite a few) but go straight to the model used in the segmentation demo on the NeuroPlatform. This model is called Mask R-CNN, and it is based on the general architecture of R-CNN models that we discussed in the previous post about object detection; hence, a brief reminder is again in order.

It all begins with the R-CNN model, where R stands for “region-based”. The pipeline is pretty simple: we take a picture and apply an external algorithm (called selective search) to it, searching for all kind of objects. Selective search is a heuristic method that extracts regions based on connectivity, color gradients, and coherence of pixels. Next, we classify all extracted regions with some neural network:

Due to the high number of proposals, R-CNN worked extremely slow. In Fast R-CNN, the RoI (region of interest) projection layer was added to the neural network: instead of putting each region from proposals through the whole network, Fast R-CNN takes the whole image through the network once, finds neurons corresponding to a particular region in the feature map in the network, and then applies the remaining part of the network to each found set of neurons. Like here:

The next step was to invent the Region Proposal Network that could replace selective search; the Faster R-CNN model is now a complete end-to-end neural network.

You can read about all these steps in more detail in our object detection post. But we’re here to learn how Faster R-CNN can be converted to solve the segmentation problem! Actually, it is extremely simple but nevertheless efficient and functional. The authors just added a parallel branch for predicting an object mask to the original Faster R-CNN model. Here is how the resulting Mask R-CNN model looks like:

The top branch in the picture predicts the class of some region and the bottom branch tries to label each pixel of the region to construct a binary mask (i.e., object vs. background). It only remains to understand where this binary mask comes from.

Fully Convolutional Networks

Let us take a closer look at the segmentation part. It is based on a popular architecture called Fully Convolutional Network (FCN):

The FCN model can be used for both image detection and segmentation. The idea is pretty straightforward, and the network is actually even simpler than usual, but it’s still a deep and interesting idea.

In standard deep CNNs for image classification, the last layer is usually a vector of the same size as the number of classes that shows the “scores” of different classes that could be then normalized to give class probabilities. This is what happens in the “class box” on the picture above.

But that if we stop at some middle layer of the CNN and instead of vectors do some more convolutions? And on the last convolutional layer, we get the same number of features as the number of classes. Then, after proper training, we can get “class scores” in every pixel of the last layer, getting a kind of a “heatmap” for every class! Here is how it works — regular classification on top and a fully convolutional approach on the bottom:

For segmentation via this network, we will use the inverses of convolution and pooling. Meet… deconvolution and unpooling!

In deconvolution, we basically do convolution but the matrix is transposed, and now the output is a window rather than a number. Here are two popular ways to do deconvolution (white squares are zero paddings), animated for your viewing convenience:

To understand unpooling, recall the pooling concept that we discussed above. To do max-pooling, we take the maximum value from some submatrix. Now we want to also remember the coordinates of the cells from which we took it and then use it to “invert” max-pooling. We create the matrix with the same shape as the initial and put maximums to the corresponding cells, reconstructing other cell values with approximations based on known cells. Some information stays lost, of course, but usually upsampling works pretty well:

Through the use of deconvolution and unpooling, we can construct pixel-wise predictions for each class, that is, segmentation masks for the image!

Segmentation demo in action

We have seen how Mask R-CNN does segmentation, but it is even better to see the result for yourself. We follow the usual steps to get the model working.

1. Login at https://mvp.neuromation.io
2. Go to “AI Models”:

3. Click “Add more” and “Buy on market”:

4. Select and buy the Object Segmentation demo model:

5. Launch it with the “New Task” button:

6. Try the demo! You can upload your own photo for segmentation. We chose this image:

7. And here you go! The model shows bounding boxes as well, but now it gives much more than just bounding boxes:

Sergey Nikolenko
Chief Research Officer, Neuromation

Anastasia Gaydashenko
Junior Researcher, Neuromation

Neuromation’s global team gathered in Tel-Aviv at the Future of AI conference to present our concept of Knowledge Mining to the international AI market. Israel has taken a promising position in AI, creating nearly 700 AI jobs, of which only 300 were filled.

This is an example of the growing demand for AI talent and products that Neuromation is filling through its distributed marketplace Platform, and its custom turn-key solutions provided by Neuromation Labs.

Stay with us, more reports from the event, including the recording of the keynote speech by Maxim Prasolov, CEO and Sergey Nikolenko, CRO, are coming soon.

This is the second post in our NeuroNuggets series. This is a series were we discuss the demos already available on the recently released NeuroPlatform. But it’s not as much about the demos themselves as about the ideas behind each of these models. We also meet the new Neuromation deep learning team that we hired at our new office in St. Petersburg, Russia.

The first installment was devoted to the age and gender estimation model, the simplest neural architecture among our demos, but even there we had quite a few ideas to discuss. Today we take convolutional neural networks for image processing one step further: from straightforward image classification to object detection. It is also my pleasure to introduce Aleksey Artamonov, one of our first hires in St. Petersburg, with whom we have co-authored this post:

Object Detection: Not Quite Trivial

Looking at a picture, you can recognize not only what objects are on it, but also where they are located. On the other hand, a simple convolutional neural network (CNN) like the one we considered in the previous post cannot do that. All they can is estimate the probability with which a given object is present on the image. For practical purposes, this is insufficient: on real world images, there are lots of objects that can interact with each other in nontrivial ways. We need to ascertain the position and class of each object in order to extract more semantic information from the scene. We have already seen it with face detection:

The simplest approach that was used before the advent of convolutional networks consisted of a sliding window and a classifier. If we need, say, to find human faces on a photo, we first train a network that says how likely it is that a given picture contains a face and then apply it to every possible bounding box (a rectangle in the photo where an object could appear), choosing the bounding boxes where this probability is highest.

This approach actually would work pretty well… if anyone could apply it. Detection with a sliding window needs to look through a huge amount of different bounding boxes. Bounding boxes have different positions, aspect ratios, scales. If we try to calculate all the options that we should look through, we get about 10 million combinations for a 1Mpix image. Which means that the naive approach would need to run the classification network 10 million times to detect the actual position of the face. Naturally, this would never do.

Classical Computer Vision: the Viola-Jones Algorithm

Our next stop is an algorithm that embodies classical computer vision approaches to object detection. By “classical” here we mean computer vision as it was before the deep learning revolution made every kind of image processing into different flavours of CNNs. In 2001 Paul Viola and Michael Jones proposed an algorithm for real-time face detection. It employs three basic ideas:

• Haar feature selection;
• boosting algorithm;
• cascade classifier.

Before describing these stages, let us make sure that we actually want the algorithm to achieve. A good object detection algorithm has to be fast and have a very low false-positive rate. We have 10 million possible bounding boxes and only a handful of faces on the photo, so we cannot afford a false positive rate much higher than 10–6 unless we want to be overwhelmed by incorrect bounding boxes. With this in mind, let us jump into the algorithm.

The first part is the Haar transform; it is best to begin with a picture:

We overlap different filters on our image. Activation of one Haar filter is the sum of the values in the white parts of the rectangle minus the sum of values under black part.

The main property of these filters is that they can be computed across the entire image very quickly. Let us consider the integral version (I*) of the original image (I); the integral version is the image where intensity at coordinate I*x,y is the total intensity over the whole rectangle that begins at the top left corner and ends in (x,y):

Let us see the Haar filter overlapped on the image and its integral version in action:

Haar features for this filter could be computed in just a few operations. E.g., the horizontal Haar filter activation shown on the Terminator’s face equals 2C — 2D + B — A + F — E, where the letters denote intensities at the corresponding points on the integral image. We won’t go into the formulas but you can check it yourself, it’s a good exercise for understanding the transform.

To select the best Haar features for face recognition, the Viola-Jones algorithm uses the AdaBoost classification algorithm. Boosting models (in particular, AdaBoost) are machine learning models that combine and build upon simpler classifiers. AdaBoost can take weak features like Haar that are just a little better than tossing a coin. But then AdaBoost learns a combination of these features in such a way that the final decision rule is much stronger and better. It would take a whole separate post to explain AdaBoost (perhaps we should, one day), so we’ll just add a couple of links and leave it at that.

The third, final idea is to combine the classifiers into a cascade. Thing is, even boosted classifiers still have too much false positives. A simple 2-feature classifier can achieve almost 100% detection rate, but with a 50% false positive rate. Therefore, the Viola-Jones algorithm uses a cascade of gradually more complex classifiers, where we can reject a bounding box on every step but have to pass all checks to output a positive answer:

This approach leads to much better detection rates. Roughly speaking, if we have 10 stages in our cascade, each stage has 0.3 false positive rate and 0.01 false negative rate, and all stages are independent (this is a big assumption, of course, but in practice it still works pretty well), the resulting cascade classifier achieves (0.3)¹⁰ ~ 3*10^–6 false positive rate and 0.9 detection level. Here is how a cascade works on a group photo:

Further research in classical computer vision went into detecting objects of specific classes such as pedestrians, vehicles, traffic signs, faces etc. For complex detectors on deep phases of a cascade we can use different kinds of classifiers such as histogram of oriented gradients (HOG) or support vector machines (SVM). Instead of using Haar features on image in grayscale we can get image channels in different color schemes (CIELab or HSV) and image gradients in different directions. All these features are computed in the integral space, summing inside a rectangle with an adjustable threshold.

An important problem that appeared already in classical computer vision is that near a real object, our algorithms will find multiple intersecting bounding boxes; you can draw different rectangles around a given face, and all will have rather high confidence. To choose the best one classical computer vision usually employs the non-maximum suppression algorithm. However, this still remains an open and difficult problem because there are many situations when two or more objects have intersecting bounding boxes, and a simple greedy implementation of non-maximum suppression would lose good bounding boxes. This part of the problem is relevant for all object detection algorithms and still remains an active area of research.

R-CNN

Initially, in object detection tasks neural networks were treated as tools for extracting features (descriptors) on late stages of the cascade. Neural networks by themselves had always been very good in image classification, i.e., prediction of the class or type of the object. But for a long time, there was no mechanism to locate this object at the image with neural networks.

With the deep learning revolution, it all changed rather quickly. By now, there are several competing approaches to object detection that are all based on deep neural networks: YOLO (“You Only Look Once”, a model that was initially optimized for speed but ), SSD (single-shot detectors), and so on. We may return to them in later installments, but in this post we concentrate on a single class of object detection approaches, the R-CNN (Region-Based CNN) line of models.

The original R-CNN model, proposed in 2013, performs a three-step algorithm to do object detection:

• generate hypotheses for reasonable bounding boxes with an external region proposal algorithm;
• warp each proposed region in the image and pass it through a CNN trained for image classification to extract features;
• pass the resulting features to a separate SVM classification model that actually classifies the regions and chooses which of them contain meaningful objects.

Here is an illustration from the R-CNN paper:

R-CNN brought the deep learning revolution to object detection. With R-CNN, mean average precision on the Pascal VOC (2010) dataset grew from 40% (the previous record) up to 53%, a huge improvement.

But improved object detection quality is only part of the problem. R-CNN worked well but was hopelessly slow. The main problem was that you had to run the CNN separately for every bounding box; as a result, object detection with R-CNN took more than forty seconds on a modern GPU for a single image! Not quite real-time. It is also very hard to train because you had to juggle together the three components, two of which (CNN and SVM) are machine learning models that you have to train separately. And, among other things, R-CNN requires an external algorithm that can propose bounding boxes for further classification. In short, something had to be done.

Fast R-CNN

The main problem of R-CNN was speed, so when researchers from Microsoft Research rolled out an improvement there was no doubt how to name the new model: Fast R-CNN was indeed much, much faster. The basic idea (we will see this common theme again below) was to put as much as they could directly into the neural network. In Fast R-CNN, the neural network is used for classification and bounding box regression instead of SVM:

In order to make detection independent of the size of the object in the image, R-CNN uses Spatial Pyramid Pooling (SPP) layers that had been introduced in SPPnet. The Idea of SPP is brilliant: instead of cropping and warping the region to construct an input image for a separate run of the classification CNN, SPP crops the region of interest (RoI) projection at deeper convolutional layer, before fully-connected layers.

This means that we can reuse lower layers of the CNN, running it only once instead of a thousand times in basic R-CNN. But a problem arises: a fully-connected layer has a certain size, and the size of our RoI can be anything. The task of the spatial pyramid pooling layer is to solve this problem. The layer divides the window into 21 parts, as shown in the figure below, and summarizes (pools) the values in each part. Thus, the size of the layer’s output does not depend on the size of the input window any more:

Fast R-CNN is 200 times faster than R-CNN to apply to a test image. But it is still insufficient for actual real-time object detection due to an external algorithm for generating bounding box hypotheses. On a real photo, this algorithm can take about 2 seconds, so regardless of how fast we make the neural network, we have at least a 2 second overhead for every image. Can we get out of this bottleneck?

Faster R-CNN

What could be faster than Fast R-CNN? The aptly named Faster R-CNN, of course! And what kind of improvement could we do to Fast R-CNN to get an even faster model? It turns out that we can get rid of the only external algorithm left in the model: extracting region proposals.

The beauty of Faster R-CNN is that we can use the same neural network to extract region proposals. We only need to augment it with a few new layers, called the Region Proposal Network (RPN):

The idea is that the first (nearest to input) layers of a CNN extract universal features that could be useful for everything, including region proposals. On the other hand, the first layers have not yet lost all the spatial information: these features are still rather “local” and correspond to relatively small patches of the image. Thus, RPN uses precomputed feature values from early layers to propose regions with objects for further classification. RPN is a fully convolutional network, and there are no fully-connected layers in the RPN architecture, so the computing overhead is almost nonexistent (10ms per image), and we can now completely remove the region proposal algorithm!

One more good idea is to use anchor boxes with different scales and aspect ratios instead of a spatial pyramid. At the time of its appearance, Faster R-CNN became the state of the art object detection model, and while there has been a lot of new research in the last couple of years, it is still going pretty strong. You can try Faster R-CNN at the NeuroPlatform.

Object Detection at the NeuroPlatform

And finally we are ready to see Faster R-CNN in action! Here is a sequence of steps that will show you how this works on a pretrained model which is already available at the NeuroPlatform.

1. Login at https://mvp.neuromation.io
2. Go to “AI Models”:

3. Click “Add more” and “Buy on market”:

4. Select and buy the object detection demo model:

5. Launch it with the “New Task” button:

6. Try the demo! You can upload your own photo for detection:

7. And here you go!

Sergey Nikolenko
Chief Research Officer, Neuromation

Alexey Artamonov
Senior Researcher, Neuromation

Today, we begin a new series of posts that we call NeuroNuggets. On February 15, right on time, we released the first version of the NeuroPlatform. So far it is still in alpha, and it will take a lot of time to implement everything that we have planned. But even now, there are quite a few cool things you can do. In the NeuroNuggets series, we will present these cool things one by one, explaining not only the technicalities of how to run something on the platform but also the main ideas behind every model. This is also my chance to present my new deep learning team hired at our new office in St. Petersburg, Russia.

In this post, we present our first installment: the age and gender estimation model. This is the simplest neural architecture among our demos, but even this network will have quite a few tricks to explain. And it is my pleasure to introduce Rauf Kurbanov, one of our first hires in St. Petersburg, with whom we have co-authored this post:

Who hired a nerd?

AI researchers tend to question the nature of intuitive. As soon as you ask how a computer can do the same thing that seems too easy for humans, you see that what is “intuitively clear” for us can be very hard to formalize. Our visual perception of human age and gender is a good example of such a subtle quality.

To us AI nerds, Eliezer Yudkowsky is familiar both as an AI safety researcher and the author of the most popular Harry Potter fanfic ever (we heartily recommend “Harry Potter and the Method of Rationality”, HPMoR for short, to everyone). And the Harry Potter series features a perfect example for this post, a fictional artifact that appears intuitively clear but is hard to reproduce in practice:

Albus Dumbledore had placed an Age Line around the Goblet of Fire to prevent anyone under the age of seventeen from approaching it. Age Line magic was so advanced that even an Ageing Potion could not fool it. Even Yudkowsky did not really dig into the mechanics of Age Line in his meticulous manner in HPMoR but today we will give it a try; and while we are on the subject, we will give a shot to gender recognition as well. As usual in computer vision, we begin with convolutional neural networks.

Convolutional neural networks

A neural network, as the name suggests, is a machine learning approach which is in a very abstract way modeled after how the brain processes information. It is a network of learning units called artificial neurons, or perceptrons. During training, the neurons learn how to convert input signals (say, the picture of a cat) into corresponding output signals (say, the label “cat” in this case), training automated recognition from real life examples.

Virtually all computer vision nowadays is based on convolutional neural networks. Very roughly speaking, CNNs are multilayer (deep) neural networks where each layer processes the image in small windows, extracting local features. Gradually, layer by layer, local features become global, able to draw their inputs from a larger and larger portion of the original image. Here is how it works in a very simple CNN (picture taken from this tutorial, which we recommend to read in full):

In the end, after several (sometimes several hundred) layers we get global features that “look at” the whole original image, and they can now be combined in relatively simple ways to obtain class labels (recognize whether it is a dog, cat, boat, or Harry Potter).

Technically, a convolutional neural network is a neural network with convolutional layers, and a convolutional layer is a transformation that applies a certain kernel (filter) to every point in the input (a “picture” with multiple channels in every pixel, i.e., a three-dimensional tensor) and generate filtered output by sliding the kernel over the input.

Let us consider a simple example of a filter: edge detection in images. In this case, the input for edge detection is an image, and each pixel in the image is defined by three numbers: the intensities of red, green, and blue in the color of that pixel. We construct a special kernel which will be applied to every pixel in the image; the output is a new “image” that shows the results of this kernel. Basically, the kernel here is a small matrix. Here’s how it works:

The kernel is sliding over every pixel in the image and the output value increases whenever there is an edge, an abrupt change of colors. In the figure above, after multiplying this simple matrix element-wise to every 3×3 window in the image we get a very nice edge detection result.

Once you understand filters and kernels, it becomes quite simple to explain convolutional layers in neural networks. You can think of them as vanilla convolutions, as in the edge detection example above, but now we are learning convolutional kernels end-to-end when training the networks. That is, we do not have to invent these small matrices by hand anymore but can automatically learn matrices that extract the best features for a specific task.

The model pipeline

Age and gender estimation sounds like a traditional machine learning task: binary classification for the genders (it might stir up some controversy but yeah, our models live in a binary world) and regression for the ages. But before we can begin to solve these problems, we need to find the faces on the photo! Classification will not work on the picture as a whole because it might, for example, contain several faces. Therefore, the age and gender estimation problem is usually broken down into two steps, face detection and age/gender estimation for the detected faces:

In the model that you can find on the NeuroPlatform, these steps are performed independently and are not trained end-to-end, so let us discuss each of them in particular.

Face detection

Face detection is a classic problem in computer vision. It was solved quite successfully even before the deep learning revolution, in early 2000s, by the so-called Viola-Jones algorithm. It was one of the most famous application of Haar cascades as features; but those days are long gone…

Today, face detection is not treated as a separate task that requires individual approaches. It is also solved by convolutional neural networks. To be honest, since the advent of deep learning it has long become clear that CNNs kick ass at object detection, and therefore we expect modern solution to an old problem to be based on CNNs as well. And we are not wrong.

But in real world machine learning, you should also consider other properties beside detection accuracy such as simplicity and inference speed. If a simpler approach works well enough, it might not be worth it to introduce very complicated models to gain a few percentage points (remind me to tell you about the Netflix prize challenge results later). Therefore, in the NeuroPlatform demo we use a more classical approach to face detection while keeping CNNs for the core age/gender recognition task. But we can learn a lot from classical computer vision too.

In short, the face detection model can be described as an SVM on HOG + SIFT feature representation. HOG and SIFT representations are hand-crafted features, the result of years of experience in building image recognition systems. These features recognize gradient orientations in localized portions of an image and perform a series of deterministic image transformations. It turns out this representation works quite well with kernel methods such as support vector machines (SVM).

Data augmentation

Here at Neuromation, we are big fans of using synthetic data for computer vision. Usually, this means that we generate sophisticated synthetic datasets from 3D computer graphics and even look towards using Generative Adversarial Networks for synthetic data generation in the future. But let us not forget the most basic tool for increasing the datasets: data augmentation.

Since we have already extracted faces on the previous step, it is enough to augment only the faces, not the whole image. In the demo, we are using standard augmentation tricks such as horizontal/vertical shifts and mirroring alongside with a more sophisticated one of randomly erasing patches of the image.

Age estimation

To predict the age, we apply a deep convolutional neural network to the face image detected on the previous processing stage. The method in the demo uses the Wide Residual Network (WRN) architecture which beat Inception architecture on mainstream object detection datasets, achieving convergence on the same task twice faster. Before we explain what residual networks are, we begin with a brief historical reference.

The ImageNet challenge

The ImageNet project is a large visual database designed to be used in visual object recognition research. The deep learning revolution of the 2010s started in computer vision with a dramatic advance in solving the ImageNet Challenge. Results on ImageNet were recognized not only within the AI community but across the entire industry, and ImageNet has become and still remains the most popular general purpose computer vision dataset. Without getting into too much details, let us just take a glance at a comparison plot between the winners of a few first years:

On the plot, the horizontal axis shows how computationally intensive a model is, the circle size indicates the number of parameters, and the vertical axis shows image classification accuracy on ImageNet. As you can see, ResNet architectures show some of the best results while staying on the better side of efficiency as well. What is their secret sauce?

Residual connections

It is well known that deeper models perform better than shallower models, they are more expressive. But optimization in deep models is a big problem: deeper models are harder to optimize due to the peculiarities of how gradients propagate from the top layers to the bottom layers (I hope one day we will explain it all in detail). Residual connections are an excellent solution to this problem: they add connections “around” the layers, and the gradient flow is now able to “skip” excessive layers during backpropagation, resulting in much faster convergence and better training:

Essentially the only difference with Wide Residual Networks is that in the original paper the tradeoff between width and depth of architecture was studied more carefully resulting in more efficient architecture with better convergence speed.

DEX on NeuroPlatform

We have wrapped the DEX model into a docker container and uploaded in to the NeuroPlatform. Let’s get started! First, enter Neuromation login page https://mvp.neuromation.io/

On your dashboard on the NeuroPlatform, you can see how much NTK is left on your balance and spend them in three sections:

• AI Models,
• Datasets,
• Generators.

Today we are dealing with the Age and Gender estimator, an AI model available at the NeuroMarket. Let us purchase our first model! We enter AI models and buy the Age&Gender model:

Then we request a new instance; it may take a while:

And here we go! We can now try the model on a demo interface:

It does tend to flatter me a bit.

Sergey Nikolenko
Chief Research Officer, Neuromation

Rauf Kurbanov
Senior Researcher, Neuromation