r40 - 22 Jun 2007 - 10:16:16 - YoshuaBengioYou are here: TWiki >  Public Web  > BabyAISchool

# Baby AI School

## General presentation

The Baby AI Shool project is meant to be a first step in our work towards Artifical Intelligence (AI). The ultimate step will be to feed well-established algorithms with data recorded from TV, so as to automatically "learn the human world". The Baby AI School project is a scaled down problem that should still be challenging enough, while allowing "algorithm-level debugging", i.e. figuring out what is happening.

Up to now, we are focusing on solving the following classification problem: a simple image and a textual question (generated sentence in natural language) related to this image are given and the algorithm has to automatically give the appropriate answer among a list of possible ones. Possible topics include color, shape, size and location, as illustrated below.

Topic Example of question given to the computer Expected answer
Location `Is the blue square at the top or at the bottom?` `At the top`
Location (relative) `There is a purple ellipse. Is it on the right or on the left of the triangle?` `On the left`
Color `There is a small triangle. What color is it?` `Green`
Size (relative) `There is a square on the top left. Is it smaller or bigger than the triangle?` `Bigger`
Size `There is a triangle on the right. Is it rather small or bigger?` `Small`
Shape `What is the shape of the green object?` `Triangle`

To give an idea of the difficulty of the task, we give (approximate) baseline error rates for the different types of questions in the table below. They were determined using classification algorithms with shallow architectures (SVM and Neural Networks) and taking into account 10000 labelled training samples. Note that most of the time, these algorithms give answers in the right topic (i.e. it is easy to detect what the question is about). The indicated Chance Error Rate assumes that the topic of the question is known (the answer is chosen randomly amongst all the possible ones for the given topic).

Topic Chance Error Rate (ER) Baseline ER for 1 object Baseline ER for 2 objects Baseline ER for 3 objects
Color 75% 10% 40% 55%
Shape 66.6% 50% 55% 60%
Size 50% 0.5% -- --
Size (relative) 50% -- 25% 30%
Location 50% 5% -- --
Location (relative) 50% -- 35% 40%
All the topics (randomly picked) 60% 20% 40% 45%

The following animation shows the kind of images we generate with only one object per image.

One of the main ideas in the Baby AI School project is to learn from images with one object, representations that can be used to solve the same problem but when there are several objects. The following section tells more about our viewpoint and our objectives.

## Scientific motivations and goals

The Baby AI School project rests on a number of tenets:

1. The algorithms must deal with very large quantities of training data, i.e. training time should scale linearly.
2. An AI should be able to interact with the most common modalities that humans use: language (through text or sound) and vision, so it should be trained mostly using these modalities.
3. The resulting predictors must have deep architectures so that they can represent high-level abstractions with a greater generalization power. Learning the deeper representations and abstractions can be done more easily if we gradually learn higher-level abstractions on top of lower-level ones, incrementally building upon already learned abstractions through a designed curriculum.
4. As the vast majority of the training data cannot be human-labeled but we should be able to take advantage of small quantities of various sorts of labeled data. To be able to cope with a large number of possible applications and tasks in the future. The first objective of learning should be to discover the statistical structure in the data in a mostly unsupervised way. Supervision is seen as extra hints to suggest useful and relevant abstractions, and possibly to refine representations to maximize predictive performance on specific tasks of economic interest).

### 1. Scaling to Large Data Sets

Since an AI is a highly complex system it will need a very large set of examples to be trained. That means we must have linear-time training algorithms and we should investigate the use of online learning methods for non-linear optimization. We have shown in a submitted paper that the natural gradient direction maximizes the probability of decreasing generalization error and we are developing an on-line approximation that can very significantly beat stochastic gradient descent. Other online or mini-batch second-order optimization methods should also be considered.

### 2. Multiple Modalities: Language and Images

We would like an AI to understand the ``common sense'' concepts that a very young child (say 2-year old) understands. In most human cultures, these concepts are often expressed with language and illustrated with images or video. Our preliminary experiments and intuition both suggest that learning representations for text and for images could work better if we add in the training criterion a term that encourages these representations to be mutually predictive. For this purpose we propose to evaluate ``multi-path'' deep belief networks with a separate path for each modality, with a learning algorithm encouraging each modality's representation to be predictive of the other, and the ability to learn in the presence of data from a single modality (since much data is text-only or images-only). We hope to show that having trained on paired data yields to better models even on tasks involving only one modality.

### 3. Deep Architectures

Because an AI must be implemented with very complex and highly variable functions (that however are not ``random'' but have structure reflecting the structure of the world around us) we cannot rely only on the smoothness assumption that is the underpinning of most non-parametric learning algorithms. We have shown fundamental limitations of Gaussian SVMs and of decision trees in this respect. More generally, complexity theory of circuits suggests fundamental limitations of shallow architectures. This strongly suggests ``learning the kernel'' or learning multiple levels of representation. Inspired by the work of Hinton, we have studied the advantages of Deep Belief Networks in this respect. These models learn factored representations that can efficiently represent highly complex statistical dependencies. We and others have clearly shown the generalization advantages of learning distributed representations for individual words in probabilistic language models: it is now time to extend these ideas to develop learning algorithms that learn distributed representations for word sequences.

An original and central conjecture that we propose to evaluate is that training of such deep architectures could be greatly facilitated by a careful choice of the order in which the training examples are presented to the learning algorithm. The idea is inspired by the way children learn: they first learn simpler concepts and build on them to learn more abstract ones. This process is more efficient in the presence of a teacher that carefully chooses the examples and the curriculum. This motivates us to develop a series of increasingly complex tasks embedded in one another (i.e., they all use the same raw input representation and the concepts learned at one stage are presumed useful in the next stage). For example, we start with images with mostly a single object (and sentences about a single object) and we then move to images with two objects (and sentences about relations between them).

### 4. Unsupervised, Semi-Supervised and Multi-Task Learning

Since we must train on very large quantities of data, it is likely that these examples in their vast majority cannot be labeled by humans for a specific task (although we could certainly consider human productions as data). It means that the algorithms we develop must be able to cope with a vast majority of unlabeled data. However, if we have small labeled data sets, we should also be able to take advantage of them. Current algorithms for Deep Belief Networks can be trained either in a purely unsupervised way or in a semi-supervised way. When labeled examples are available they can be used to fine-tune the representations developed with unlabeled data so as to make them more relevant to the task of interest. One possible issue is that the structure of the input distribution may not be relevant to the task, and we can address that with what we call partially supervised training.

Show attachmentsHide attachments
Topic attachments
I Attachment Action Size Date Who Comment
swf bw1-0.swf manage 47.6 K 18 Jun 2007 - 15:21 JeromeLouradour
png im_3obj.png manage 1.3 K 18 Jun 2007 - 15:23 JeromeLouradour
Edit | WYSIWYG | Attach | Printable | Raw View | Backlinks: Web, All Webs |  | More topic actions
Public.BabyAISchool moved from ProjetTV.PublicBabyAIPage on 21 Jun 2007 - 14:54 by DumitruErhan - put it back