Statistical Learning of Complex Data with Complex Distributions

Statistical Learning of Complex Data with Complex Distributions

Pr. Yoshua Bengio
Canada Research Chair in Statistical Learning Algorithms
Adaptative Computer Science Laboratory (LISA)
Departement of Computer Science and Operations Research
Université de Montréal


As we move ever more into a digital age, organizations and individuals are acquiring increasing quantities of data, and advances in computer technology make it possible to efficiently process gigabytes or even terabytes of data. But algorithms able to understand that deluge of data are lagging. This is making it all the more important to invest in research to develop better automated tools for interpretation and analysis of data, so that it can be used to take decisions.

The field of statistical learning can be viewed as a set of theories and techniques whose purpose is precisely the discovery of useful regularities from a data source, for improved predictions or understanding. These algorithms are often rather general in nature, able to work on data of different types, from the array of pixels measured by a camera to the values in a retail database. In most cases, the raw input to these algorithms is a flow of "examples", each of which being characterized by many observed variables, related by unknown intricate statistical relationships.

Statistical machine learning is a core component of data-mining, a corpus of techniques applied in industrial settings generally involving large quantities of data, with the objective of extracting something useful from the data. Mathematically, one can think of a learning algorithm as an object that maps a data set into a function. The data set is a collection of examples (possibly ordered). The learned function is something we can use to answer questions, typically about new examples coming from the same underlying and unknown process that generated our data set.

Recent years have seen impressive advances in statistical learning methodology and data-mining, with a plethora of applications in many areas of science, industry, and business. However, despite these advances, the progress offered by machine learning to many long-standing problems remains relatively modest. Typically, these harder problems involve learning from data with very rich statistical structure. In practice, and especially with most data-mining applications, traditional machine learning and statistics research has focused on relatively simple problems. The data distribution was explicitly or implicitly assumed to have a rather simple form, either because it is known ahead of time up to a few parameters, because it is assumed to be rather smooth, or because it only involves a few interactions between variables. Another challenge arises because data is almost always incomplete; that is, relatively few training instances have been fully analyzed to reveal their target structure (e.g. a complete sentence analysis, a complete image parse, objective robot position, or true cell conditions), even though raw input observations are readily available (unanalyzed sentences, raw images, sensor readings, microrarray data). The need to infer missing components while training a predictive model remains a source of conceptual and computational difficulty.

One long-term goal of machine learning research is to produce methods that are applicable to complex tasks, such as perception, reasoning, intelligent control, and other artificially intelligent behaviors. However, despite impressive progress on both the academic and technological sides, the aforementioned challenges have yet to be adequately met and these long-term goals remain elusive.