Research
Our research focusses on some of the frontiers of statistical learning, on what we view as some of the most challenging questions, and in the context of MITACS, with the expectation of technologically useful results in the span of the next two years. What are these frontiers?
- Scaling issues:
to cope with larger datasets and more complex distributions,
our algorithms must scale well, in several ways:
- computational: scaling of computing resources (time, memory) with respect to the number of available examples, and the complexity of the underlying distribution (e.g. dimension of the state-space, in the case of stochastic processes).
- statistical: scaling of the required number of examples with respect to the complexity of the task (i.e., the complexity of the distribution from which we learn, or the complexity of the function we wish to obtain through learning).
- human effort: scaling of the quantity of human effort necessary to engineer a solution to a problem, as we increase the complexity of the task. Practical data-mining involves an expert "human-in-the-loop" that understands the learning algorithms and the domain of application. Such experts are expensive and slow, and we seek to automate as much of the process as possible.
- Optimization issues:
some of the most interesting learning algorithms (those
that have the potential to capture highly complex distributions) often
pose hard optimization problems.
- One area of our focus is in developing new training algorithms for complex unsupervised learning problems, since they can naturally model missing or hidden variables, as well as be trained in a semi-supervised way (when labels are missing for many examples). With the exception of the most trivial form of dimensionality reduction (principal components analysis), global training algorithms are simply not available for general unsupervised learning problems. Instead, iterative training algorithms are used, such as k-means, expectation-maximization, or contrastive-divergence, which are only guaranteed to find local solutions. We are breaking new ground in developing global training algorithms for these problems.
- Representation issues: vector space representations of data,
although traditional in machine learning and statistics, are not
adequate for capturing many phenomena of interest.
In many applications of interest
there is a need to consider richer representations such as graphs
and relations.
- Transactional data are common. A stylized problem is a network of email users, for example the Enron network[ 1 ]. The data would consist of emails sent between individuals in the network. The large number of transactions involved means that attention may focus only on the "header" information (e.g. sender, recipient(s), cc recipient(s), bcc recipients, date and time at which email was sent), and not the "payload" (e.g. body of the message). Graphs can be constructed to represent such networks, with nodes corresponding to email addresses and edges occurring between nodes whose communication intensity surpasses some threshold. Additional features may be available for the nodes, either being derived from the transactional data (e.g. volume of messages sent on weekday vs. weekend, "burstiness" of transactions, etc.) or from other external information.
These technological challenges are being driven both by the science and the needs of our industrial partners. Although signals intelligence agencies such as the Communication Security Establishment Canada (CSEC) have traditionally focused on mathematical research on cryptography, an emerging area of great interest is data mining. The high-level problem is that of "drinking from a fire hose". High volumes of data from various sources are available, on a scale that makes it impossible for human analysts to examine even a tenth of a percent of the data. In addition to being large scale, the data is complex, relational and incomplete. Automated quantitative tools are needed to filtering and model this complex data and identifying patterns of potential interest.
The emergence of comprehensive population-wide electronic health records is a recent development in Canada. Major health care regions in Canada, such as Alberta Health Services, need to use sophisticated machine learning techniques to bridge the gap between raw information and useful health care knowledge; for example, to perform surveillance and uncover novel risk factors and co-morbidities. However, here one faces a similar challenge regarding the scale, complexity, and incompleteness of the data.
The objectives stated in our 2006 application remain unchanged in their long-term nature, but our achievements in the last two years will allow us to focus on exploiting these achievements, both at a scientific and at an applied level, with our partners.
The 2006 objectives can be summarized as follows:
- Make bold steps in developing learning algorithms that require little a priori knowledge and yet can capture the statistical structure in complex distributions, i.e., learn highly-varying functions.
- Having recognized that learning such complex distributions involved difficult optimization problems, our objective was to concentrate our efforts on breaking through these optimization barriers.
- To constrain our efforts in directions compatible with the needs of our partners, we had proposed to develop solutions that could scale well with the size of the datasets and that could take advantage of data from inter-related tasks.
We believe to have made such bold steps, with the main achievements being in the area of optimization strategies for learning models involving non-convex criteria, as summarized in the previous section. Following on Hinton's breakthrough paper in 2006, we have contributed to the development of a new field: learning algorithms for deep architectures, whose first major networking event took place in December 2007 in Vancouver in the form of a Deep Learning Workshop that we organized.
Project's recent progress
Our MITACS-sponsored research prior to 2006[ 2,3 ] suggested that state-of-the-art learning algorithms suffer from one or both of the following closely related limitations: (1) learning more complex functions requires solving difficult non-convex optimization problems, (2) most non-parametric estimators are local in data-space.
Consider the input to a machine as a vector (e.g., with one vector element per pixel, for an image). Local estimators are those that implicitly or explicitly partition the set of possible inputs into local regions. It is difficult for these estimators to generalize to inputs far from the training examples (i.e. not belonging to one of these local regions). Analyses of classical non-parametric learning algorithms have often used the term ``curse of dimensionality'' to name a possible limitation of estimators that are local in data-space: the number of examples required to obtain a given level of generalization performance may grow exponentially with the dimensionality of the input space. During the last few years, we have shown how modern local non-parametric learning algorithms (such as those above) also suffer from the curse of dimensionality[ 4, 2, 3, 5]. These theoretical results show that what matters is not really the dimensionality of the input, but the number of variations (e.g., number of ups and downs, or number of bumps) that one wishes the learned function to capture.
In contrast to many existing machine learning methods that are local in data-space,\footnote{Examples of such algorithms include classical non-parametric learning algorithms, such as k-nearest neighbours and Parzen windows predictors, decision trees, kernel machines with a local (e.g., Gaussian) kernel (e.g., Support Vector Machines, Gaussian Processes), and most non-parametric semi-supervised~(see[ 6 ] for references) and unsupervised manifold learning algorithms (such as LLE, Isomap, kernel PCA, or spectral clustering; see[ 7 ] for references).} the total number of patterns that can be distinguished using a distributed representation scales rapidly (possibly exponentially) with the dimension of the representation (i.e. the number of features). To obtain a learned function that behaves differently in different regions of data-space, algorithms that are local in data-space require different tunable parameters for each of these regions.
To avoid the limitations of estimators that are local in data-space, we have turned to deep architectures, which can generalize non-locally and efficiently represent highly-varying function, but involve a difficult non-convex optimization problem.
Depth of architecture refers to the number of levels of composition of non-linear operations in the function learned. Humans often describe complex concepts in hierarchical ways, with multiple levels of abstraction. Deep learning methods aim at learning feature hierarchies with features from higher levels of the hierarchy formed by the composition of lower level features. Automatically learning features at multiple levels of abstraction allows a system to learn complex functions mapping the input to the output directly from data, without depending completely on human-crafted features. This is especially important for higher-level abstractions, which humans often do not know how to specify explicitly in terms of raw input. The ability to automatically learn powerful features has become increasingly important as the amount of data and range of applications to machine learning methods continues to grow. Another important advantage offered by deep learning strategies is the ability to naturally leverage unsupervised data as well as data from similar tasks (the multitask setting) to boost performance on large and challenging problems that routinely suffer from many missing labels[ 8 ]. One of the important ideas behind our recent work[ 9, 10, 5 ] is that deep architectures can be trained by exploiting the notion of learning more complex concepts by composing already learned simpler concepts.
Distributed representations[ 11, 12 ] found in multi-layer neural networks have been presented as ways to obtain non-local estimators[ 4 ]. So Bengio has turned his attention to the commonly used one-hidden-layer neural networks (which enjoy a distributed representation in its hidden layer). He has found that when the number of hidden units is allowed to increase arbitrarily, they become kernel machines with a local kernel[ 13 ] and they can still require an exponential number of examples to efficiently represent some functions[ 3 ]. This is because the depth of the architecture is only 2. A number of theoretical results from the research in computational complexity basically converge to the following statement: when a function can be represented compactly with an architecture of depth $k$, representing it with an architecture of depth $k-1$ may require an exponential size architecture[ 14 ]. The consequence for machine learning is that a shallow architecture may also need to be fat (with many parameters to tune) in order to capture a highly-varying function, and without other constraints, this entails poor generalization when trying to learn highly-varying functions.
Bengio and Vincent's recent work on deep architectures[ 15, 16, 17, 18, 10, 3, 5 ] follows up on a breakthrough that happened in 2006: Hinton and collaborators at U. of Toronto introduced Deep Belief Networks[ 19 ], with a learning algorithm that greedily trains one layer at a time, exploiting an unsupervised learning algorithm for each layer, a Restricted Boltzmann Machine (RBM)[ 20 ]. Shortly after, Bengio's group and Yann Le Cun's group presented at NIPS'2006 related algorithms introducing variants of the single-layer block and online training algorithms[ 15, 21 ], suggesting a general principle of layerwise unsupervised learning to initialize deep neural networks better than random initialization. Whereas it was not known how to successfully train deep architectures before 2006, these papers provided a variety of positive answers, and more recent papers produced algorithms breaking through the state of the art is several areas[ 22, 23, 24, 16, 25, 26, 8, 27, 18 ]. Our experiments also strongly indicate that this strategy helps to circumvent the difficulty of optimizing the lower layers of the architecture[ 10 ]. A flurry of papers have been published in the last year, expanding on these results in many directions, as the area of learning algorithms for deep architectures is attracting increasing interest from the research community and funding agencies. The reader is referred to[ 5 ], accepted for publication in {\em Foundations and Trends in Machine Learning} (see url below). It proposes a long-term vision for research on learning deep architectures, and reviews the principles behind Deep Belief Networks and related deep architectures.
Other unsupervised learning algorithms involving a non-convex optimization problem benefited from our attention. In the past few years we had developed convex relaxations of unsupervised learning problems that allow reasonable global training algorithms to be achieved[ 28, 29, 30 ]. Moreover, recently we have achieved a significant breakthrough in developing global, exact training algorithms for non-convex unsupervised learning problems, which completely avoids the use of approximations. The first step in this direction was achieved in[ 30 ] , which shows how a generalized form of dimensionality reduction can be solved exactly through the development of a global unsupervised training algorithm, even when the underlying problem is not convex. All previous work on this problem had employed local optimization techniques, even when the training problem was not proved to be hard.
We have also made significant progress in making algorithms applicable to large datasets. We refer here to our work on Bayesian Ensemble Learning[ 31, 32 ] which was inspired by ensemble algorithms developed in machine learning over the last decade. In particular, we develop a sum of trees model similar to boosting. Unlike boosting, a statistical model underlies the structure, enabling full statistical assessment of model uncertainty via Bayesian computational techniques. The model is very competitive in terms of predictive accuracy, and can be applied to large datasets. The NIPS publication was one of just 25 papers selected from 833 submissions for full oral presentations at this prestigious machine learning conference. A more complete version of this paper has received a favourable review at the Annals of Applied Statistics and a revision has been invited.
As outlined above, our MITACS research over the last few years have yielded impressive academic results, some of which we consider breakthroughs. We have also been very successful in establishing and improving connections with industrial partners, and some of our algorithms have been transferred already. We expect in the next two years to reap the fruits of some of these breakthroughs, both in the scientific impacts and in the industrial impacts of the recently discovered algorithms. We also want to continue nourishing the minds of the young researchers arising in our group, by connecting the applied work with fundamental questions.
Our technological objectives are the following:
- Apply the recently discovered principles regarding algorithms for learning complex functions, such as deep architectures and complex networks of interactions.
- Exploit the recent understanding of the optimization difficulty associated with learning algorithms such as deep architectures in order to create more breakthroughs.
- Extend the recent discoveries to the setting of a changing environment, something practically important for our partners.
- training time and memory consumption should scale well (eg. linearly) with the dataset size, e.g., using online learning,
- the algorithms should be able to take advantage of large quantities of unlabelled data (since most large sources of data are not human-labelled),
- the algorithms should learn concepts in such a way that they can quickly (with few examples) generalize to new related tasks, and exploit labelled and unlabelled examples from tasks other than the target one[ 33 ].
- Further investigate and build on the recently proposed successful specific strategies for learning deep architectures.
- Research exact global algorithms for optimizing non-convex criteria (guaranteed to find the global optimum solution) for classical unsupervised and semi-supervised learning problems.
- Continuation strategy: the idea of layer-wise greedy unsupervised training[ 19, 21, 1, 9 ] can be seen as an approximate continuation method[34], i.e., a global optimization techniques starting from a smoothed approximation of the target training criterion (easier to optimize) and gradually considering less smoothed versions of the training criterion, tracking local minima along the way.
- Exploiting an unsupervised local training signal for each level: in addition to a network-wide (generally supervised) training signal, which may be too weak to train the lower layers, these algorithms exploit an unsupervised signal in order to guide the training of each layer, as explained below.
- Explore alternative algorithms for the single-layer building block.
- Algorithms for joint unsupervised training of all the layers.
- The curriculum strategy.
Regarding global algorithms for optimizing non-convex criteria, we plan to continue building on our recent work on convex and non-convex optimization in machine learning [ 30, 35 ]. Using this approach we have recently achieved what we consider a breakthrough: an efficient, exact training algorithm for unsupervised and semi-supervised support vector machines (not yet published). Previous work had developed convex relaxations[ 28, 36, 37 ] or employed global optimization heuristics[ 38 ]. Our new approach requires no approximation, and yet is more efficient. These new algorithms are much more scalable than the semidefinite programming based relaxations, and should give better results than the heuristics or relaxations people currently use.
The key insight to achieving an efficient global algorithm as in[ 35 ] is to reformulate the problem in terms of a symmetric representation of the missing latent components (via an equivalence relation or outer product matrix) rather than working with the original representation itself. By using tools from convex duality and eigenvector analysis a scalable algorithm can then be expressed as an eigenvector computations nested inside an outer minimization loop, which achieves a global solution even when the optimization problem is not convex.
The research we are currently pursuing is to extend these techniques to handle more general unsupervised learning problems based on structured latent variable models. It appears as though efficient global training algorithms might be possible for classical architectures, including Kalman filters, hidden Markov models, and even stochastic context free grammars. Scalable, global training algorithms for these models would revolutionize work on unsupervised language learning, and unsupervised dynamical system identification respectively.
Of course, not all problems will admit polynomial time global training algorithms. However, even in these cases, we expect our new algorithmic ideas to offer improved local training approaches over the state of the art. With our advanced tools we expect to make new progress on learning models with more complex latent structures, such as deep circuits (deep networks) or dynamic programs (latent tree decompositions).
While our research focuses on advances in core statistical machine learning technology, we are well aware that the application of machine learning algorithms for mining industrial data is seldom direct, and usually requires a substantial amount of domain-specific data preprocessing. Our experience of collaboration with our partners has however shown that this step can be carried out without too many difficulties through a close collaboration with the partner. Also throughout our research, we keep at heart the applicability of our algorithms to industrial data sets, by focusing on the issue of scalability, and researching algorithms that are able to use partially labeled data.
Finally a very important component of research on statistical machine learning is that the proposed learning algorithms be evaluated and compared using rigorous statistical methodology, since these evaluations are experiments subject to uncertainty in the measurements. We have considerable experience in conducting such evaluations, and even contributed to the field in terms of mathematical analysis and algorithms for estimating the uncertainty around empirical estimates of performance such as cross-validation[ 39, 40 ]. For the evaluations required by this project, we will use the data sets from our partners, as well as controlled artificial data sets (where we know what the machine should learn) and public-domain benchmark data sets (to allow others to reproduce, validate and compare our work).
Although the directions above have a long-term scope and will guide our investigations in the next two years, we present here more focused sub-projects which will be starting points for the proposed research, in the short term.
Current algorithms for deep architectures are based on single-layer building blocks (like the Restricted Boltzmann Machine), which implement strategy 2 above (Exploiting an unsupervised local training signal for each level). What would be good unsupervised criteria for training them? One possibility is that each layer, in addition to being globally useful (i.e. following the gradient of a network-wide training signal) should transform the input to preserve information while capturing more of the structure in the data, i.e., the distribution of the transformed data is simpler, helping to separate factors explaining the input. We can show that this corresponds to a bound on the log-likelihood of a deep architecture. To further our understanding of learning in deep architectures, we consider it important to explore alternatives (such as the denoising auto-encoder that we introduced in[ 18 ] to the currently known single-layer building blocks. This has lead us to dvelp the denoising auto-encoder[ 18 ]. We plan to generalize the denoising auto-encoder to obtain a deep generative model. The algorithm may also be improved by combining generative and supervised criteria as in our hybrid of discriminant and generative RBM[ 17 ]. A convenient aspect of the denoising auto-encoder is that it gives complete freedom to parametrize the encoder and decoder. For example, a higher capacity encoder could capture some form of explaining away[ 19 ].
Except the wake-sleep algorithm[ 19 ], we do not have good algorithms for the unsupervised training of a deep architecture as a whole. Most examples are expected to be unlabelled, so exploring alternative algorithms for unsupervised joint training of deep architectures is appealing. We propose to extend the principles behind the log-likelihood bounds (exploited to justify the training algorithm for a single-layer denoising auto-encoder) to explore unsupervised joint training algorithms for all the layers of a deep stack of denoising auto-encoders.
Drawing inspiration from human and animal learning[ 41 ], we propose to train models in phases: we start with examples that illustrate simpler concepts, allowing the learner to build representations of the more abstract concepts on top of the representations for the simpler ones. The idea of "starting small"[ 42, 43 ] and going through a sequence of gradually more difficult tasks is not new in machine learning and cognitive science, but it has recently gained more momentum (e.g., with the Wubble world[ 44 ] and with a DARPA research program on bootstrap learning). What we call the "curriculum strategy" draws from similar inspiration, with a viewpoint oriented towards reducing the difficulty with local minima: our hypothesis is that a multi-phase easy-to-hard training process can be used to guide training towards much better local minima, ones that would have very low probability of being found from a random initialization. We suggest[ 5 ] that appropriately chosen forms of curriculum training can be seen as a form of continuation methods[ 34 ]. A continuation method is a global optimization technique starting from a smoothed approximation of the criterion to minimize and gradually considering less smoothed versions of the training criterion, tracking local minima along the way. Smoothing the training criterion makes it easier to optimize, since for well-behaved criteria, smoothing a lot eventually yields a convex criterion, that does not have local minima, only a global minimum.
An emerging research area is learning dynamic networks. This work is in conjunction with partner CSEC, who has collaborated with the group and Chipman in particular since 2004. CSEC has a data mining group with active interest in network mining, as it pertains to their mandate of signals intelligence.
Our partnership with CSE is recent and they have been very satisfied with the relationship: they agreed to fund us for this next phase of our project. They are interested in a number of interesting data-mining questions:
- Network representation: can a descriptive model, possibly with a visual representation, be constructed that portrays the network structure in the data? Visualization may be most relevant for small networks.
- Clustering: Are there email addresses that can be grouped together, either because (1) they send many messages to each other or (2) they send messages in a similar manner (e.g. hockey fans might experience spikes in communication during the Stanley Cup playoffs)?
- Edge prediction: If nodes in the network are observed, but not all transactions are available, this could be considered to be a graph with "unknown" edges (i.e. we do not know whether some pairs of email addresses communicate or not). Can we use available information to predict these unknown edges?
- Node prediction: If nodes have categorical labels that are measured externally to transactions, can we build a model that uses graph structure to predict these node labels? For example nodes could be considered "hockey fans" or "everyone else".
To set the context we first describe modeling of static networks, i.e., those whose underlying structure is not assumed to vary over time. Such networks may arise in a wide variety of contexts, including email communication networks, social networks (both online and real), citation networks in peer-reviewed journals and blogs, and real networks such as the internet or part of it. In all cases it is useful to think of the network as a graph, with (minimally) nodes representing objects (e.g., individuals in a social network) and edges representing a relation between objects (e.g., friendship ties in a social network). Such relational data is sometimes augmented with additional information, such as node or edge features (e.g., in social networks, personal information on individuals and type or strength of the friendship relation). Modelling of network data can involve a number of different tasks. For instance, link prediction focuses on scenarios in which some edges in the network are not observed. Predictive models would be learned from available data, and used to predict unobserved edge variables. Edge variables might be binary (presence/absence of a relation), count (number of emails sent), or real-valued (strength of friendship). Predictive models for the binary case include exponential random graph models, latent class models and latent variable models. [ 45 ] and references therein provide background on all these models. The latter two models devote significant effort to the use of unobserved (i.e., latent) variables as a means for accounting for dependence relations such as homophily (the friend of my friend is a friend). [ 46 ] considered the choice of the number of clusters and the dimensionality of the latent space, showing that effective choice can yield useful predictive models.
A second task is node prediction, in which we desire to predict some feature associated with each node, using either network structure, other features of the node, or a combination. For example, in a transactional communication network, [ 47 ] demonstrated that features derived from individual communication transactions (e.g. messaging frequency, node degree, etc.) could be effective for prediction of a node feature. Chipman and Bouveyron are working on using as features a set of latent variables arising from the link prediction problem. Interesting extensions include combination of these methods.
In the above problems, the observations on the network may change over time, but the assumption is that the underlying probabilistic model describing the network is not changing. A more interesting and challenging problem occurs when the underlying model is permitted to vary with respect to time. A hierarchical (or multi-layer) approach could be taken to describe a sequence of realizations of the network. For instance the network parameters Θ{t} at time step t would be modeled as a perturbation of those at the previous time step, Θ{t} = Θ{t-1} + ε{t}.. The distribution of perturbations ε{1} ... ε{t} would be parameterized, so as to control the magnitude of deviations. Allowing the ε{i} to be independent and identically distributed would correspond to a random walk of the graph model. [ 48 ] considered such a model for latent positions in the link prediction model described above. Introduction of a functional dependence of the parameters of the ε{i} distribution would allow for systematic change with respect to time or other observable features. Such models have connections with both hierarchical Bayesian statistical models (e.g. work by Chipman), and with deep belief networks that allow multiple levels of representation (Vincent and Bengio). The research interests of Chipman in MCMC-based algorithms for Bayesian learning and Vincent and Bengio in learning multiple levels of representation in deep architectures will provide complimentary approaches to development of learning algorithms for such models.
Note that the following references can be found on this website's publications page: [ 2, 3, 4, 5, 6, 7 , 9, 10, 13, 15, 16, 17, 18, 30, 31, 39, 40, 35, 46]
[1] Priebe, C., Conroy, J., Marchette, D., & park, Y. (2005). Scan statisand Mathematical Organization Theory, 11(3), 229-247.
[8] Collobert, R., & Weston, J. (2008). A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the Twenty-fifth International Conference on Machine Learning (ICML 2008).
[11] Hinton, G. E. (1986). Learning distributed representations of concepts. In Proceedings of the Eighth Annual Conference of the Cognitive Science Society, pp. 1-12 Amherst 1986. Lawrence Erlbaum, Hillsdale.
[12] Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533-536.
[14] Hastad, J., & Goldmann, M. (1991). On the power of small-depth threshold circuits. Computational Complexity, 1, 113-129.
[19] Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527-1554.
[20] Freund, Y., & Haussler, D. (1994). Unsupervised learning of distributions on binary vectors using two layer networks. Tech. rep. UCSC-CRL-94-25, University of California, Santa Cruz.
[21] Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y. (2007). Efficient learning of sparse representations with an energy-based model. In NIPS 19.
[22] Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504-507.
[23] Salakhutdinov, R., & Hinton, G. (2007a). Learning a nonlinear embedding by preserving class neigh- bourhood structure. In Proceedings of AISTATS 2007 San Juan, Porto Rico. Omnipress.
[24] Salakhutdinov, R., & Hinton, G. (2008). Using deep belief nets to learn covariance kernels for gaus- sian processes. In Platt, J. C., Koller, D., Singer, Y., & Roweis, S. (Eds.), Advances in Neural Information Processing Systems 20. MIT Press, Cambridge, MA.
[25] Salakhutdinov, R., & Hinton, G. (2007b). Semantic hashing. In Proceedings of the 2007 Workshop on Information Retrieval and applications of Graphical Models (SIGIR 2007) Amsterdam. Elsevier.
[26] Hadsell, R., Erkan, A., Sermanet, P., Scoffier, M., Muller, U., & LeCun, Y. (2008). Deep belief net learning in a long-range vision system for autonomous off-road driving. In Proc. Intelligent Robots and Systems (IROS'08).
[27] Weston, J., Ratle, F., & Collobert, R. (2008). Deep learning via semi-supervised embedding. In Proceed- ings of the Twenty-fifth International Conference on Machine Learning (ICML 2008).
[28] Xu, L., Neufeld, J., Larson, B., & Schuurmans, D. (2004). Maximum margin clustering. In Advances in Neural Information Processing Systems (NIPS) 17.
[29] Xu, L., Wilkinson, D., Southey, F., & Schuurmans, D. (2006). Discriminative unsupervised learning of structured predictors. In Proceedings of the Twenty-third International Conference on Machine Learning (ICML).
[32] Chipman, H. A., George, E. I., & McCulloch, R. E. (2008). Bayesian ensemble learning. Annals of Applied Statistics.
[33] Raina, R., Battle, A., Lee, H., Packer, B., & Ng, A. Y. (2007). Self-taught learning: transfer learning from unlabeled data. In ICML, pp. 759-766.
[34] Allgower, E. L., & Georg, K. (1980). Numerical Continuation Methods. An Introduction. No. 13 in Springer Series in Computational Mathematics. Springer-Verlag.
[36] de Bie, T., & Cristianini, N. (2003). Convex methods for transduction. In Advances in Neural Information Processing Systems (NIPS) 16.
[37] de Bie, T., & Cristianini, N. (2006). Fast SDP relaxations of graph cut clustering, transduction, and other combinatorial problems. Journal of Machine Learning Research, 7.
[38] Chapelle, O., Chi, M., & Zien, A. (2006). A continuation method for semi-supervised SVMs. In Pro- ceedings of the Twenty-third International Conference on Machine Learning (ICML).
[41] Skinner, B. F. (1958). Reinforcement today. American Psychologist, 13, 94-99.
[42] Elman, J. L. (1993). Learning and development in neural networks: The importance of starting small.. Cognition, 48, 781-799.
[43] Rohde, D., & Plaut, D. (1999). Language acquisition in the absence of explicit negative evidence: How important is starting small?. Cognition, 72, 67-109.
[44] Kerr, W., Hoversten, S., Hewlett, D., Cohen, P. R., & Chang, Y.-H. (2007). Learning in wubble world. In Proceedings of the IEEE Int. Conference on Development and Learning.
[45] Computation, 18, 1527-1554. Hoff, H. (2008). Modeling homophily and stochastic equivalence in symmetric relational data. In Platt, J., Koller, D., Singer, Y., & Roweis, S. (Eds.), Advances in Neural Information Processing Systems 20, pp. 657-664 Cambridge, MA. MIT Press.
[47] Nahm, E. (2005). Classification models for transactional graph data. Master's thesis, Department of Mathematics and Statistics, Acadia University.
[48] 791-798 New York, NY, USA. ACM. Sarkar, P., & Moore, A. (2005). Dynamic social network analysis using latent space models. SIGKDD Explorations, 7(2), 31-40.
