ILP Group

A reading group on inductive logic programming, Spring and Summer 2007. For much more on ILP, see:

• The ILPnet2 web site, which has links to many data sets and systems.
• The book Relational Data Mining edited by Dzeroski and Lavrac, 2001.

Schedule

• Meeting 1 (Feb. 12): Flach and Lavrac (2002) "Learning in Clausal Logic: A Perspective on Inductive Logic Programming". Survey paper on ILP.
• Meeting 2 (Feb. 26): Muggleton and Buntine (1988) "Machine Invention of First-Order Predicates by Inverting Resolution". One of the first papers on ILP. Introduces CIGOL, a bottom-up clause learning system.
• Meeting 3 (Mar. 5): Quinlan (1990) "Learning Logical Definitions from Relations". Another of the founding papers of ILP. Introduces FOIL, a top-down clause learning system.
• Meeting 4 (Mar. 12): Muggleton (1995) "Inverse Entailment and Progol" and Srinivasan et al. (1997) "Carcinogenesis Predictions Using ILP". Progol is an ILP system that's still used today. Just reading about one system after another might get boring, so we'll focus on Progol's application to predicting whether compounds are carcinogenic.
• Meeting 5 (Mar. 19): Conklin and Witten (1994) "Complexity-Based Induction". This will take us in a more theoretical direction, toward understanding a possible objective function for ILP.
• Meeting 6 (Apr. 2): Hendrik Blockeel, Luc De Raedt, Nico Jacobs and Bart Demoen (1999) "Scaling Up Inductive Logic Programming by Learning from Interpretations". This paper is about TILDE, a system for learning relational decision trees. As another reference, see Section 6.4 of Luc De Raedt's draft book.
• Meeting 7 (Apr. 9): Chapter 3, "Representations for Mining and Learning", in Luc De Raedt's draft book. This is about the differences and equivalences between different data representations, such as attribute vectors, relational databases, graphs, logic programs, etc.
• Meeting 8 (Apr. 30): Landwehr, Kersting and De Raedt (accepted for publication) "Integrating Naive Bayes and FOIL" and Davis et al. (2007) "Change of Representation for Statistical Relational Learning". These papers are on the nFOIL and SAYU systems, which learn probabilistic classifiers whose features are constructed with ILP techniques.
• Meeting 9 (May 29): Getoor, Friedman, Koller and Pfeffer (1999) "Learning Probabilistic Relational Models" and Getoor, Friedman, Koller and Taskar (2002) "Learning Probabilistic Models of Link Structure" (JMLR). These are some of the original papers on learning joint probabilistic models for multiple attributes/predicates in a relational setting.
• Meeting 10 (June 4): Neville and Jensen (2007) "Relational Dependency Networks" (JMLR). This paper is about combining relational classifiers for individual predicates to make a joint probability model over all predicates. We'll also look at a short paper from ILP '06 on a similar theme: Ramon, Croonenborghs, Fierens, Blockeel and Bruynooghe (2006) "Generalizing Ordering-search for Learning Directed Probabilistic Logical Models.
• Meeting 11 (June 11): Kok and Domingos (2005) "Learning the Structure of Markov Logic Networks" and Mihalkova and Mooney (2007) "Bottom-Up Learning of Markov Logic Network Structure".

Other Papers of Interest

• General Theory
  • De Raedt (1997) "Logical Settings for Concept-Learning". Makes distinctions between learning from entailment (the traditional ILP setting), learning from interpretations, learning from proofs, etc.
  • Conklin and Witten (1994) "Complexity-Based Induction". Charles says: "My favourite ILP paper. It sets out the problem to be solved more clearly than other papers I've read, and uses a fairly clean MDL formulation. The downside is that they don't have any way of finding the logic program that is best according to their criterion."
  • Muggleton (1998) "Completing Inverse Entailment". Addresses the issue of the completeness of certain search operators for inducing theories.
• Systems
  • Muggleton and Feng (1990) "Efficient Induction of Logic Programs". Introduces GOLEM, a more efficient version of CIGOL.
  • Muggleton (1995) "Inverse Entailment and Progol". Describes Progol, a modern predictive ILP system.
  • Srinivasan's ALEPH system, which comes from the same origins as Progol and is perhaps the most stable and complete ILP system.
  • Dehaspe, van Laer and De Raedt (1994). "Applications of a Logical Discovery Engine". Introduces the CLAUDIEN system.
  • Blockeel and De Raedt (1998) "Top-Down Induction of First-Order Logical Decision Trees". Introduces TILDE, a system for learning first-order decision trees.
  • The CLASSIC'CL system from Universitat Freiburg, which can emulate CLAUDIEN, WARMR, and several other systems.
• Applications
  • Bratko and Muggleton (1995) "Applications of Inductive Logic Programming. Survey paper, probably somewhat out of date.
  • King, Muggleton, Lewis, and Sternberg (1992) "Drug Design by Machine Learning: The Use of Inductive Logic Programming to Model the Structure-Activity Relationships of Trimethoprim Analogues Binding to Dihydrofolate Reductase".
  • Srinivasan, Muggleton, King and Sternberg (1994) "Mutagenesis: ILP Experiments in a Non-Determinate Biologial Domain". First use of the mutagenesis data set that is used in a lot of ILP papers.
  • Srinivasan, King, Muggleton and Sternberg (1997) "Carcinogenesis Predictions Using ILP". Early use of the now-standard carcinogenesis data set.
  • King, Whelan, Jones, Reiser, Bryant, Muggleton, Kell and Oliver (2004) "Functional Genomic Hypothesis Generation and Experimentation by a Robot Scientist". Machine learning hits Nature.
• Integrating ILP with Probability
  • Muggleton (1994) "Bayesian Inductive Logic Programming". An early attempt to make ILP Bayesian.
  • Dehaspe (1997) "Maximum Entropy Modeling with Clausal Constraints". Introduces the MACCENT system for maximum entropy classification (i.e., logistic regression) with clauses learned by an ILP system as features.
  • Getoor, Friedman, Koller and Pfeffer (1999) "Learning Probabilistic Relational Models". Possibly the first work on learning joint probabilistic models for many attributes (or predicates) in a relational setting.
  • Getoor, Friedman, Koller and Taskar (2002) "Learning Probabilistic Models of Link Structure" (JMLR). Extends the PRM work to learn a certain class of models for predicting relations.
  • Popescul, Ungar, Lawrence and Pennock (2003) "Statistical Relational Learning for Document Mining". Describes a technique called "structural logistic regression". Like MACCENT, it does logistic regression with ILP-learned clauses as features.
  • Fierens, Ramon, Blockeel and Bruynooghe (2005) "A Comparison of Approaches for Learning Probability Trees. Discusses techniques for learning relational/logical decision trees with probability distributions at the leaves.
  • Kok and Domingos (2005) "Learning the Structure of Markov Logic Networks". Uses an ILP-style search technique to construct a set of Markov logic clauses (with weights) that maximize a penalized log-likelihood function on the data.
  • Paes, Revoredo, Zaverucha, and Costa (2005) "Probabilistic First-Order Theory Revision from Examples".
  • N. Landwehr, K. Kersting, L. De Raedt (2005). "nFOIL: Integrating Naive Bayes with FOIL". In M. Veloso and S. Kambhampati, editors, Proceedings of the Twentieth National Conference on Artificial Intelligence (AAAI-05), pages 795-800, Pittsburgh, Pennsylvania, USA, July 9-13, 2005. Long paper accepted for publication in the Journal of Machine Learning Research (JMLR). Describes an algorithm for learning naive Bayes (and tree-augmented naive Bayes) models with relational features.
  • J. Davis et al. (2007) "Change of Representation for Statistical Relational Learning". Describes the SAYU-VISTA system, which learns (tree-augmented) naive Bayes models with relational features. Also does a form of predicate invention.
  • Revoredo, Paes, Zaverucha, and Costa (2006) "Combining Predicate Invention and Revision of Probabilistic FOL Theories". Connects predicate invention to inventing hidden variables in Bayes nets. Was presented at ILP 2006; can't find copy online
  • J. Neville and D. Jensen (2007) "Relational Dependency Networks". JMLR 8:653-692. About constructing joint models by learning a relational classifier for each predicate given the other predicates.
  • Ramon, Croonenborghs, Fierens, Blockeel and Bruynooghe (2006) "Generalizing Ordering-search for Learning Directed Probabilistic Logical Models. Short paper at ILP 2006.
  • Mihalkova and Mooney (2007) "Bottom-Up Learning of Markov Logic Network Structure". To appear at ICML '07.
  • Mihalkova, Huynh and Mooney (2007) "Mapping and Revising Markov Logic Networks for Transfer Learning". To appear at AAAI '07.
• Predicate Invention
  • Kijsirkul, Numao and Shimura (1992) "Discrimination-Based Constructive Induction of Logic Programs". Introduces the CHAMP system for predicate invention, cited as inspiration by Craven and Slattery (2001). It was in AAAI '92; can't find copy online.
  • Kramer (1995) "Predicate Invention: A Comprehensive View". Tech report that surveys predicate invention up through 1995.
  • Stahl (1996) "Predicate Invention in Inductive Logic Programming". Survey in Advances in Inductive Logic Programming (De Raedt, ed.). Can't find copy online.
  • Martin and Vrain (1997). "Systematic Predicate Invention in Inductive Logic Programming".
  • Craven and Slattery (2001) "Relational Learning with Statistical Predicate Invention: Better Models for Hypertext". Learn document classification rules with FOIL, and invent new predicates that are defined by Naive Bayes classifiers on documents.
  • Foster and Ungar (2002?) "Learning Object Permanence by Ontological Leaps". Position paper; no algorithm.
  • Kok and Domingos (2007) "Statistical Predicate Invention". To appear at ICML '07.