Overview
This site contains the course materials for the ESSLLI 2023 course of Formal techniques for neural-symbolic modeling.
course description: This is intended to be an advanced course on current methods for combining symbolic logic and neural networks, with applications to problems in natural language processing (NLP) and language modeling. In particular, we focus on techniques that use symbolic knowledge and declarative constraints to train machine learning models by compiling the corresponding symbolic logic into a differentiable form, also known as the logic as a loss function family of approaches. Details of current approach in NLP (in particular, the basics of how pre-trained transformers work), as well as the formal and algorithmic techniques needed to doing this (e.g., SAT-based methods, tractable representations of logic), will be covered in detail and drawn from the broader literature of neural-symbolic learning and reasoning.
lecturers: Kyle Richardson (Allen Institute for AI), Vivek Srikumar (University of Utah).
Schedule
Please see inside of the slides for more references, code, links and other pointers.
Foundations: Logic and Transformers
- Day 1 Course introduction and logical foundations. topics: propositional logic, SAT, tractable representations and knowledge compilation. [slides] (Kyle)
General background knowledge compilation and circuits [Darwiche and Marquis 2002] [Darwiche, 2022], [Marquis, Pierre 2008] general logic and SAT [Davis, Segal and Weyuker 1994, ch12], [Kroening and Strichman 2016], [Biere, A et al. 2021], sat-based knowledge compilation [Huang and Darwiche 2007],[Oztok and Darwiche 2018]
- Day 2 Introduction to language modeling and transformers. topics: classical vs. modern LMs, contextual models and attention, model tuning. [slides] (Kyle)
Key Reading classic transformer papers [Vaswani, Ashish et al. 2017], [Radford, Alec et al. 2018], [Devlin, Jacob et al 2018], [Raffel, Colin et al. 2020], [Brown, Tom et al. 2020]. general references [Jurafsky, Dan et al. 2023, Ch.3,7,11], [Transformers from Scratch, Peter Bloem], [Daume III, Hal. 2017, Ch7,10], math for ML [Kolter, Zico 2020]
Logic for Model Training
- Day 3 Training models with symbolic logic via the semantic loss function. topics: logic as loss, weighted model counting (WMC), circuits [slides] (Kyle)
Key Reading semantic loss [Xu, Jingyi et al. 2018], [Ahmed, Kareem et al. 2022] probabilistic inference and WMC [Chiva, Mark et al. 2008], learning with constraints in NLP [Minervini, Pasquale et al 2018], [Li, Tao, et al 2019] SDDs [Darwiche, Adnan 2011] other reading [Ahmed, Kareem et al 2022]
Logical and Probabilistic Reasoning for Model Correction
- Day 4 Symbolic methods for reasoning over model predictions. topics: SAT methods and their semantics, different types of inference, circuits, NLP applications [slides] (Kyle)
Key Reading MPE and MAXSAT [Park, James 2002], [Sang, Tian et al. 2007], related NLP work on model correction [Kassner, Nora et al. 2021], [Kassner, Nora etal 2023],[Mitchell, Eric et al. 2022], [Gu, Yuling et al. 2023], [Jung, Jaehun et al. 2022], statistical relational learning background [De Raedt, Luc et al. 2016], [De Raedt and Kimmig 2015], problog [Manhaeve, Robin et al. 2018]
Logical relaxations and applications in NLP
- Day 5 Training models using multi-valued Logic topics: systems of fuzzy-logic, technical issues, applications to NLP. [slides(pdf)] [slides(ppt)] (Vivek)
Key Reading logical relaxations for NLP [Li, Tao et al. 2019], [Li, Tao, et al 2019], [Grespan, Mattia et al. 2021], [Rocktaschel, Tim et al. 2015], [Grespan, Mattia et al. 2023],[Asai, Akari et al. 2020] technical aspects of fuzzy logic [Emilie van Krien et al 2022], [Klement, Erich et al 2000]
Helpful Resources
Many examples thoughout the course taken from the work of the UCLA Automated Reasoning Group (See link for many relevant lectures). For a general survey of neural-symbolic methods, see [Marra et al. 2020], for statistical relational learning see [De Raedt, L et al. 2016], for logic and AI see [Darwiche, Ardnan 2020]
Below are some pointers to code resources:
- solvers [Z3 solver], [python-sat], [problog], [pyDatalog]
- NLP and general ML [transformers], [PyTorch], [pylon-lib], [hf datasets]
- knowledge compilation [pysdd]
- other useful utilities [sympy]