alan-turing-institute / foundation-models-reading-group

Information and materials for the Turing's Foundation Models reading group.

Robots in Disguise: The Foundation Models Reading Group

Public repo for The Alan Turing Institute's reading group on foundation models as part of the activities of Foundational AI theme.

If you're based at the Turing, follow #robots-in-disguise on the Turing Slack for the most recent updates.

To see all the slides and reading materials for previous sessions, see the archive.

Note that this originated from the Research Engineering Team's reading group on Transformers.

Overview

The group meets every week on Mondays at 11-12. Everyone is welcome to join! If you have any questions email Ryan or Fede and remember to go through our Code of Conduct before joining.

Please get in touch if you would like to give a talk (either about your research or a topic you think is relevant to the reading group) add suggestions and emoji preferences to the list of proposed topics on HackMD!

Upcoming Schedule

Date	Topic	Room	Lead
01/07/24	Technical: A perspective on the fundamentals of transformers	Ursula Franklin	Ed Gunn
08/07/24	Invited Talk: Equally Safe Online? A participatory approach to tackling Gender-Based Violence	David Blackwell	Gavin Abercrombie
15/07/24	TBC: Conference Overview: Coling/LREC	Cipher	Fede Nanni
22/07/24	Invited Talk: Designing a Value-driven GAI Framework for Social Good: Embedding Social Good Values into GAI Models	Ursula Franklin	Victor OK Li, Jacqueline CK Lam and Jon Crowcroft
29/07/24	TBC	Ursula Franklin	TBC
05/08/24	Invited Talk: The growth of parallelism in machine learning inference	Ursula Franklin	Tim Harris (Microsoft)
12/08/24	TBC	Ursula Franklin	TBC
19/08/24	TBC	David Blackwell	TBC
26/08/24	TBC	David Blackwell	TBC
02/09/24	TBC	David Blackwell	TBC
09/09/24	TBC	Ursula Franklin	TBC
16/09/24	TBC	David Blackwell	TBC
23/09/24	TBC	David Blackwell	TBC

Material for sessions

01/07/24

A perspective on the fundamentals of transformers

• ICASSP Tutorial
• Transformers primer
  • Vaswani et al.'17
  • Elhage et al.'21
• Optimisation
  • Ji et al.'20
  • Tarzanagh et al.'23b
  • Tarzanagh et al.'23a
  • Vasudeva et al.'24
• Approximation
  • Yun et al.'19
  • Kajitsuka & Sato'23
  • Guo et al.'19
  • Child et al.'19
  • Yun et al.'20
  • Beltagy et al.'20
  • Zaheer et al.'20
• Memorisation
  • Blog
  • Hopfield'82
  • Krotov, Hopfield'16
  • Dermircigil et al.'17
  • Krotov, Hopfield'20
  • Ramsauer et al.'20
• In-context learning
  • Oswald et al.'22
  • Xie et al.'21

08/07/24

Equally Safe Online? A participatory approach to tackling Gender-Based Violence

We are in the midst of an ‘epidemic of online abuse’, which disproportionately affects women and minoritised groups. In recent years, technology companies and computer science researchers have made efforts to automate the identification of hate speech and other toxic or abusive language. However, existing resources are limited in a number of important ways, such as their lack of theoretical grounding and stakeholder input.The EPSRC funded project Equally Safe Online aims to harness stakeholder expertise to co-design resources and methods to tackle online GBV. In this talk, I will discuss outcomes and ongoing work from the project, focusing on participatory design for NLP, perspectivist approaches to dataset creation, and generation of counterspeech against hateful language.

15/07/24

Conference Overview: Coling/LREC

22/07/24

Designing a Value-driven GAI Framework for Social Good: Embedding Social Good Values into GAI Models