readings
A collection of papers, blog posts, and books I've read or skimmed. My main interests are:
- Foundational or historically important computer science
- AI/ML
- Systems - storage, networking, operating systems, distributed systems
- Practical programming/software engineering
papers
- Amazon Aurora: Design Considerations for High Throughput Cloud Native Relational Databases, 2017
- Amazon DynamoDB: A Scalable, Predictably Performant, and Fully Managed NoSQL Database Service, 2022
- Analysis of the Increase and Decrease Algorithms for Congestion Avoidance in Computer Networks, 1989
- An Algorithm for the Machine Calculation of Complex Fourier Series, James Cooley and John Tukey
- A Mathematical Theory of Communication, Claude Shannon
- A Name-Based Mapping Scheme for Rendezvous
- A New Solution to Dijkstra's Concurrent Programming Problem, Leslie Lamport
- A Note on Distributed Computing
- A Path Towards Autonomous Machine Intelligence, Yann LeCun 2022
- Architectural Styles and the Design of Network-based Software Architectures, Roy Fielding
- Are you living in a computer simulation?, Nick Bostrom
- Are You Sure You Want to Use MMAP in Your Database Management System?
- Attention Is All You Need, 2017 (transformer neural networks)
- Backpropagation Applied to Handwritten Zipcode Recognition, LeCun et al.
- Bayesian Flow Networks, Graves et al. 2023
- Berkeley DB
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- BigTable: A Distributed Storage System for Structured Data
- Bitcoin: A Peer-to-Peer Electronic Cash System
- BwE: Flexible, Hierarchical Bandwidth Allocation for WAN Distributed Computing, Google 2015
- Cache-Oblivious Algorithms and Data Structures, Erik Demaine 2002
- Can programming be liberated from the von Neumann style? A functional style and its algebra of programs
- Can we survive technology?, John von Neumann
- Cassandra - A Decentralized Structured Storage System
- Cicero - Human-level play in the game Diplomacy
- Chain Replication for Supporting High Throughput and Availability
- Challenges of Real-World Reinforcement Learning
- Cloud Control with Distributed Rate Limiting, sigcomm 2007
- Colossus under the hood: a peek into google's scalable storage system
- Communicating Sequential Processes, Tony Hoare
- Computing Machinery and Intelligence, Alan Turing
- Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web
- Convolutional Networks for Images, Speech, and Time-Series, Yan LeCun, Yoshua Bengio
- Dapper, a Large-Scale Distributed Tracing Infrastructure
- Deep Learning Recommendation Model for Personalization and Recommendation Systems, Facebook 2019
- Distributed Representations, Geoffrey Hinton et al.
- Dremel: Interactive Analysis of Web-Scale Datasets
- Dynamo: Amazon's Highly Available Key-value Store
- Efficient Estimations of Word Representations in Vector Space
- Efficient Transformers: A Survey, 2020
- Engineering a Sort Function, Bentley & McIlroy at Bell Labs
- Experiments on Learning by Back Propagation, Plaut, Nowlan, Hinton
- FlumeJava: Easy, Efficient Data-Parallel Pipelines
- Formal Analysis of the Remote Agent Before and After Flight on NASA remote debugging
- F1: A Distributed SQL Database That Scales (built on spanner)
- F1 Query: Declarative Querying at Scale
- Generative Adversarial Nets, Ian Goodfellow et al
- Generative Agents: Interactive Simulacra of Human Behavior
- Gorilla: A Fast, Scalable, In-Memory Time Series Database
- Gradient Based Learning Applied to Document Recognition, LeCun et al
- High-Resolution Image Synthesis with Latent Diffusion Models, 2021
- Hints for Computer System Design, Butler Lampson
- Hive - A Petabyte Scale Data Warehouse Using Hadoop
- How do Committees Invent?, Conway (Conway's Law paper)
- How to Write a Proof, Leslie Lamport
- Human-level control through deep reinforcement learning, DeepMind
- ImageNet Classification with Deep Convolutional Neural Networks
- In Search of an Understandable Consensus Algorithm (RAFT)
- Indexing the Distance: An Efficient Method to KNN Processing
- Interpreting the Data: Parallel Analysis with Sawzall
- Is this the simplest (and most surprising) sorting algorithm ever?
- It's as simple as one, two, three..., Richard Feynman
- John Carmack's 1998 .plan file
- Kafka: a Distributed Messaging System for Log Processing
- Language Models are Few-Shot Learners aka GPT3, OpenAI 2020
- Logstore: A Cloud Native and Multi-Tenant Log Database, Alibiba SIGMOD 2021
- LVars: Lattice Based Data Structures for Deterministic Parallelism
- Large-scale cluster management at Google with Borg
- Live Migration of Virtual Machines
- Machine Learning: The High Interest Credit Card of Technical Debt, Google 2014
- Maglev: A Fast and Reliable Network Load Balancer
- Making CRTs Byzantine Fault Tolerant, Martin Kleppmann
- Making reliable distributed systems in the presennce of software errors, Joe Armstrong Erlang Thesis
- MapReduce: Simplified Data Processing on Large Clusters
- Mastering the game of Go without human knowledge, Deepmind (AlphaGo Zero)
- Megastore: Providing Scalable, Highly Available Storage for Interactive Services, Google
- Mercator: A Scalable, Extensible Web Crawler
- Millions of Tiny Databases, Marc Brooker et al from AWS 2020
- Monarch: Google's Planet-Scale In-Memory Time Series Database
- Metastable Failures in Distributed Systems
- NICE: Non-Linear Independent Components Estimation, Dinh, Krueger, Bengio 2014
- NinjaMail: the Design of a High-Performance Clustered, Distributed E-mail System, Behren et al., UC Berkeley 2000
- On Computable Numbers, With an Application to the Entscheidungsproblem
- On Designing and Deploying Internet Scale Services, James Hamilton
- On Performance Stability in LSM-based Storage Systems
- On a Heuristic Point of View Concerning the Production and Transformation of Light External, Einstein photoelectric effect
- On the Electrodynamics of Moving Bodies, Einstein special relativity 1905
- On the Measure of Intelligence, Francois Chollet
- Out of the Tar Pit
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture of Experts Layer, Noam Shazeer et al 2017
- Pathways: Asynchronous Distriuted Data Flow for ML, Google 2022
- PaLM: Scaling Language Modeling with Pathways
- Paxos Made Live - An Engineering Perspective
- Performance Isolation and Fairness for Multi-Tentant Cloud Storage, OSDI 2015
- Photon: Fault-tolerant and Scalable Joining of Continuous Data Streams
- Preserving I/O Prioritization in Virtualized OSes
- Procella: Unifying serving and analytical data at YouTube
- Programming as Theory Building, Peter Naur
- Profiling a warehouse-scale computer, Google 2014
- RAMP-Tao: Layering Atomic Transactions on Facebook's Online Tao Data Store
- Recursive Functions of Symbolic Expressions and Their Computation By Machine, John McCarthy (i.e. the LISP paper)
- Reflections on Trusting Trust
- RocksDB: Evolution of Development Priorities in a Key-Value Store Serving Large Applications
- Scaling Laws for Neural Language Models
- Slicer: Auto-Sharding for Datacenter Applications
- Snap: a Microkernel Approach to Host Networking, Google 2019
- Spanner: Google's Globally Distributed Database
- Sparks of Artificial General Intelligence: Early Experiments with GPT-4, Microsoft Research 2023
- Solving Quantitative Reasoning Problems with Language Models
- Swift: Delay is Simple and Effective for Congestion Control in the Datacenter
- Tao: Facebook's Distributed Data Store for the Social Graph
- The Anatomy of a Large-Scale Hypertextual Web Search Engine
- The Byzantine Generals Problem, Leslie Lamport
- The Case for Learned Index Structures
- The Cathedral and the Bazaar, Eric Raymond
- The Chubby lock service for loosely-coupled distributed systems
- The Datacenter as a Computer, Luis Barroso and Urs Holzle
- The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Oout-of-order Data Processing
- The Design and Implementation of a Log-Structured File System, Mendel Rosenblum and John K. Ousterhout
- The Forward-Forward Algorithm: Some Preliminary Investigations, Geoffrey Hinton 2022
- The Google File System
- The Hadoop Distributed File System
- The Linux Scheduler: A Decade of Wasted Cores
- The PageRank Citation Ranking: Bringing Order to the Web
- The Part-Time Parliament, Leslie Lamport
- The Power of the Context, Alan Kay
- The RAMCloud Storage System, Ousterhout et al.
- The Slab Allocator, An Object-Caching Kernel Memory Allocator
- The Swirlds Hashgraph Consensus Algorithm
- The Tail at Scale
- Time, Clocks, and the Ordering of Events in a Distributed System
- Toolformer: Language Models Can Teach Themselves to Use Tools
- Top 10 Algorithms in Data Mining
- Transformer Feed-Forward Layers Are Key-Value Memories
- virtio: Towards a De-Facto Standard For Virtual I/O Devices
- Virtual Consensus in Delos
- Web search for a planet: the Google cluster architecture, Barosso, Dean, Holzle
- What Every Computer Scientist Should Know about Floating Point Arithmetic
- What Every Programmer Should Know about Memory
- Why Google Stores Billions of Lines of Code in a Single Repository
- Zanzibar: Google's Consistent, Global Authorization System
- ZooKeeper's Atomic Broadcast Protocol - Theory and Practice
blog posts
- The Bug in Paxos Made Simple, Marc Brooker
- Developers spend most of their time figuring out the system
- The Illustrated Transformer, Jay Alammar
- Google Research, 2022 & Beyond: Language, Vision and Generative Models, Jeff Dean et al
books
- Designing Data Intensive Applications, Martin Kleppmann
- Distributed Systems, Andrew Tanenbaum
- Introduction to Algorithms, Cormen, Leiserson, Rivest, and Stein
- Linux Kernel Development, Robert Love
- Society of Mind, Marvin Minsky
- The Algorithm Design Manual, Skiena
- The Mythical Man Month, Fred Brooks
- The Structure and Interpretation of Computer Programs, Abelson and Sussman
- Artificial Intelligence: A Modern Approach, Peter Norvig and Stuart Russell
- Computer Architecture: A Quantitative Approach, David Patterson and John Hennessy