Storagesystems
- Haystack (https://lnkd.in/gSZYcmmB)
- f4: Facebook’s Warm BLOB Storage System (https://lnkd.in/gMEfTpAh)
- The Hadoop Distributed File System (https://lnkd.in/gSUqafDg)
- The Google File System (https://lnkd.in/giUResea)
- Facebook's Tectonic Filesystem: Efficiency from Exascale (https://lnkd.in/geg7-ub9)
- Pelican: A Building Block for Exascale Cold Data Storage (https://lnkd.in/gSse26YK)
- CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data (https://lnkd.in/gUbnK4rH)
- RADOS: a scalable, reliable storage service for petabyte-scale storage (https://lnkd.in/gKwbmzTx)
- Megastore: Providing Scalable, Highly Available Storage for Interactive Services (https://lnkd.in/gT7mSDQN)
- The Design and Implementation of a Log-Structured File System (https://lnkd.in/gVuka_Ym)
- The RAMCloud Storage System (https://lnkd.in/gC3SQccF)
Analytics
- Monarch: Google's Planet-Scale In-Memory Time Series Database (https://lnkd.in/gbqa7HNa)
- Gorilla: A Fast, Scalable, In-Memory Time Series Database (https://lnkd.in/gd_nUJbu)
- Scuba: Diving into Data at Facebook (https://lnkd.in/gfBrJcge)
- The Unified Logging Infrastructure for Data Analytics at Twitter (https://lnkd.in/gwhNUMnF)
- Cubrick: Indexing Millions of Records per Second for Interactive Analytics (https://lnkd.in/g-n9GUMD)
- Shark: SQL and Rich Analytics at Scale (https://lnkd.in/gqXHq5BG)
- Realtime Data Processing at Facebook (https://lnkd.in/gQdMN4kP)
Clustermanager and Scheduling
-
Large-scale cluster management at Google with Borg (https://lnkd.in/gT7bG2SF)
-
Apollo: Scalable and Coordinated Scheduling for Cloud-Scale Computing (https://lnkd.in/gEEdRmcD)
-
Apache Hadoop YARN: Yet Another Resource Negotiator (https://lnkd.in/g9SVx_Ft)
-
Twine: A Unified Cluster Management System for Shared Infrastructure (https://lnkd.in/gbnuqutm)
Streamprocessing
- MillWheel: Fault-Tolerant Stream Processing at Internet Scale (https://lnkd.in/gC7VjCfG)
- The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing (https://lnkd.in/g-PyJUPa)
- Apache Flink™: Stream and Batch Processing in a Single Engine (https://lnkd.in/gpzRA6v3)
- Drizzle: Fast and Adaptable Stream Processing at Scale (https://lnkd.in/g9Hbnvp7)
- Kafka, Samza and the Unix Philosophy of Distributed Data (https://lnkd.in/grtHkFWN)
- Discretized Streams: Fault-Tolerant Streaming Computation at Scale (https://lnkd.in/gbzc3_Ke)
- Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark (https://lnkd.in/gnQQP2UY)
- Noria: dynamic, partially-stateful data-flow for high-performance web applications (https://lnkd.in/gYtpef34)
Pubsub
- Kafka: a Distributed Messaging System for Log Processing (https://lnkd.in/dkfPsFwH)
- Scribe: Transporting petabytes per hour via a distributed, buffered queueing system (https://lnkd.in/dTyTBE_t)
- LogDevice: a distributed data store for logs (https://lnkd.in/dvVTBz46)
- Scalog: Seamless Reconfiguration and Total Order in a Scalable Shared Log (https://lnkd.in/d7xmexrQ)
- CORFU: A Shared Log Design for Flash Clusters (https://lnkd.in/dxiquk5h)
- The FuzzyLog: A Partially Ordered Shared Log (https://lnkd.in/da4ikmEa)
- Ubiq: A Scalable and Fault-tolerant Log Processing Infrastructure (https://lnkd.in/dQTfCDwH)
Graph processing in distributed setting.
- Pregel: A System for Large-Scale Graph Processing (https://lnkd.in/ggpew7yq)
- PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs (https://lnkd.in/g6f9Mjzk)
- GraphX: Graph Processing in a Distributed Dataflow Framework (https://lnkd.in/gixUZP46)
- Gemini: A Computation-Centric Distributed Graph Processing System (https://lnkd.in/gCs2R5EJ)
- TAO: Facebook’s Distributed Data Store for the Social Graph (https://lnkd.in/gfesm_Hn)
Consensus and replicated state machines.
- Paxos Made Simple (https://lnkd.in/gk6nxyVj)
- Implementing Fault-Tolerant Services Using the State Machine (https://lnkd.in/gPwNde-i)
- The Chubby lock service for loosely-coupled distributed systems (https://lnkd.in/gFXKTrXR)
- ZooKeeper: Wait-free coordination for Internet-scale systems (https://lnkd.in/gWTYBxQN)
- In Search of an Understandable Consensus Algorithm (https://lnkd.in/gqrKhvsK)
- Virtual Consensus in Delos (https://lnkd.in/g5bitkdM)
Peertopeer systems and information dessimination.
- Gossip-Based Broadcast (https://lnkd.in/gT74Zb8Z)
- Gossiping in Distributed Systems (https://lnkd.in/g55DFbuP)
- Peer-to-peer membership management for gossip-based protocols (https://lnkd.in/g_XE4TiE)
- Gossip-based Peer Sampling (https://lnkd.in/gSPwEkaW)
- SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol (https://lnkd.in/gxZtR3Nh)
- Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems (https://lnkd.in/gyURBizm)
- Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications (https://lnkd.in/grVF9crk)
Additional May be Repeated articles will categorize later.
Short Name Title Link Extra links
1 Apache Kafka Kafka: A Distributed Messaging System for Log Processing (https://notes.stephenholiday.com/Kafka.pdf) 2 Apache Cassandra Cassandra - A Decentralized Structured Storage System (https://www.cs.cornell.edu/projects/ladis2009/papers/lakshman-ladis2009.pdf) 3 Apache Flink Apache Flink: Stream and Batch Processing in a Single Engine (https://asterios.katsifodimos.com/assets/publications/flink-deb.pdf) 4 Apache Spark Spark: Cluster Computing with Working Sets (https://www.usenix.org/legacy/event/hotcloud10/tech/full_papers/Zaharia.pdf) 5 Apache Zookeeper ZooKeeper: Wait-free coordination for Internet-scale systems (https://www.usenix.org/legacy/event/atc10/tech/full_papers/Hunt.pdf) 6 BigTable Bigtable: A Distributed Storage System for Structured Data (https://research.google.com/archive/bigtable-osdi06.pdf) 8 Apache Impala Apache Impala: A Modern, Open-Source SQL Engine for Hadoop (https://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper28.pdf) 9 Apache Druid Druid: A Real-time Analytical Data Store (http://static.druid.io/docs/druid.pdf) 10 Timer Wheel Hashed and Hierarchical Timing Wheels (http://www.cs.columbia.edu/~nahum/w6998/papers/sosp87-timing-wheels.pdf) 11 MillWheel MillWheel: Fault-Tolerant Stream Processing at Internet Scale (https://research.google.com/pubs/archive/41378.pdf) 12 Dynamo Dynamo: Amazon’s Highly Available Key-value Store (https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf) 13 Google File System The Google File System (https://research.google.com/archive/gfs-sosp2003.pdf) 14 MapReduce MapReduce: Simplified Data Processing on Large Clusters (https://research.google.com/archive/gfs-sosp2003.pdf) 15 Spanner Spanner: Google’s Globally-Distributed Database (https://research.google.com/archive/spanner-osdi2012.pdf) 16 Zab Zab: High-performance broadcast forprimary-backup systems (http://www.cs.cornell.edu/courses/cs6452/2012sp/papers/zab-ieee.pdf) 17 Paxos Paxos Made Simple (https://lamport.azurewebsites.net/pubs/paxos-simple.pdf) 18 Chubby The Chubby lock service for loosely-coupled distributed systems (https://research.google.com/archive/chubby-osdi06.pdf) 19 Dremel Dremel: Interactive Analysis of Web-Scale Datasets (https://research.google/pubs/pub36632/) 20 Megastore Megastore:Providing Scalable, Highly Available Storage for Interactive Services (https://research.google/pubs/pub36971.pdf) 21 Raft In Search of an Understandable Consensus Algorithm (Extended Version) (https://raft.github.io/raft.pdf) 22 Flexible Paxos Flexible Paxos: Quorum Intersection Revisited (https://arxiv.org/abs/1608.06696) 23 Thrift Thrift: Scalable Cross-Language Services Implementation (https://thrift.apache.org/static/files/thrift-20070401.pdf) 24 Maglev Maglev: A Fast and Reliable Software Network Load Balancer (https://research.google.com/pubs/archive/44824.pdf) 25 LSM The Log-Structured Merge-Tree (LSM-Tree) (https://www.cs.umb.edu/~poneil/lsmtree.pdf) 26 Chord Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications (https://pdos.csail.mit.edu/papers/chord:sigcomm01/chord_sigcomm.pdf) 27 Kademlia Kademlia: A Peer-to-peer Information System Based on the XOR Metric (https://www.scs.stanford.edu/~dm/home/papers/kpos.pdf) 28 Mesa Mesa: Geo-Replicated, Near Real-Time, Scalable Data Warehousing (https://research.google/pubs/pub42851/ ) 29 SCRIBE SCRIBE: A large-scale and decentralized application-level multicast infrastructure https://rowstron.azurewebsites.net/PAST/jsac.pdf 30 PAST Storage management and caching in PAST- A large-scale, persistent peer-to-peer storage utility https://people.mpi-sws.org/~druschel/publications/PAST-hotos.pdf 31 Pastry Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems? https://www.cs.cornell.edu/people/egs/615/pastry.pdf 32 Linearizability Linearizability: A Correctness Condition for Concurrent Objects http://cs.brown.edu/~mph/HerlihyW90/p463-herlihy.pdf 33 Time and Clocks Time, Clocks, and the Ordering of Events in a Distributed System http://lamport.azurewebsites.net/pubs/time-clocks.pdf 34 CRDTs CRDTs: Consistency without concurrency control http://hal.archives-ouvertes.fr/docs/00/39/79/81/PDF/RR-6956.pdf 35 Photon Photon: Fault-tolerant and Scalable Joining of Continuous Data Streams https://research.google/pubs/pub41318/ 36 TAO TAO: Facebook’s Distributed Data Store for the Social Graph https://www.usenix.org/system/files/conference/atc13/atc13-bronson.pdf 37 Pregel Pregel: A System for Large-Scale Graph Processing https://15799.courses.cs.cmu.edu/fall2013/static/papers/p135-malewicz.pdf 38 Dapper Dapper: A-large-scale-distributed-tracing-infrastructure https://research.google/pubs/pub36356.pdf 39 Raft Refloated Raft Refloated: Do We Have Consensus? https://www.cl.cam.ac.uk/~ms705/pub/papers/2015-osr-raft.pdf 40 Percolator Large-scale Incremental Processing Using Distributed Transactions and Notifications https://research.google/pubs/pub36726.pdf 41 Monarch Monarch: Google’s Planet-Scale In-Memory Time Series Database https://research.google/pubs/pub50652/ 42 Borg Large-scale cluster management at Google with Borg https://research.google/pubs/pub43438.pdf 43 Borg - Next Borg: the Next Generation https://research.google/pubs/pub49065.pdf 44 Amazon Aurora Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases https://web.stanford.edu/class/cs245/readings/aurora.pdf 45 Gorilla Gorilla: A Fast, Scalable, In-Memory Time Series Database http://www.vldb.org/pvldb/vol8/p1816-teller.pdf 46 HDFS The Hadoop Distributed File System https://storageconference.us/2010/Papers/MSST/Shvachko.pdf 47 Autopilot Autopilot: workload autoscaling at Google https://dl.acm.org/doi/10.1145/3342195.3387524 48 Consistent hashing Consistent hashing and random trees: distributed caching protocols for relieving hot spots on the World Wide Web https://dl.acm.org/doi/pdf/10.1145/258533.258660 49 SEDA SEDA: An Architecture for Well-Conditioned, Scalable Internet Services http://www.sosp.org/2001/papers/welsh.pdf 50 Bitcask Bitcask: A Log-Structured Hash Table for Fast Key/Value Data https://riak.com/assets/bitcask-intro.pdf 51 DynamoDB Amazon DynamoDB: A Scalable, Predictably Performant, and Fully Managed NoSQL Database Service https://www.usenix.org/system/files/atc22-elhemali.pdf 52 Isolation levels A critique of ANSI SQL isolation levels https://dl.acm.org/doi/pdf/10.1145/223784.223785 54 Deletable Bloom Filter The deletable bloom filter https://arxiv.org/pdf/1005.0352 55 Hash Coding Space\Time Trade-offs in Hash Coding with Allowable Errors https://dl.acm.org/doi/pdf/10.1145/362686.362692 56 Expedite Byzantine Shifting Gears- Changing Algorithms on the Fly To Expedite Byzantine Agreement https://www.sciencedirect.com/science/article/pii/089054019290035E 57 Scalability cost Scalability! But at what COST? https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf 58 Foundation DB FoundationDB: A Distributed Unbundled Transactional Key Value Store https://www.foundationdb.org/files/fdb-paper.pdf 59 Monolith Monolith: Real Time Recommendation System With Collisionless Embedding Table https://arxiv.org/pdf/2209.07663 60 Memcache at Facebook Scaling Memcache at Facebook https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf 61 MilliSampler A microscopic view of bursts, buffer contention, and loss in data centers https://dl.acm.org/doi/pdf/10.1145/3517745.3561430 https://engineering.fb.com/2023/04/17/networking-traffic/millisampler-network-traffic-analysis/ 62 FlexiRaft FlexiRaft: Flexible Quorums with Raft https://www.cidrdb.org/cidr2023/papers/p83-yadav.pdf 63 Minesweeper Scalable Statistical Root Cause Analysis on AppTelemetry https://arxiv.org/abs/2010.09974 64 Shard Manager Shard Manager: A Generic Shard ManagementFramework for Geo-distributed Applications 65 FlumeJava FlumeJava: Easy, Efficient Data-Parallel Pipelines https://research.google/pubs/pub35650.pdf 65 66 Heron Twitter Heron: Stream Processing at Scale https://dl.acm.org/doi/pdf/10.1145/2723372.2742788 66 67 Dataflow The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in MassiveScale, Unbounded, OutofOrder Data Processing https://research.google/pubs/pub43864.pdf 67 68 Flink State Management in Apache Flink http://www.vldb.org/pvldb/vol10/p1718-carbone.pdf 68 69 Dgraph Dgraph: Synchronously Replicated, Transactional and Distributed Graph Database