Skip to main content

Designing Data-Intensive Applications (2nd Edition): Notes

books
data
My chapter notes on the 2nd edition of Designing Data-Intensive Applications.
Published

November 11, 2025

Chapter 1. Trade-offs in Data Systems Architecture

*Operational System >Backend of the application, both read and write >Point queries >Create, Update, Delete Analytical Systems > Read only copy of operational system for data analysis > Bulk import (ETL) or event stream

Data lake - Raw data in files Data Warehouse - Structured data

Cloud Vs Selfhosting Cloud * Need to scale up/down

Why distributed systems over single node? * Fault tolerance/High Availabilty * Scalibility * Latency * Elasticity * Using specialized hardware

Downside: Troubelshooting is often difficult and developed under observability observability is collecting data about the execution of a system, and allowing it to be queried in ways that allows both high-level metrics and individual events to be analyzed

Microservice - small seperate unit and Serverless, or function-as-a-service (FaaS) - autoallocates and frees resources, built based on time used

Chapter 2. Defining Nonfunctional Requirements

Response time = 2Network latency(to and fro) + 2queuing delay(in and out) + service time Average, Median, and Percentiles are the common measures Keeping Tail latencies happy is also important Tail latency amplification - the chance of getting a slow call increases if an end-user request requires multiple backend calls, and so a higher proportion of end-user requests end up being slow

Chapter 3. Data Models and Query Languages Declarative - specify the pattern of the data you want—what conditions the results must meet, and how you want the data to be transformed (e.g., sorted, grouped, and aggregated)—but not how to achieve that goal. The database system’s query optimizer can decide which indexes and which join algorithms to use, and in which order to execute various parts of the query. Declarative saves headache how to implement parallelism and db takes care underneath

Relational Cost high for combining and displaying a profile Document databases can store both normalized and denormalized data, but they are often associated with denormalization—partly because the JSON data model makes it easy to store additional, denormalized fields, and partly because the weak support for joins in many document databases makes normalization inconvenient

Fact tables are individial events, enriched with dimension tables OLTP systems go with Normalization. Analytics systems often fare better with denormalized data. Structure of tables in a data warehouse: a star schema (fact in middle, surrounded by dimensions), snowflake schema(dimensions are further broken to subdimension), and one big table(no normalisation)

When to use which model? document data model are schema flexibility but traversing second or more level is difficult relational model counters by providing better support for joins, many-to-one, and many-to-many relationships.

A graph consists of two kinds of objects: vertices (also known as nodes or entities) and edges. Adjaceny list, each verted stores ids of neighbors one way. Good for traversal, In Adjaceny matrix , row and column correspond on

In the adjacency list model, each vertex stores the IDs of its neighbor vertices that are one edge away. Alternatively, you can use an adjacency matrix, a two-dimensional array where each row and each column corresponds to a vertex, where the value is zero when there is no edge between the row vertex and the column vertex, and where the value is one if there is an edge. The adjacency list is good for graph traversals, and the matrix is good for machine learning

Property Graphs: A property graph is made of vertices and edges, each with unique IDs, labels, and key–value properties. Vertices hold incoming/outgoing edges, while edges connect vertices and describe their relationships.

Cypher Query Language - Neo4j Graph Language Graph Queries in SQL - like AGE- pg graph db Triple-Stores and SPARQL - all information is stored in the form of very simple three-part statements: (subject, predicate, object) Datalog - consists of facts, and each fact corresponds to a row in a relational table. answers arrived recursively.

Event Sourcing and CQRS = Using a log of immutable events as source of truth and deriving materialized views from it.

DataFrames= Similar to realtional table, manipulated via seris of commands Matrices = 2-d numbers, easier for linear algebra ops, rem adjaceny matrix above eg: used in the financial industry for representing time series data, such as the prices of assets and trades over time Arrays = large multidimensional array usess : geospatial measurements (raster data on a regularly spaced grid), medical imaging, or observations from astronomical telescopes

Chapter 4. Storage and Retrieval

Log-Structured Storage Store hash map in memory with key and offset. The hash map is not persisted. Thats why restarts take long time in dbs like clickhouse and hash table should fit in memory. Range queries are not efficient. SSTable file format Sorted String Table: Hashmap stored in sorted order and time to time flushed. Typical datastructures used red-black tree, skip list, or trie. in-memory data structure is called the memtable.

Bloom filter If the Bloom filter says that a key is not present, we can safely skip that SSTable, since we can be sure that it doesn’t contain the key.

If the Bloom filter says the key is present, we have to consult the sparse index and decode the block of key-value pairs to check whether the key really is there. If it was a false positive, we have done a bit of unnecessary work, but otherwise no harm is done—we just continue the search with the next-oldest segment.

Compaction strategies >Size-tiered compaction >Leveled compaction

B-Trees B-trees keep key-value pairs sorted by key, which allows efficient key-value lookups and range queries. B-trees break the database down into fixed-size blocks or pages, and may overwrite a page in-place.write-ahead log (WAL). This is an append-only file to which every B-tree modification must be written before it can be applied to the pages of the tree itself.

Data Storage for Analytics >Cloud Data Warehouses Query engine=Trino, Apache DataFusion, and Presto Storage format = Parquet, ORC, Lance, or Nimb Table Format = Apache Iceberg or Databricks’s Delta Data catalog = Snowflake’s Polaris and Databricks’s Unity Catalog, Apache Iceberg

Column-Oriented Storage Column Compression Sort Order in Column Storage Writing to Column-Oriented Storage Query Execution: Compilation and Vectorization

Full-Text Search

The data structure that many search engines use to answer such queries is called an inverted index. This is a key-value structure where the key is a term, and the value is the list of IDs of all the documents that contain the term (the postings list

How to cope with typos? Trie and Levenshtein automaton can help

Vector Embeddings semantic search indexes use embedding models to translate a document into a vector of floating-point values, called a vector embedding. Embedding models generate vector embeddings that are near each other (in this multi-dimensional space) when the embedding’s input documents are semantically similar

OLTP system typically accessed with primary key or a secondary index. OLTP takes log-structured storage engines for high throughput. B-trees tend to be better for reads, providing higher read throughput. Data warehouses use column oriented

Finally, vector databases are used for semantic search on text documents and other media; they use vectors with a larger number of dimensions and find similar documents by comparing vector similarity.

Embedding models use much larger vectors. earch engines use distance functions such as cosine similarity or Euclidean distance to measure the distance between vectors. Cosine similarity measures the cosine of the angle of two vectors to determine how close they are

Chapter 5. Encoding and Evolution several ways of turning data structures into bytes on the network /disk. The translation from the in-memory representation to a byte sequence is called encoding (also known as serialization or marshalling), and the reverse is called decoding (parsing, deserialization, unmarshalling). * Programming language–specific encodings eg:pickle for python objects * Textual formats like JSON, XML, and CSV * Binary schema–driven formats like Protocol Buffers and Avro. Protocol Buffers requires a schema for any data that is encoded.Protbuf saves hassel of saving field name everytime and saves once for schema. * Many frameworks couple service definitions and server code together. In some cases, such as with the popular Python FastAPI framework, servers are written in code and an IDL is generated automatically. In other cases, such as with gRPC, the service definition is written first, and server code scaffolding is generated.

Modes of dataflow * Databases * Service calls RPC and REST APIs * Workflow engines * Event-driven architectures (using message brokers or actors)

Chapter 6. Replication Replicas quickly reflect writes from one node on other nodes, but backups store old snapshots of the data so that you can go back in time. Backups can go to objectstore like s3

Single-Leader Replication Leader receives and streams changes to followers. Followers ack . Read goes to followers or leaders. however, write goes to leader.

Synchronous Versus Asynchronous Replication synchronous replication is that the follower is guaranteed to have an up-to-date copy of the data Disadvantage is that if the synchronous follower doesn’t respond, the write cannot be processed.

Multi-Leader Replication - allow more than one node to accept writes. Topoligies - Circle, star, all to all Sync Engines and Local-First Software like google calendar needs multi leader Conflict avoidance useful for id generation last write wins = discarding concurrent writes

Two families of algorithms for automatic conflict resolution: > Conflict-free replicated datatypes (CRDTs) = in distributed databases such as Redis Enterprise, Riak, and Azure Cosmos DB > Operational Transformation (OT) = used for real-time collaborative editing of text, e.g. in Google Docs

Leaderless Replication Client sends write to multiple nodes simultaneosuly Challenge in Catching up on missed writes

Quorums for reading and writing * Rule: w + r > n ( sets of nodes used by the read and write operations overlap in at least one node) . if there are n replicas, every write must be confirmed by w nodes to be considered successful, and we must query at least r nodes for each read version number or timestamp can help in finding which write is latest

Chapter 7. Sharding primary reason for sharding a database is scalability: sharding is used to implement multitenant systems. Sharding types: 1. key range 2. Hash of Key A consistent hashing algorithm is a hash function that maps keys to a specified number of shards in a way that satisfies two properties: * the number of keys mapped to each shard is roughly equal, and * when the number of shards changes, as few keys as possible are moved from one shard to another.

Chapter 8. Transactions Literally executing transactions in a serial order slows things. Two-phase locking needs lot of corrindation. Serializable snapshot isolation (SSI) is new where optimistic approach, allowing transactions to proceed without blocking. When a transaction wants to commit, it is checked, and it is aborted if the execution was not serializable.

Chapter 9. The Trouble with Distributed Systems

distributed algorithms rely on timeouts to determine whether a remote node is still available. Quorum needed to arrive at meaningful information. Distributed systems help in Scale, latency and fault tolerance.

Chapter 10. Consistency and Consensus Coordination services like ZooKeeper and etcd are also built on top of consensus algorithms. They provide locks, leases, failure detection, and change notification features that are useful for managing the state of distributed applications.

Chapter 11. Batch Processing Modern batch processing has been heavily influenced by MapReduce Batch processing landscape has changed by below: * more sophisticated caching and execution strategies * focus has shifted toward usability. * New processing models such as dataflow APIs, query languages, and DataFrame APIs are now widely supported. * Job and workflow orchestration has also matured * Batch storage layers are shifting from distributed filesystems (DFSs) like HDFS, GlusterFS, and CephFS to object storage systems such as S3 * if the job’s working set is larger than the available memory, the sorting approach has the advantage that it can make efficient use of disks * Datasets that are too large to fit in memory or local disk present a problem, is the challenge