Built a real-time news stream processing engine supporting infinite publisher streams, windowed aggregations, compile-once transformations, SSE delivery, and backpressure-aware async pipelines.

Real-Time News Stream Processing Engine” That gives you:

continuous event streams windows transforms fanout prioritization backpressure infinite producers

without sounding like observability/log infrastructure.

Why News Works Well

News naturally behaves like streaming events.

Example event:

{ “source”: “Reuters”, “category”: “technology”, “headline”: “AI chip demand rises”, “timestamp”: … }

You can simulate:

multiple publishers breaking news bursts category subscriptions rolling trends live feeds

Very realistic for stream systems.

Mapping from Log Concepts → News Concepts Streaming System Concept News Equivalent log event news article producer publisher severity priority stream live feed aggregation trend analysis alerts breaking news fanout subscribers filtering topic/category filters Your Architecture Becomes News Publishers ↓ Ingress Queue ↓ Normalizer ↓ Transform Engine ↓ Window Aggregator ↓ Subscriber Queues ↓ SSE Clients

Looks clean and product-oriented. Infinite Loop Producers Each publisher continuously emits articles:

while running: article = generate_news() await ingress_queue.put(article)

Shortfalls of traditional monitoring ppaorach: * Many services * polyglot persistence * dynamic * variety of correlation

Debugging with Metrics Versus Observability

Common ones: six high-cardinality dimensions per event: time, app, host, user, endpoint, and status

Why Intuitive Troubleshooting Has Stopped Working for You * sense-analyze-respond needed due to complexities * core analysis loop eg: Most of the errors are coming from a specific endpoint –> there’s a subset of requests that seem to have a much higher latency than others.–> one of the slow requests seems to be hanging up in a caching subsystem –> all the slow caching calls seem to be referring to the same object ID. –> a specific cached object has become corrupted somehow –> test that hypothesis by looking at the payload of those objects. –> execute a command to flush that cached object and watch for the effects

• Observability allows us to probe deep into our systems to understand behavior, down to the level of individual requests between services.

Structured Events * Numerical values generated by a metric reflect an aggregated report of system state over a predefined period of time

• High cardinality and high dimensionality not feasible. eg: “all Canadian users, running iOS11 version 11.0.4, using the French language pack, who installed the app last Tuesday, running firmware version 1.4.101, who are storing photos on shard3 in region us-west-1.”
• Structured logs are machine parsable and can be useful for the goals of observability, presuming they are redesigned to resemble a structured event.

Stitching Events into Traces Distributed traces are an interrelated series of events.

Components of trace: UUID Trace ID Span ID Parent ID Timestamp Duration

Azure Monitor: 1. Detect and analyze issues across applications and dependencies 2. Correlate infrastructure problems with insights from the VMs and containers 3. Create visualizations for logs and other kinds of telemetry

Managing an Azure DevOps Project with Azure Boards Automating Code Builds and Automated Tests with Azure Pipelines Resource groups can be used to group resources by project, department, service, feature

• An application data-intensive if data management is one of the primary challenges in developing the application
• An compute-intensive systems the challenge is parallelizing some very large computation
• Standard building blocks of ddia 1. database 2. cache 3. stream processing 4.batch processing and 5. search indexes

*Operational System >Backend of the application, both read and write >Point queries >Create, Update, Delete Analytical Systems > Read only copy of operational system for data analysis > Bulk import (ETL) or event stream

Data lake - Raw data in files Data Warehouse - Structured data

Cloud Vs Selfhosting Cloud * Need to scale up/down

• Selfhosting

• Predicatable load

• can add custom features

• service availability in own hands

• cheap for large use

• data secure

• Seperation of storage and compute is the craze In a traditional systems architecture, the same computer is responsible for both storage (disk) and computation (CPU and RAM). In cloud-native systems, they are indeed seperated

• Multitenant - no seperate machines for each customer, compute but rather better hardware utilization, easier scalability, and easier management by the cloud provider.

Why distributed systems over single node? * Fault tolerance/High Availabilty * Scalibility * Latency * Elasticity * Using specialized hardware

Downside: Troubelshooting is often difficult and developed under observability observability is collecting data about the execution of a system, and allowing it to be queried in ways that allows both high-level metrics and individual events to be analyzed

Microservice - small seperate unit and Serverless, or function-as-a-service (FaaS) - autoallocates and frees resources, built based on time used

Chapter 2. Defining Nonfunctional Requirements

• Performance >Response - Time taken to serve a request - “time it takes to load the home timeline” or the “time until a post is delivered to followers” >Throughput - How many requests served - “posts per second” and “timeline writes per second”. To avoid retry tsunami, implement exponential backoff with jitters and add circuit breaker or token bucket algorithm.

Response time = 2Network latency(to and fro) + 2queuing delay(in and out) + service time Average, Median, and Percentiles are the common measures Keeping Tail latencies happy is also important Tail latency amplification - the chance of getting a slow call increases if an end-user request requires multiple backend calls, and so a higher proportion of end-user requests end up being slow

• Reliability and Fault tolerance If it continues providing the required service to the user in spite of certain faults occurring Faults come from hardware/software/humans.

• Scalability Shared-Memory- Vertical scaling Shared-Disk - Mid of up and down options. Storage shared , compute seperate Shared-Nothing Architecture - Horizonatal scaling A good general principle for scalability is to break a system down into smaller components that can operate largely independently from each other. Another good principle is not to make things more complicated than necessary

• Maintainability

• Easy to operate. Self healing

• Simple to understand via design patterns and domain-driven design

• Can easily evolve using agile via TDD, refactoring and writing lots of tests

Chapter 3. Data Models and Query Languages Declarative - specify the pattern of the data you want—what conditions the results must meet, and how you want the data to be transformed (e.g., sorted, grouped, and aggregated)—but not how to achieve that goal. The database system’s query optimizer can decide which indexes and which join algorithms to use, and in which order to execute various parts of the query. Declarative saves headache how to implement parallelism and db takes care underneath

• Object-relational mapping (ORM) Helps in > Translation between persistent relational and the in-memory object representation > Caching results of database queries > Can help schema management and migrations

Relational Cost high for combining and displaying a profile Document databases can store both normalized and denormalized data, but they are often associated with denormalization—partly because the JSON data model makes it easy to store additional, denormalized fields, and partly because the weak support for joins in many document databases makes normalization inconvenient

Fact tables are individial events, enriched with dimension tables OLTP systems go with Normalization. Analytics systems often fare better with denormalized data. Structure of tables in a data warehouse: a star schema (fact in middle, surrounded by dimensions), snowflake schema(dimensions are further broken to subdimension), and one big table(no normalisation)

When to use which model? document data model are schema flexibility but traversing second or more level is difficult relational model counters by providing better support for joins, many-to-one, and many-to-many relationships.

A graph consists of two kinds of objects: vertices (also known as nodes or entities) and edges. Adjaceny list, each verted stores ids of neighbors one way. Good for traversal, In Adjaceny matrix , row and column correspond on

In the adjacency list model, each vertex stores the IDs of its neighbor vertices that are one edge away. Alternatively, you can use an adjacency matrix, a two-dimensional array where each row and each column corresponds to a vertex, where the value is zero when there is no edge between the row vertex and the column vertex, and where the value is one if there is an edge. The adjacency list is good for graph traversals, and the matrix is good for machine learning

Property Graphs: A property graph is made of vertices and edges, each with unique IDs, labels, and key–value properties. Vertices hold incoming/outgoing edges, while edges connect vertices and describe their relationships.

Cypher Query Language - Neo4j Graph Language Graph Queries in SQL - like AGE- pg graph db Triple-Stores and SPARQL - all information is stored in the form of very simple three-part statements: (subject, predicate, object) Datalog - consists of facts, and each fact corresponds to a row in a relational table. answers arrived recursively.

Event Sourcing and CQRS = Using a log of immutable events as source of truth and deriving materialized views from it.

DataFrames= Similar to realtional table, manipulated via seris of commands Matrices = 2-d numbers, easier for linear algebra ops, rem adjaceny matrix above eg: used in the financial industry for representing time series data, such as the prices of assets and trades over time Arrays = large multidimensional array usess : geospatial measurements (raster data on a regularly spaced grid), medical imaging, or observations from astronomical telescopes

Chapter 4. Storage and Retrieval

Log-Structured Storage Store hash map in memory with key and offset. The hash map is not persisted. Thats why restarts take long time in dbs like clickhouse and hash table should fit in memory. Range queries are not efficient. SSTable file format Sorted String Table: Hashmap stored in sorted order and time to time flushed. Typical datastructures used red-black tree, skip list, or trie. in-memory data structure is called the memtable.

Bloom filter If the Bloom filter says that a key is not present, we can safely skip that SSTable, since we can be sure that it doesn’t contain the key.

If the Bloom filter says the key is present, we have to consult the sparse index and decode the block of key-value pairs to check whether the key really is there. If it was a false positive, we have done a bit of unnecessary work, but otherwise no harm is done—we just continue the search with the next-oldest segment.

Compaction strategies >Size-tiered compaction >Leveled compaction

B-Trees B-trees keep key-value pairs sorted by key, which allows efficient key-value lookups and range queries. B-trees break the database down into fixed-size blocks or pages, and may overwrite a page in-place.write-ahead log (WAL). This is an append-only file to which every B-tree modification must be written before it can be applied to the pages of the tree itself.

Data Storage for Analytics >Cloud Data Warehouses Query engine=Trino, Apache DataFusion, and Presto Storage format = Parquet, ORC, Lance, or Nimb Table Format = Apache Iceberg or Databricks’s Delta Data catalog = Snowflake’s Polaris and Databricks’s Unity Catalog, Apache Iceberg

Column-Oriented Storage Column Compression Sort Order in Column Storage Writing to Column-Oriented Storage Query Execution: Compilation and Vectorization

Full-Text Search

The data structure that many search engines use to answer such queries is called an inverted index. This is a key-value structure where the key is a term, and the value is the list of IDs of all the documents that contain the term (the postings list

How to cope with typos? Trie and Levenshtein automaton can help

Vector Embeddings semantic search indexes use embedding models to translate a document into a vector of floating-point values, called a vector embedding. Embedding models generate vector embeddings that are near each other (in this multi-dimensional space) when the embedding’s input documents are semantically similar

OLTP system typically accessed with primary key or a secondary index. OLTP takes log-structured storage engines for high throughput. B-trees tend to be better for reads, providing higher read throughput. Data warehouses use column oriented

Finally, vector databases are used for semantic search on text documents and other media; they use vectors with a larger number of dimensions and find similar documents by comparing vector similarity.

Embedding models use much larger vectors. earch engines use distance functions such as cosine similarity or Euclidean distance to measure the distance between vectors. Cosine similarity measures the cosine of the angle of two vectors to determine how close they are

Chapter 5. Encoding and Evolution several ways of turning data structures into bytes on the network /disk. The translation from the in-memory representation to a byte sequence is called encoding (also known as serialization or marshalling), and the reverse is called decoding (parsing, deserialization, unmarshalling). * Programming language–specific encodings eg:pickle for python objects * Textual formats like JSON, XML, and CSV * Binary schema–driven formats like Protocol Buffers and Avro. Protocol Buffers requires a schema for any data that is encoded.Protbuf saves hassel of saving field name everytime and saves once for schema. * Many frameworks couple service definitions and server code together. In some cases, such as with the popular Python FastAPI framework, servers are written in code and an IDL is generated automatically. In other cases, such as with gRPC, the service definition is written first, and server code scaffolding is generated.

Modes of dataflow * Databases * Service calls RPC and REST APIs * Workflow engines * Event-driven architectures (using message brokers or actors)

Chapter 6. Replication Replicas quickly reflect writes from one node on other nodes, but backups store old snapshots of the data so that you can go back in time. Backups can go to objectstore like s3

Single-Leader Replication Leader receives and streams changes to followers. Followers ack . Read goes to followers or leaders. however, write goes to leader.

Synchronous Versus Asynchronous Replication synchronous replication is that the follower is guaranteed to have an up-to-date copy of the data Disadvantage is that if the synchronous follower doesn’t respond, the write cannot be processed.

• Setting Up New Followers - Pg uses log sequence number to setup and catchup
• Handling node outages - Follower failure: Catch-up recovery, Leader failure: Failover
• Replication logs > Statement based > Write Ahead log > logical row based How to solve replication lag? Eventual consistency can be any time delay
• Reading own writes( read-after-write consistency)
• Monotonic reads guarantees user dont see things moving backward in time.
• Consistent Prefix Reads- writes that are causally related to each other are written to the same shard

Multi-Leader Replication - allow more than one node to accept writes. Topoligies - Circle, star, all to all Sync Engines and Local-First Software like google calendar needs multi leader Conflict avoidance useful for id generation last write wins = discarding concurrent writes

Two families of algorithms for automatic conflict resolution: > Conflict-free replicated datatypes (CRDTs) = in distributed databases such as Redis Enterprise, Riak, and Azure Cosmos DB > Operational Transformation (OT) = used for real-time collaborative editing of text, e.g. in Google Docs

Leaderless Replication Client sends write to multiple nodes simultaneosuly Challenge in Catching up on missed writes

• Read repair
• Hinted handoff
• Anti-entropy

Quorums for reading and writing * Rule: w + r > n ( sets of nodes used by the read and write operations overlap in at least one node) . if there are n replicas, every write must be confirmed by w nodes to be considered successful, and we must query at least r nodes for each read version number or timestamp can help in finding which write is latest

Chapter 7. Sharding primary reason for sharding a database is scalability: sharding is used to implement multitenant systems. Sharding types: 1. key range 2. Hash of Key A consistent hashing algorithm is a hash function that maps keys to a specified number of shards in a way that satisfies two properties: * the number of keys mapped to each shard is roughly equal, and * when the number of shards changes, as few keys as possible are moved from one shard to another.

Chapter 8. Transactions Literally executing transactions in a serial order slows things. Two-phase locking needs lot of corrindation. Serializable snapshot isolation (SSI) is new where optimistic approach, allowing transactions to proceed without blocking. When a transaction wants to commit, it is checked, and it is aborted if the execution was not serializable.

Chapter 9. The Trouble with Distributed Systems

• Packet lost
• Clock not in sync
• Process pause

distributed algorithms rely on timeouts to determine whether a remote node is still available. Quorum needed to arrive at meaningful information. Distributed systems help in Scale, latency and fault tolerance.

Chapter 10. Consistency and Consensus Coordination services like ZooKeeper and etcd are also built on top of consensus algorithms. They provide locks, leases, failure detection, and change notification features that are useful for managing the state of distributed applications.

Chapter 11. Batch Processing Modern batch processing has been heavily influenced by MapReduce Batch processing landscape has changed by below: * more sophisticated caching and execution strategies * focus has shifted toward usability. * New processing models such as dataflow APIs, query languages, and DataFrame APIs are now widely supported. * Job and workflow orchestration has also matured * Batch storage layers are shifting from distributed filesystems (DFSs) like HDFS, GlusterFS, and CephFS to object storage systems such as S3 * if the job’s working set is larger than the available memory, the sorting approach has the advantage that it can make efficient use of disks * Datasets that are too large to fit in memory or local disk present a problem, is the challenge

What problem does Databricks solve?

Databricks offers a unified platform combining data lakes (flexible, cheap) and data warehouses (structured, governed, quality) known as a data lakehouse.

Storage --> Schema --> Table1, Table2