Distributed-Systems

Most applications need to do two different kinds of work: create, update, and delete data safely, and retrieve it in a shape that screens and reports can use efficiently. At Endowus, we handle these needs with CQRS and event sourcing across services built with Scala, Akka, and Kafka. CQRS — Command Query Responsibility Segregation — is a design pattern that separates those two kinds of work instead of using one model for both. Commands such as placing an investment go through a write model that enforces business rules, while screens query a read model shaped for fast retrieval, filtering, and display. Event sourcing stores the history of a business record as a sequence of events instead of overwriting it with only its latest state. When an entity accepts a command, it saves an event describing what happened and applies that event to update its state. If the entity restarts, it can rebuild the same state by replaying its events in order. In our implementation, a separate event processor uses the stored events to update the read model asynchronously. This means the entity can complete a command before the read model has caught up, creating a short eventual-consistency gap. For example, when a client clicks Invest, the write-side entity can process the instruction and return a confirmation before the event processor has updated the read model. If the client then opens Activities & Balances, the screen can still return the older state and leave the new investment temporarily invisible. This post explains how we address that gap with a Read Routing Table (RRT). For each recent client-initiated change, RRT leaves a short-lived breadcrumb pointing to the entity that changed. A subsequent read uses those breadcrumbs to fetch and merge the latest state for only those entities into the normal read-model response.