Event Sourcing
A way of storing application state without storing the current state in the database column directly with stronger data integrity, among other benefits, if used right.
I recently learned from a colleague about a programming concept called Event Sourcing. Perhaps it could even be called a paradigm of state storage?
Here's a sketch illustrating the difference between the Event Sourcing way of storing events and replaying to arrive at the current state at any point in time, versus the database way of storing the current state directly in a NoSQL field or SQL column:
Not just logging
The first thing I wondered when I learned about Event Sourcing was whether this was just a fancy way of describing the simple technique of storing logs in a separate database table, in which case, perhaps it should just be called Logging!
But after some digging and talking to a colleague, Event Sourcing is not just Logging. Yes, you can get some positive properties of back-tracking and replaying to a specific moment in time if you stored logs while updating the product.stock
column in the database, but because you're not relying on a replaying mechanism to arrive at the current state in your application but rather still be relying on the database column, you will miss out on many other -- depending on context -- nice features of using Event Sourcing.
Bank ledgers
The classic example of a system that uses Event Sourcing for sure is a banking system that keeps track of individual customer's account balances.
To avoid the tricky situation of a "All hail the column!" throwdown by bankers to their customers, each transaction is recorded as a transaction in a database and there is no second source of truth in the form of a "Balance" column. This means that every time you log in to your bank's online banking system and request the Account Overview page, their back end system will lookup the entire history of transactions in a given period and replay the computations to arrive at the current state!
This surprised me when I first heard it because, wow, how slow is that, recomputing the state on every request?! But then I remembered that there are complementary techniques to make repeated requests faster, like caching the result, transactions length, and timeframe, and comparing the new request with the cached one. If they match, then no need to recompute, just return the result; otherwise, re-compute and cache again.
And now I understand why my banks always show my Account Overview as either a moving window of this month's transactions or a moving window of the last 30 transactions. It's a reasonable technique to keep the replay costs of Event Sourcing low in this context, since as a user I'm most of the time trying to find out recent transactions anyway! But to make that happen, since my balance is always the arithmetic calculation of every single transaction I’ve made since the beginning of my account (which in my oldest bank account’s case is probably over 20 years worth of transactions), the system probably uses a mechanism for storing a snapshot of how much the balance is across various periods leading up to now, and then computing from that. Otherwise each time it would still end up loading every single transaction beginning from 20 years ago. That implementation detail is still fuzzy to me at the moment.
Poorer read times?
Intuitively, I think Event Sourcing is going to be generally slower to respond to a request asking for a particular state (e.g. account balance).
But beware the blind pursuit of speed! If there were 1000 records to be retrieved and 1000 simple arithmetic operations to be carried out to arrive at an account balance, would that palpably slower than reading 1 record and returning the value in one of the columns? Excluding any optimisation, probably yes.
But let’s say you were implementing Event Sourcing to keep track of the Subscription Plan of a SaaS company’s user account, you might only have, say, 10 or 50 records to retrieve and computations to be made. Would that difference be perceivable? Probably less so. And then, with optimisations like said caching, we’re on our way to the equivalent user experience but improved data integrity and debuggability!
I think an important takeaway here is that Event Sourcing should not be the default way of approaching state storage. Certain contexts call for it; others are probably okay with the store-in-a-column approach.
Concurrency and Idempotency
So we’re throwing big words out now, huh? I guess we are…
Concurrency is many computations happening at the same time, which can lead to problems like database transaction locking (i.e. a request to Server A wants to add 10 dollars to an account balance, and another request, which got load-balanced to Server B wants to add another 20 dollars to the same account - if both Servers tried to update an entry in the same database table, there could be, in some cases, a transaction lock that would prevent the successful completion of either one of them).
Idempotency is a property of a system whereby an action can be applied multiple times without changing the result beyond the initial application. Like if you pushed the button at the pedestrian crossing 5 times, it shouldn’t cause the the traffic lights to provide 5 times of pedestrian crossing signals, just once.
The reason I brought these concepts up is because I’ve been told that Event Sourcing solves some of these problems very well. I don’t know how, and I’m not interested enough at the moment to want to dig into a hypothetical situation to find out. If I have to implement a subsystem at work that uses Event Sourcing, I’ll do another write up to share my learnings with you.