Distribution transactions or any situation where a single event results in the changes of two or more separate data sources which cannot be committed atomically are the well known problematic part of distributed applications. Let's take a look how to approach it.
The problem is, especially for naive implementations, it works most of the time, but there is something named Fallacies of Distributed Programming . Just in case, let's have a quick recap. We are talking about a set of false assumptions people tend to have about distributed systems like the network is reliable, latency is zero, bandwidth is infinite, topology doesn’t change and so on. And this makes distributed transactions quite challenging. I think this is a really crucial topic and every distributed systems developer should be familiar with that.
A classic approach how to tackle distributed transactions is the Two-phase commit (~2PC) . The point of 2PC is it performs tentative operations first and if all succeed the commits are confirmed. So basically, it prepares the operation which is valid for certain amount of time or until acknowledged. This is possible to implement in a reactive way using asynchronous messaging. Of course, we need to have at-least-once delivery guarantee and therefore we need to have some kind of persistent storage and also we have to take care of deduplication or we have to use idempotent messages and so on. But the main problems are, it does not scale well, it’s slow and has certain deadlock issues. As I said, it was possible to implement this in Akka, but there is a better way.
Saga  is a real world solution how to tackle distributed transactions. The point of the saga is that each of the local commits also contains a counterpart. The counterpart is a compensating action of the corresponding transaction. The textbook example is that we have commits or subtransactions if you will, like book a hotel, buy tickets, rent a car with compensating actions like canceling the hotel or the car and so on. If something goes wrong, we apply all compensating actions and reverse the transaction. Again, we have to take care of requirements like at-least-once delivery, deduplication and so on.
But this is the way you should go.
The Big Trade-Off
But there is an alternative option. Try to avoid distributed transactions where possible. Meaning that every business event needs to result in a single synchronous commit and the other data source could be updated asynchronously. So basically we’re going to give up all-or-nothing approach and we’re going to introduce eventual consistency in our system. And this is often the best of our bad options.
The outcome of this article should be. Distributed transaction are hard, expensive, fragile and do not scale well and should be avoided where possible. If you cannot avoid them, go for the saga pattern.
And that's all for today, hope you found it interesting. As always, if you have any questions or comments, don't hesitate to let us know.
References & Further Reading