Twenty services, one patient

A hospital is a queue of people moving through rooms. Registration, then the waiting area, then a consulting room, then the pharmacy, then the cashier. Each of those is a different department, with different staff, different hours, and a different idea of what “done” means.

That maps almost too neatly onto services, and the neatness is a trap. The moment registration and pharmacy are separate deployables with separate databases, you have to answer a question a single application never asks: when a patient is registered, and the queue board needs to know, and the pharmacy will need to know later, and billing will need to know later still — what exactly happens?

The two obvious answers are both wrong

The first is orchestration: one coordinating service that calls the others in the right order. It works, briefly. Then that service knows about all twenty, every new module edits it, and its failure takes down registration. You have rebuilt the single application, with network latency and worse error handling.

The second is to make the state change and the notification commit together, across systems — a distributed transaction. Two-phase commit across twenty services and a message broker is operationally miserable, and in practice it converts a broker outage into a hospital that cannot register patients. That is a worse failure than any it prevents.

What we do instead

When the patient service records a visit, it writes two things in the same database transaction: the visit row, and a row in an outbox_events table describing what happened. One transaction, one database, ordinary guarantees. Either both are there or neither is.

A separate dispatcher polls that table and publishes pending rows to the broker, marking them dispatched as it goes. If the broker is down, rows accumulate and go out later. If the dispatcher dies mid-publish, the row is still pending and goes again.

At-least-once is not a compromise we settled for. It is the guarantee we chose, because the alternative failure is silent.

A message may arrive twice. It will never quietly fail to arrive. In a clinical system that asymmetry is the whole point: a duplicated queue notification is a visible annoyance, and a dropped one is a patient nobody calls.

The cost is idempotency, and the consumer pays it

Every handler has to be safe to run twice on the same message, and must acknowledge only after it has actually succeeded. That is real work and it never stops being real work — it is the price of the guarantee, not a detail of it.

Around the handlers, the broker configuration does the rest: durable quorum queues, a dead-letter exchange, and a delivery limit so a message that fails forever is set aside instead of retried forever.

One footgun worth naming, because it costs an evening the first time. Queue arguments are immutable. Changing a queue’s dead-letter exchange or delivery limit does not take effect on a queue that already exists — the declaration either errors or is quietly ignored, depending on how it was written. So there is a maintenance command that deletes the consumer queues and lets them be redeclared with current arguments, and it has to run with the services stopped. Discovering that during an incident is worse than reading it here.

Authorization cannot be a service call

If every request to every service required a round trip to the auth service, that service would become the one thing whose outage stops the hospital — which is exactly what splitting into services was supposed to avoid.

So it never gets called. The auth service holds a private key and is the only thing that can sign an access token. The token carries the organisation, the role, and a flat list of permissions. Every other service verifies it locally against the public key, fetched once from a well-known endpoint and cached. Authorization becomes a local computation on a signed claim, and twenty services can keep working through an auth service restart.

Reads do not cross services either

Tariffs, locations, beds, staff records — the data one service owns and several others need — are projected into each consumer by the same event stream, and read locally. A service that must make a synchronous call to answer a read has just inherited the other service’s uptime, and told nobody.

What falls out for free

The live patient journey board. Because every stage already publishes an event, the monitoring service can subscribe to all of them and maintain a real-time view of where every patient in the building currently is.

Nobody built a patient-tracking feature. It was a consequence of the architecture, available for the cost of one more subscriber. That is usually the sign the seams were cut in roughly the right place: the capability you did not plan for turns out to be nearly free.