Real-time data

“Real time” is an answer to a question nobody has asked yet. The question is who acts on the data, how quickly, and what goes wrong if it is too old. Only when you know that can you say how fast the flow needs to be, and what it is allowed to cost.

How fresh is fresh enough?

It depends on what happens to a stale value. A stock level that lags behind can sell something you don’t have. A report may be read once a week and still need this morning’s figures for the meeting. An alarm from a machine has to arrive while there is still time to act, whether the receiver is a person or an automatic stop. We ask two questions per flow: how old may the data be at the moment it is used, and what does it cost when it is older than that?

The answers shape the architecture, one flow at a time. Batch and event-driven are two tools, not two camps. Pick one for everything and you either pay for real time where nobody acts in real time, or wait for a nightly run when someone needed the answer that morning.

What an event-driven flow needs

• A queue, so the receiver can fetch at its own pace without the sender waiting.
• Receivers that cope with the same event arriving twice. Check what the queue promises: at-least-once delivery means the receiver has to recognise a duplicate.
• Retries with growing intervals, and a clear end state for whatever still doesn’t get through.
• A way to replay a day of events afterwards, with a plan for which effects should run again and which have already happened.
• Monitoring that alerts on missing flow, not just on errors. A flow that stops produces no error message.

The last one is easy to forget. When a service crashes, you get an error in the log. When a source stops sending, you get a number that stops moving, and nobody notices until someone wonders why it is standing still. So we build alerts on missing data too. If the data goes on to data analysis and reporting or drives process automation, the same flows feed them.