A batch job that runs for ten minutes a night is billed for about five hours a month. A streaming pipeline is billed for every hour of the month, roughly 730 of them, whether events are flowing or not. That gap is the starting point for any decision about real-time data processing.
The request usually arrives as "can we make the dashboards real time?" It sounds like a small upgrade. In practice, moving from nightly batch to streaming changes your architecture, your running costs, your on-call roster and the skills your team needs.
Sometimes it's clearly worth it. A fraud check that runs tomorrow morning is useless. But a large share of real-time requests turn out to mean "fresher than yesterday", and that's usually solved for a fraction of the cost.
Streaming got easier, which isn't the same as cheaper
Managed services have taken most of the cluster babysitting out of streaming. Engines such as Apache Flink handle both unbounded streams, which "have a start but no defined end", and bounded data, the classic batch case. Flink guarantees exactly-once state consistency after failures by periodically checkpointing state to durable storage.
At the same time, customer-facing products compete on immediacy: live delivery tracking, instant fraud checks, in-session recommendations. Expectations leak into internal reporting, and suddenly the sales dashboard "needs" to be live too. Easier tooling doesn't make every use case worth paying for around the clock.
Streaming vs batch, side by side
Batch collects data over a period and processes it in one go: nightly, hourly or every 15 minutes. It's simple to build, easy to rerun when something breaks, and you pay for compute only while the job runs.
Streaming processes each event, or a small window of events, as it arrives, usually within seconds. It needs always-on infrastructure, careful handling of late and duplicate events, and monitoring at all hours, because a stalled stream means stale decisions.
| Factor | Batch | Streaming |
|---|---|---|
| Typical freshness | Minutes to a day | Seconds |
| Compute pattern | Runs on a schedule, then stops | Always on |
| Build complexity | Lower | Higher: ordering, late data, state, replays |
| Fixing bad data | Rerun the job | Replay from retained events, or correct downstream |
| Operations | Check the job succeeded | Watch consumer lag and health continuously |
| Best for | Reporting, finance, planning, model training | Decisions that lose value within minutes |
Between the two sits micro-batch: small batches every few minutes. Spark Structured Streaming with a trigger interval, or a scheduled job every five minutes against a change table, gives many dashboards "fresh enough" data with batch-style simplicity and cost.
What real-time data processing costs to run
The defining cost of streaming is that it bills while nothing is happening. The figures below are AWS list prices for US East (N. Virginia), in USD, taken from AWS's public pricing pages and worked examples. Australian regions are priced differently, so check the Asia Pacific (Sydney) and Asia Pacific (Melbourne) rates on the same pages, or in the AWS Pricing Calculator, before you budget in AUD.
- Kinesis Data Streams. In On-demand Standard mode, AWS's pricing examples use a per-stream charge of USD 0.040 per stream-hour, plus USD 0.08 per GB ingested and USD 0.040 per GB retrieved. That per-stream charge alone is about USD 29 a month for every stream before a single byte flows. In provisioned mode, each shard is USD 0.015 per hour, used or not. There's also an On-demand Advantage mode with no per-stream charge and lower per-GB rates, but it carries an account-level minimum throughput commitment.
- Managed Service for Apache Flink. Billed per Kinesis Processing Unit (KPU) hour, USD 0.11 in US East, where one KPU is 1 vCPU and 4 GB of memory. Each streaming application is charged one extra KPU for orchestration, plus running application storage (50 GB per KPU, at USD 0.10 per GB-month in AWS's example). The smallest possible application, one KPU plus the orchestration KPU, works out to roughly USD 160 a month in compute before storage.
Those numbers aren't large on their own. The bigger costs are people: engineering time for ordering, duplicates and schema changes; on-call cover, because a stream that stops at 2am matters at 2am; and testing, which is harder for a system that never stops than for a job you can rerun.
That's why our first question is always "what decision changes if this data is seconds old instead of an hour old?"
Where streaming pays off
Streaming earns its running cost when the data loses value within minutes and an automated action, not just a chart, depends on it.
- Fraud and abuse detection. Card payments, account takeover and promo abuse need stopping before the transaction completes.
- Operational alerting and IoT. Cold chain, equipment and network telemetry, where a late alert means spoiled stock or downtime.
- Live customer experiences. Delivery tracking, stock availability at checkout, in-session recommendations.
- Inventory and pricing on fast channels. Overselling during a sale costs more than the stream does.
- Event-driven integration. Change data capture (Debezium, or AWS DMS with ongoing replication) keeps systems in sync without nightly file drops, and can simplify the architecture overall.
When batch is the right call
Batch, or frequent micro-batch, is usually the better answer for:
- Management, finance and board reporting.
- Sales and marketing dashboards checked a few times a day.
- Forecasting, planning and most model training.
- Regulatory and compliance reporting with fixed periods.
- Anything where a person reads the number and decides later.
For real-time dashboards, ask who watches the screen and what they do when a number moves. If nobody acts within minutes, a 15-minute or hourly refresh gets the same business outcome. A common middle path is streaming ingestion into the warehouse with dashboards that refresh on a schedule: data lands quickly, but you're not running a stream processor for charts nobody's watching.
Four questions before you fund it
- What decision or action happens on this data, and who or what takes it?
- How much value is lost for each minute of delay?
- Can the action be automated, or does a person need to review it first?
- Is the data available in real time at the source, or does the source system itself batch it?
If the answers are "an automated action", "a lot", "yes" and "yes", streaming is likely worth it. If any answer is weak, start with micro-batch and revisit in six months. The foundations are the same either way: our plain-English guide to warehouses and lakehouses covers where the data lands, and a lean governance framework keeps it trustworthy.
How JRC Software approaches it
Our data engineering, analytics and insights team starts by listing every "real-time" request and writing down the freshness each one actually needs, with the person who asked for it. In our experience most organisations end up mostly batch or micro-batch, with one or two streaming pipelines where the value is obvious.
You get a cost model in your own region and currency, including the always-on components and the support time, before anything is built. When streaming is justified, we design it to be operable: managed services where they fit, event retention long enough to replay a bad day, consumer-lag alarms in CloudWatch or Azure Monitor routed to a real roster, and a runbook for "the stream stopped".
Two things we often talk teams out of: running their own Kafka cluster for one pipeline, and streaming into a dashboard that's refreshed once a morning. Our engineers in Melbourne and Sri Lanka cover more of the day between them, which helps with pipelines that never stop.
If you're weighing up a real-time project and want a second opinion before you commit, get a data architecture opinion from our team.
Frequently asked questions
What is real-time data processing?
Real-time data processing means handling each event, such as a payment, sensor reading or click, within seconds of it happening, rather than collecting data and processing it later in batches. It runs on always-on streaming infrastructure and is used where decisions lose value quickly: fraud detection, operational alerting and live customer experiences.
Is streaming always more expensive than batch?
For the same workload, usually yes. Streaming components run continuously and are billed by the hour, while batch jobs consume compute only while they run. Streaming also needs more engineering and on-call effort. The extra cost is justified when fresher data produces measurable value, for example by preventing fraud losses or equipment downtime.
Do real-time dashboards need streaming?
Usually not. Many dashboards described as real time are looked at a few times a day, and an automated refresh every 15 minutes or hourly gives the same practical result. Streaming makes sense for screens that people or systems act on within minutes, such as operations centres, live logistics tracking or incident monitoring.
Can we start with batch and add streaming later?
Yes, and it's often the best approach. A batch platform with clean models, quality tests and clear ownership makes it easier to add streaming for specific use cases later. Capturing source changes as events from the start, where your systems support change data capture, keeps that option open without paying for always-on processing before you need it.
Data & Analytics
Get a data architecture opinion
Tell us where you're stuck and we'll come back with practical next steps.
Get in touch