Shift Left or
Stay Right?

Here's a truth that marketing materials won't tell you: most production data platforms use both batch and streaming. The question isn't "which is better?" but "which is right for this specific use case?"

Consider two scenarios from the same company:

Same company, same data, different paradigms. Both essential. This guide examines both approaches honestly: their genuine strengths, hidden complexities, and what you should understand before choosing.

Architecture Patterns

Let's look at how modern data platforms implement each paradigm. These aren't theoretical. They're patterns running in production at scale.

Modern Batch Architecture

The modern batch stack has evolved significantly from traditional ETL. dbt brought software engineering practices to data transformation. Open table formats like Apache Iceberg enable flexible, scalable data lakehouse architectures.

Key Components Compared

Code Examples

Same problem, different solutions. Let's see how each paradigm handles real-world use cases.

Example 1: Daily Revenue by Category

A straightforward aggregation. This is where batch typically shines.

-- Batch: Simple, readable, testable
SELECT
    DATE(order_timestamp) AS order_date,
    category,
    SUM(amount) AS daily_revenue,
    COUNT(*) AS order_count
FROM {{ ref('orders') }}
WHERE order_timestamp >= DATEADD('day', -30, CURRENT_DATE())
GROUP BY 1, 2

-- Streaming: Continuous, requires windowing
SELECT
    TUMBLE_START(order_time, INTERVAL '1' DAY) AS order_date,
    category,
    SUM(amount) AS daily_revenue,
    COUNT(*) AS order_count
FROM orders
GROUP BY
    TUMBLE(order_time, INTERVAL '1' DAY),
    category

Example 2: Fraud Detection (Velocity Check)

Detect when a user makes transactions from geographically distant locations within an impossible timeframe.

# Batch: Run hourly - detects fraud after the fact
from pyspark.sql import functions as F
from pyspark.sql.window import Window

user_window = Window.partitionBy("user_id").orderBy("timestamp")

flagged = (
    transactions
    .withColumn("prev_lat", F.lag("lat").over(user_window))
    .withColumn("prev_lon", F.lag("lon").over(user_window))
    .withColumn("prev_time", F.lag("timestamp").over(user_window))
    .withColumn("distance_km", haversine_udf("lat", "lon", "prev_lat", "prev_lon"))
    .withColumn("time_diff_min", (F.col("timestamp") - F.col("prev_time")) / 60)
    .filter((F.col("distance_km") > 500) & (F.col("time_diff_min") < 30))
)

-- Streaming: Detect in real-time, block before damage
CREATE TABLE payments (
    txn_id STRING,
    user_id STRING,
    amount DECIMAL(10, 2),
    lat DOUBLE,
    lon DOUBLE,
    event_time TIMESTAMP(3),
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH ('connector' = 'kafka', ...);

-- Detect impossible velocity: >500km in <30 min
INSERT INTO fraud_alerts
SELECT p1.user_id, p1.txn_id, p2.txn_id,
       ST_Distance(ST_Point(p1.lon, p1.lat), ST_Point(p2.lon, p2.lat)) / 1000 AS km
FROM payments p1, payments p2
WHERE p1.user_id = p2.user_id
  AND p1.event_time < p2.event_time
  AND p2.event_time < p1.event_time + INTERVAL '30' MINUTE
  AND ST_Distance(...) > 500000;

Operational Complexity

The 3 AM pager test: which system is easier to debug and fix when something goes wrong?

Batch Operations

Streaming Operations

Decision Framework

Use this framework to choose the right paradigm for each use case. Don't force one approach everywhere.

Clear Wins for Batch

Clear Wins for Streaming

The Bottom Line

Neither paradigm is universally superior. They optimize for different outcomes:

• Batch optimizes for simplicity, completeness, and cost-effectiveness for periodic workloads
• Streaming optimizes for latency, reactivity, and continuous processing

Most modern data platforms use both. Stream for operational use cases where latency matters. Batch for analytical use cases where completeness matters. Don't let ideology drive architecture decisions.

My recommendation: Start with batch. It's simpler to build, test, and operate. Add streaming pipelines for specific use cases where the latency requirement justifies the complexity. Measure everything. Let data guide your architecture evolution, not hype.