Change Data Capture: Architecture, Failure Modes, and How to Keep It Working

Transparency note: This analysis is based on production patterns, internal benchmarks, and publicly documented system behaviors. Numbers without explicit citations are observed across enterprise deployments; cited numbers link to original sources. Actual performance varies by workload, scale, and configuration.

What Most Teams Get Wrong

Change data capture aims to efficiently track and apply changes across distributed systems. It assumes data distribution remains balanced during processing.

Trigger: Uneven data distribution. Consequence: Executor OOM due to task skew. Impact: Task completion time increases by 50% in Spark jobs.

How It Actually Works (Under the Hood)

• Log-based CDC captures changes from transaction logs.
• Trigger-based CDC uses database triggers to detect changes.
• Batch processing applies changes in bulk to target systems.
• Streaming processing applies changes in near real-time.
• Checkpointing ensures data consistency across failures.

Hard Numbers (defaults and thresholds)

Real-World Constraints

• spark.executor.memory must be tuned to workload size
• spark.sql.shuffle.partitions impacts shuffle performance
• spark.speculation can cause redundant task execution
• Log-based CDC requires access to transaction logs
• Trigger-based CDC can impact source database performance

Failure Modes (Trigger → Mechanism → Consequence → Impact)

What the failure looks like live

21/10/23 14:32:10 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 1456 ms on executor 1 (host: 10.0.0.1) with OOM

Production Reality (What Breaks at Scale)

At 1TB+ data volumes, task skew becomes pronounced because data distribution is uneven; the only mitigation that works is to increase the number of shuffle partitions and optimize data partitioning.

Expert insight: Task skew often arises from poorly distributed keys; pre-sorting data by key can mitigate this issue significantly.

Hidden Costs of Maintenance

• Ongoing tuning of executor memory settings
• Frequent monitoring of task distribution
• Adjusting shuffle partitions for different workloads
• Managing speculative execution settings
• Ensuring consistent access to transaction logs

How Engines Differ

X vs Alternatives

How to Keep It Actually Working

• Set spark.executor.memory to 2g for high-volume jobs
• Increase spark.sql.shuffle.partitions to 500 for large shuffles
• Disable spark.speculation for stable environments
• Pre-sort data by key to avoid task skew
• Monitor Spark UI for task distribution anomalies

Standards and Industry Guidance

Standards and frameworks that apply to change data capture in production environments:

• ISO/IEC 25010 - SQuaRE — reliability (maturity, availability, fault tolerance) is the relevant quality characteristic for production pipelines
• NIST SP 800-53 Rev. 5 — SI-4 (monitoring) and CP-10 (information system recovery) apply to pipeline observability and failure recovery
• ISO 8000 - Data Quality — the data quality discipline pipelines exist to maintain end-to-end
• ISO/IEC 27001 — change-management discipline for production pipeline modifications

Where It Matters Most

E-commerce

Real-time inventory updates using log-based CDC.

Finance

Fraud detection with streaming CDC for transaction monitoring.

Healthcare

Patient data synchronization across systems with batch CDC.

The Underlying Principle (and Where Solix Fits)

The principle behind change data capture is to maintain data consistency across distributed systems by efficiently tracking and applying changes. Solix CDP implements this by providing a scalable and flexible platform for managing CDC workflows, while acknowledging that other vendors also target this critical need with their solutions.

Prerequisite Concepts

• Apache Spark — A unified analytics engine for large-scale data processing.
• Data Partitioning — The process of dividing data into distinct subsets for parallel processing.
• Executor Memory — Memory allocated to each executor in a Spark cluster.

Frequently Asked Questions

What is change data capture in simple terms?

Change data capture is a method to track and apply changes in data sources to ensure consistency across systems.

How is change data capture different from data replication?

CDC tracks and applies only changes, whereas data replication copies entire datasets.

Why is my change data capture suddenly slow?

Check for task skew or insufficient shuffle partitions causing delays in processing.

How do I tell if change data capture is broken?

Look for increased lag or failed tasks in the Spark UI indicating processing issues.

Related Glossary Terms

• Metadata Management
• Data Governance
• Data Lineage
• Data Catalog

Trademark Notice

Product names, logos, brands, and other trademarks referenced on this page are the property of their respective trademark holders. References to third-party products are for descriptive and informational purposes only and do not imply affiliation, endorsement, or sponsorship by the trademark holders. Solix Technologies is not affiliated with, endorsed by, or sponsored by any third party referenced on this page unless explicitly stated.