Serverless ETL with Auto-Scaling Triggers: A Performance-Driven Design on AWS Lambda and Step Functions

Abstract

The proliferation of cloud-native architectures has catalyzed a fundamental shift in data engineering paradigms, with serverless computing emerging as a transformative approach for Extract, Transform, and Load operations that exhibit variable workload patterns and irregular temporal characteristics. This article investigates the design, implementation, and comprehensive performance evaluation of a production-grade serverless ETL architecture leveraging AWS Lambda for compute execution and Step Functions for workflow orchestration, systematically addressing critical challenges including cold-start latency penalties, concurrency management under burst loads, and cost optimization across heterogeneous data volumes. Through rigorous empirical analysis spanning batch sizes from megabyte-scale events to hundred-gigabyte datasets under diverse concurrency scenarios, this article demonstrates that properly architected serverless ETL pipelines achieve linear scalability characteristics with near-perfect correlation between execution time and input data volume, while delivering substantial cost reductions for sporadic and low-frequency workloads compared to persistent cluster-based infrastructure. The experimental evaluation reveals critical performance thresholds, including cold-start latency profiles, break-even points between serverless and traditional architectures based on execution frequency, and auto-scaling responsiveness patterns that inform deployment decisions for production environments. The article establishes that serverless ETL represents a workload-dependent optimization rather than a universal best practice, with economic advantages manifesting primarily in scenarios characterized by unpredictable data arrival patterns, intermittent processing requirements, and elastic scaling demands that traditional infrastructure cannot efficiently accommodate without incurring significant idle resource costs and operational overhead.