Cloud-Enabled Data Lake Architectures for Large-Scale Autonomous Vehicle Datasets with Neural Network Integration

Abstract

The exponential growth of autonomous vehicle (AV) technologies has led to the generation of massive, heterogeneous datasets requiring scalable and efficient storage, processing, and analysis. This paper proposes a cloud-enabled data lake architecture tailored for large-scale autonomous vehicle datasets, enabling seamless data ingestion, integration, and retrieval across multimodal sources such as LiDAR, radar, cameras, and vehicle-to-infrastructure (V2I) communication streams. By leveraging neural networks, the architecture facilitates advanced data analytics, including object detection, trajectory prediction, and anomaly detection, thereby enhancing decision-making for autonomous systems. The integration of cloud-native services ensures elasticity, high availability, and real-time data processing, while schema-on-read capabilities provide flexibility in handling structured and unstructured data. Experimental validation demonstrates that the proposed architecture reduces data latency, supports scalable training of deep learning models, and enhances the robustness of AV applications. This work contributes to building intelligent, AI-powered ecosystems that accelerate the safe deployment of large-scale autonomous transportation systems.