Version-Controlled Analytics: Integrating DBT with GIT for Scalable Data Pipelines
DOI:
https://doi.org/10.15680/IJCTECE.2024.0703002Keywords:
DBT, Git, Data Pipelines, Version Control, CI/CD, Data Engineering, Cloud Data Warehouses, Automation, Collaboration.Abstract
DBT (Data Build Tool) has revolutionized data transformation workflows, enabling data engineers to model, test, and document data within cloud data warehouses. When coupled with Git for version control, DBT enables more efficient collaboration, reproducibility, and error tracking in data engineering teams. This paper explores how integrating DBT with Git can streamline the development and deployment of data pipelines. The research focuses on the advantages of using Git for managing DBT projects, ensuring collaborative workflows, maintaining data pipeline versions, and automating deployments. We discuss best practices for integrating DBT with Git to improve data pipeline efficiency, reduce errors, and ensure a smoother CI/CD process in modern data engineering environments.
References
1. DBT Labs. (2021). DBT: Transforming Data Engineering with Version Control. Retrieved from https://www.dbt.com
2. Vivekchowdary, Attaluri (2023). Just-in-Time Access for Databases: Harnessing AI for Smarter, Safer Permissions. International Journal of Innovative Research in Science, Engineering and Technology (Ijirset) 12 (4):4702-4712.
3. Johnson, T., & Stevens, M. (2020). Version Control for Data Engineers: Integrating DBT with Git. Journal of Cloud Data Engineering, 10(2), 53-64.
4. Tan, S., & Ouyang, Y. (2020). Automating Data Workflows: Git and DBT in Data Engineering. Data Science Review, 12(3), 102-115.
5. Owen, R. (2022). CI/CD in Data Engineering: Streamlining Data Pipelines with Git and DBT. Journal of Big Data Technologies, 15(1), 75-86.
6. Dhruvitkumar, V. T. (2022). Enhancing Multi-Cloud Security with Quantum-Resilient AI for Anomaly Detection.
7. Zhao, L. (2021). Collaboration and Version Control in Cloud Data Engineering with Git and DBT. International Journal of Data Engineering, 13(4), 99-109.
