AI-Driven Lineage: The Foundation for Fair and Transparent Systems
DOI:
https://doi.org/10.15680/IJCTECE.2021.0406001Keywords:
AI Lineage, Data Provenance, Fairness, Accountability, Transparency, Explainable AI, Responsible AI, Data Governance, Model Interpretability, Ethical AI PracticesAbstract
As artificial intelligence (AI) systems become increasingly integral to decision-making processes across various sectors, ensuring their fairness, accountability, and transparency has become paramount. AI-driven lineage— the ability to trace and document the entire lifecycle of data and model transformations—emerges as a critical component in achieving these objectives. This paper explores the role of AI-driven lineage in fostering responsible AI practices, focusing on its impact on fairness, accountability, transparency, and ethics (FATE). We examine existing tools and methodologies, propose a comprehensive framework for implementing AI-driven lineage, and discuss its implications for regulatory compliance and ethical governance.
References
1. Moreau, L., et al. The Open Provenance Model Core Specification. Future Generation Computer Systems, 27(6), 743–756.
2. Davidson, S. B., & Freire, J. Provenance and scientific workflows: Challenges and opportunities. Proceedings of the 2008 ACM SIGMOD.
3. Gebru, T., et al. Datasheets for Datasets. arXiv preprint arXiv:1803.09010.
4. Holland, S., et al. The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards. arXiv preprint arXiv:1805.03677.
5. Schelter, S., et al. Automatically Tracking Metadata and Provenance of Machine Learning Experiments. Data Engineering Bulletin, 41(4), 39–50.
6. NISTRisk Management Framework (AI RMF) 1.0. National Institute of Standards and Technology.
7. European CommissionArtificial Intelligence Act – Proposal for Regulation of the European Parliament.
8. Mittelstadt, B., et al.The ethics of algorithms: Mapping the debate. Big Data & Society, 3(2).
9. Pasquale, F. The Black Box Society: The Secret Algorithms That Control Money and Information. Harvard University Press.
10. Koshy, R., et al. Data Governance and Lineage in Regulated AI Systems. Journal of Data and Information Quality, 14(3), 1–25.
11. Amershi, S., et al. Software Engineering for Machine Learning: A Case Study. Proceedings of the ICSE-SEIP '19, 291–300.
12. Sculley, D., et alHidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems, 28, 2503–2511.
13. Wang, D., et alDesigning Transparency for Machine Learning Systems. CHI Conference on Human Factors in Computing Systems.
14. Lin, T., et al. Provenance-Driven Monitoring in Machine Learning Pipelines. Proceedings of the VLDB Endowment, 14(6), 991–1003.
15. Pachyderm. (20https://www.pachyderm.io
16. OpenLineage. (2024). https://openlineage.io
17. MLflow Documentation. (2023). https://mlflow.org
18. Apache Atlas. https://atlas.apache.org
19. Comet ML. https://www.comet.com
20. Microsoft. Responsible AI Standard. https://www.microsoft.com/en-us/ai/responsible-ai
