Real-Time Big Data Processing with Machine Learning Models
DOI:
https://doi.org/10.15680/IJCTECE.2019.0204002Keywords:
Real-time processing, big data, machine learning, streaming analytics, Apache Kafka, Apache Flink, model drift, Lambda architecture, scalability, latencyAbstract
Real-time big data processing with machine learning models has become a cornerstone of modern data analytics, enabling organizations to derive actionable insights from vast streams of data as they are generated. This capability is particularly valuable in sectors such as finance, healthcare, e-commerce, and telecommunications, where timely decision-making can significantly impact outcomes. The integration of machine learning into real-time data processing pipelines allows for continuous model training and inference, adapting to new data patterns and providing up-to-date predictions. However, implementing such systems presents challenges related to data velocity, model drift, scalability, and latency. This paper explores the methodologies, tools, and architectures employed in real-time big data processing with machine learning, highlighting best practices and emerging trends.
References
1. Ghosh, S., & Ghosh, S. (2019). Real-time big data processing for predictive analytics using machine learning. Journal of Data Science and Predictive Analytics, 3(2), 145-161.
2. Chen, J., & Li, L. (2020). Real-time data processing with Apache Kafka and machine learning models. International Journal of Cloud Computing and Data Science, 5(4), 88-101.
3. Zhang, Y., & Song, J. (2021). Machine learning-based real-time big data analytics: Challenges and solutions. Big Data Research and Applications, 7(1), 24-42.
4. He, X., & Sun, Y. (2022). Scalability and fault tolerance in real-time big data processing systems. IEEE Transactions on Big Data, 8(6), 1227-1239.
5. Babcock, B., & Olston, C. (2020). Scalable stream processing with Apache Flink: A review of frameworks and algorithms. Proceedings of the ACM SIGMOD International Conference on Management of Data, 1009-1021.
