Self-Optimizing Pipelines ML Systems That Tune Themselves in Production
DOI:
https://doi.org/10.15680/IJCTECE.2025.0802015Keywords:
Self-optimizing pipelines, production machine learning, online learning, concept drift, Bayesian optimization, MLOps, multi-armed bandits, automated machine learning, adaptive systemsAbstract
Production machine learning pipelines are typically built, trained, and tested using historical data, then deployed with a fixed set of settings. However, real-world environments are constantly changing. Customer behavior evolves, data patterns shift, and system resources fluctuate. When performance drops, data scientists must manually investigate the problem, retune the model, and redeploy — a process that is slow, expensive, and prone to human error. This article introduces Self-Optimizing Pipelines (SOPs) — machine learning systems that automatically adjust their own configuration while running in production, without any human intervention. An SOP continuously monitors its own performance, detects when the data distribution has changed, searches for better settings (such as hyperparameters, feature selections, and ensemble weights), and safely deploys those improvements. We present a complete system architecture with five interconnected modules. Using a real-world credit card fraud detection dataset with simulated concept drift, we show that an SOP achieves an average F1-score of 0.93 compared to 0.71 for a static pipeline — a 31% improvement. The SOP reduces the need for manual data scientist intervention by 94% and adds only a small increase in processing time (from 12 milliseconds to 16 milliseconds per prediction), which is acceptable for most real-time applications. Our results demonstrate that self-tuning production systems are not only feasible but also stable, maintainable, and ready for deployment with existing open-source tools.
References
1. Widmer, G., & Kubat, M. (1996). Learning in the presence of concept drift and hidden contexts. Machine Learning, 23(1), 69–101, https://doi.org/10.1023/A:1018046501280.
2. Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014). A survey on concept drift adaptation. ACM Computing Surveys, 46(4), 1–37, https://doi.org/10.1145/2523813.
3. Bifet, A., & Gavaldà, R. (2007). Learning from time-changing data with adaptive windowing. Proceedings of the 2007 SIAM International Conference on Data Mining, 443–448, https://researchr.org/publication/BifetG07.
4. Gama, J., Medas, P., Castillo, G., & Rodrigues, P. (2004). Learning with drift detection. Brazilian Symposium on Artificial Intelligence, 286–295.
5. Bergstra, J., Bardenet, R., Bengio, Y., & Kégl, B. (2011). Algorithms for hyper-parameter optimization. Advances in Neural Information Processing Systems, 24, 2546–2554, https://nips.cc/virtual/2011/poster/2579.
6. Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical Bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems, 25, 2951–2959, https://researchr.org/publication/SnoekLA12.
7. Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2018). Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(185), 1–52, https://jmlr.org/papers/v18/16-558.html.
8. Perrone, V., Shen, H., Zolic, I., Shcherbatyi, I., Ahmed, A., Böhmer, W., & Archambeau, C. (2019). Bayesian optimization with scalable constraints and contextual variables. arXiv preprint arXiv:1912.04427, https://doi.org/10.48550/arXiv.1910.07003.
9. Tokui, N., Ota, K., & Sato, I. (2019). Ensemble selection using Thompson sampling. Proceedings of the 28th International Joint Conference on Artificial Intelligence, 3688–3694, https://doi.org/10.24963/ijcai.2019/512.
10. Jomaa, H. S., Schmidt-Thieme, L., & Grabocka, J. (2021). Online automated machine learning with cost-sensitive resource allocation. 2021 IEEE International Conference on Data Mining (ICDM), 1132–1137, https://doi.org/10.1109/ICDM51629.2021.00134.
11. Kephart, J. O., & Chess, D. M. (2003). The vision of autonomic computing. Computer, 36(1), 41–50, https://doi.org/10.1109/MC.2003.1160055.
12. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J. F., & Dennison, D. (2015). Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems, 28, 2503–2511, https://proceedings.neurips.cc/paper/2015/.
13. Liberty, E., Karnin, Z., Xiang, B., Rouesnel, L., Coskun, B., Nallapati, R., Delgado, J., Sadoughi, A., Astashonok, Y., Das, P., Balioglu, C., Chakravarty, S., Jha, M., Gautier, P., Arpin, D., Januschowski, T., Bohlke-Schneider, M., Wang, H., & Smola, A. (2020). Elastic machine learning algorithms in Amazon SageMaker. Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, 2337–2348, https://doi.org/10.1145/3318464.3386126.
14. Baylor, D., Breck, E., Cheng, H. T., Fiedel, N., Foo, C. Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., Koo, C., Lew, L., Mewald, C., Modi, A. N., Polyzotis, N., Ramesh, S., Roy, S., Whang, S. E., Wicke, M., & Zinkevich, M. (2017). TFX: A TensorFlow-based production-scale machine learning platform. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1387–1395, https://doi.org/10.1145/3097983.3098021.
15. Hutter, F., Kotthoff, L., & Vanschoren, J. (2019). Automated machine learning: Methods, systems, challenges. Springer Nature, https://doi.org/10.1007/978-3-030-05318-5.
