Observability-Driven Reliability Engineering for Distributed Data Platforms
DOI:
https://doi.org/10.15680/IJCTECE.2024.0704019Keywords:
Observability, OpenTelemetry, Distributed Systems, Site Reliability Engineering, CloudWatch, GrafanaAbstract
Reliability has emerged as an important project of data processing, streaming, and mission-critical analytics workloads in cloud-native environments and is supported by distributed data platforms. Nonetheless, detection, diagnostics, and prevention of service degradation by a distributed architecture have proven hard due to the heterogeneous and dynamic nature of distributed architectures. To offer a persistent state of visibility of data-platform elements, the paper suggests a plan of reliability engineering based on observability that considers the integration of telemetry data collection, distributed tracing, metrics, logs, and service-level indicators. The solution used is advised depending on the utilization of OpenTelemetry as the instrumented telemetry to apply and integrate observability pipelines and monitoring, visualization, alert, and operational analysis systems (e.g., Amazon CloudWatch and Grafana). The framework focuses on telemetry signal correlation, dependency failure analysis, anomaly detecting, and service-level-level objective monitoring to reduce the mean time to detect and resolve. Other practices that are a part of reliability are incident response, capacity planning, fault isolation, and further performance evaluation. The method offers a systematic solution in the conversion of the raw operational telemetry into operational reliability data and can be generalized to the non-homogenous distributed environment. The paper evaluates the building blocks, observability processes, reliability processes, and benefits of the proposed solution. The resultant framework explains how the observability can become an operational ability of boosting availability, resilience, diagnosability and operational integrity of the distributed data platform once it begins itself as an inactive practice of monitoring.
References
[1] M. Usman, S. Ferlin, A. Brunstrom, and J. Taheri, “A survey on observability of distributed edge & container-based microservices,” IEEE Access, vol. 10, pp. 86904–86919, 2022, doi: 10.1109/ACCESS.2022.3193102.
[2] S. He, B. Feng, L. Li, X. Zhang, Y. Kang, Q. Lin, S. Rajmohan, and D. Zhang, “STEAM: Observability-preserving trace sampling,” in Proc. ACM Joint European Software Engineering Conf. and Symp. Foundations of Software Engineering (ESEC/FSE), 2023, pp. 1750–1761, doi: 10.1145/3611643.3613881.
[3] J. Mace, R. Roelke, and R. L. Fonseca, “Pivot tracing: Dynamic causal monitoring for distributed systems,” ACM Trans. Comput. Syst., vol. 35, no. 4, pp. 1–28, 2018, doi: 10.1145/3208104.
[4] J. Kaldor, J. Mace, M. Bejda, E. Gao, W. Kuropatwa, J. O'Neill, K. W. Ong, B. Schaller, P. Shan, B. Viscomi, V. Venkataraman, K. Veeraraghavan, and Y. J. Song, “Canopy: An end-to-end performance tracing and analysis system,” in Proc. 26th ACM Symp. Operating Systems Principles (SOSP), 2017, pp. 34–50, doi: 10.1145/3132747.3132749.
[5] M. Du, F. Li, G. Zheng, and V. Srikumar, “DeepLog: Anomaly detection and diagnosis from system logs through deep learning,” in Proc. ACM SIGSAC Conf. Computer and Communications Security (CCS), 2017, pp. 1285–1298.
[6] F. Pina, J. Correia, R. Filipe, F. Araujo, and J. Cardoso, “Nonintrusive monitoring of microservice-based systems,” in Proc. IEEE 17th Int. Symp. Network Computing and Applications (NCA), 2018.
[7] F. Ahmed, U. Jahangir, H. Rahim, and K. Ali, “Centralized log management using Elasticsearch, Logstash and Kibana,” in Proc. 2020 Int. Conf. Information Science and Communication Technology (ICISCT), 2020, doi: 10.1109/ICISCT49550.2020.9080053.
[8] X. Jiang, Y. Pan, M. Ma, and P. Wang, “Look deep into the microservice system anomaly through very sparse logs,” in Proc. ACM Web Conf. (WWW), 2023, pp. 2970–2978, doi: 10.1145/3543507.3583338.
[9] Z. Huang, P. Chen, G. Yu, H. Chen, and Z. Zheng, “Sieve: Attention-based sampling of end-to-end trace data in distributed microservice systems,” in Proc. IEEE Int. Conf. Web Services (ICWS), 2021, pp. 436–446, doi: 10.1109/ICWS53863.2021.00063.
