Implementing Clinical-Grade Data Pipelines for AI- Driven Diagnostics Using HPC and High-Speed Fabrics
DOI:
https://doi.org/10.15680/d1txb658Keywords:
High-Performance Computing (HPC), Clinical Data Pipeline, Artificial Intelligence (AI), Medical Imaging AnalysisAbstract
This paper has applied a modeling pipeline consisting of AI-based diagnostics made possible through the use of computing based on the principles of high-performance computing (HPC). This system receives, interprets and processes multi-modal healthcare information and data (ICU monitor streams, medical imaging and genomic data). It is also low-latency and heavily throughput with use of high-speed fabrics such as infinity band and NVlink in addition to processing ICU data (1,200 records/sec), imaging (350 GB/hour) and genomic data (420GB/hour). AI model outcomes are promising: having an ICU risk prediction F1-score of 0.76, EfficientNet- B0 of 99.5% internal and External imaging accuracy and 95% of predictions of genomic variations, XGBoost, EfficientNet-B0, and External imaging model are high. The system also ensures fault-tolerance, data integrity besides the scaling resource efficiency among the distributed nodes of HPC. The experience of integrating in NYU Hospitals and City of Hope increased the clinical process, decreasing the alert time in the ICUs to 40 per cent, clinician satisfaction increased to 4.5/5. The article indicates that at present HPC-capable AI pipelines are able to deliver reproducible, real-time and clinically actionable insights to offer precision healthcare
References
[1] Li, J., Wang, S., Rudinac, S., & Osseyran, A. (2024). High-performance computing in healthcare: An automatic literature analysis perspective. Journal of Big Data, 11(1). https://doi.org/10.1186/s40537- 024-00929-2
[2] Softić, A. (2025). Accelerating Innovation in Healthcare Through High-Performance Computing: Applications and Future Perspectives. Accelerating Innovation in Healthcare Through High- Performance Computing: Applications and Future Perspectives, 287–299. https://doi.org/10.5644/pi2025.220.15
[3] Koch, M., Arlandini, C., Antonopoulos, G., Baretta, A., Beaujean, P., Bex, G. J., Biancolini, M. E., Celi, S., Costa, E., Drescher, L., Eleftheriadis, V., Fadel, N. A., Fink, A., Galbiati, F., Hatzakis, I., Hompis, G., Lewandowski, N., Memmolo, A., Mensch, C., Vignali, E. (2023). HPC+ in the medical field: Overview and current examples. Technology and Health Care, 31(4), 1509–1523. https://doi.org/10.3233/thc- 229015
[4] Lewandowski, N., & Koller, B. (2023). Transforming medical sciences with high-performance computing, high-performance data analytics and AI. Technology and Health Care, 31(4), 1505– 1507. https://doi.org/10.3233/thc-237000
[5] Wei, Y., Liu, W., Schmidt, B., Zou, Q., & Jiang, L. (2025). HPC and AI in bioinformatics. Future Generation Computer Systems, 174, 108019. https://doi.org/10.1016/j.future.2025.108019
[6] Bajpai, K. (2025). HyperFabric Interconnect (HFI): a unified, scalable communication fabric for HPC, AI, quantum, and neuromorphic workloads. Preprints.org. https://doi.org/10.20944/preprints202512 2404.v1
[7] Yang, E., & Velazquez-Villarreal, E. (2025). AI-HOPE: an AI-driven conversational agent for enhanced clinical and genomic data integration in precision medicine research. Bioinformatics, 41(7). https://doi.org/10.1093/bioinformatics/btaf359
[8] Yang, E., Waldrup, B., & Velazquez-Villarreal, E. (2025). Conversational Artificial intelligence for integrating social determinants, genomics, and clinical data in precision Medicine: Development and Implementation Study of the AI-HOPE-PM System. JMIR Bioinformatics and Biotechnology, 6, e76553. https://doi.org/10.2196/76553
[9] Pendyala, V. S., Kapadia, M., Periyapatnaroopakumar, B., Anandani, M., & Nagendran, N. (2025). A big data pipeline approach for predicting Real-Time Pandemic Hospitalization risk. Algorithms, 18(12), 730. https://doi.org/10.3390/a18120730
[10] Bontempi, D., Nuernberg, L., Pai, S., Krishnaswamy, D., Thiriveedhi, V., Hosny, A., Mak, R. H., Farahani, K., Kikinis, R., Fedorov, A., & Aerts, H. J. W. L. (2024). End-to-end reproducible AI pipelines in radiology using the cloud. Nature Communications, 15(1), 6931. https://doi.org/10.1038/s41467- 024-51202-2
[11] Bendazzoli, S., Persson, S., Astaraki, M., Pettersson, S., Grozman, V., & Moreno, R. (2025). MAIA: a collaborative medical AI platform for integrated healthcare innovation. Npj Artificial Intelligence, 1(1). https://doi.org/10.1038/s44387-025-00042-6
[12] Shakor, M. Y., & Khaleel, M. I. (2024). Recent advances in big medical image data analysis through deep learning and cloud computing. Electronics, 13(24), 4860. https://doi.org/10.3390/electronics13244860
[13] Seethala, S. C. (2020). AI-Enabled Data Pipelines: Modernizing data warehouses in healthcare for Real- Time analytics. International Research Journal of Innovations in Engineering and Technology, 04(12), 43–45. https://doi.org/10.47001/irjiet/2020.412007
[14] Khan, S. N., Danishuddin, Khan, M. W. A., Guarnera, L., & Akhtar, S. M. F. (2026). Multi-modal AI in precision medicine: integrating genomics, imaging, and EHR data for clinical insights. Frontiers in Artificial Intelligence, 8, 1743921. https://doi.org/10.3389/frai.2025.1743921
[15] Brancato, V., Esposito, G., Coppola, L., Cavaliere, C., Mirabelli, P., Scapicchio, C., Borgheresi, R., Neri, E., Salvatore, M., & Aiello, M. (2024). Standardizing digital biobanks: integrating imaging, genomic, and clinical data for precision medicine. Journal of Translational Medicine, 22(1), 136. https://doi.org/10.1186/s12967-024-04891-8
[16] Lin, P., Tsai, Y., Yeh, Y., & Shen, M. (2022). Cutting-Edge AI technologies meet precision medicine to improve cancer care. Biomolecules, 12(8), 1133. https://doi.org/10.3390/biom12081133
[17] Tiwari, A., Mishra, S., & Kuo, T. (2025). Current AI technologies in cancer diagnostics and treatment. Molecular Cancer, 24(1), 159. https://doi.org/10.1186/s12943-025-02369-9
[18] Chang, J. S., Kim, H., Baek, E. S., Choi, J. E., Lim, J. S., Kim, J. S., & Shin, S. J. (2025). Continuous multimodal data supply chain and expandable clinical decision support for oncology. Npj Digital Medicine, 8(1), 128. https://doi.org/10.1038/s41746-025-01508-2
[19] Salehi, S. S., Saadatfar, H., Oyelere, S. S., Hussain, S., Joloudari, J. H., Ledari, M. T., Arslan, E., & Barzegar, B. (2026). Enhancing healthcare outcome with scalable processing and predictive analytics via cloud healthcare API. Frontiers in Digital Health, 7, 1687131. https://doi.org/10.3389/fdgth.2025.1687131
[20] Chew, B., & Ngiam, K. Y. (2025). Artificial intelligence tool development: what clinicians need to know? BMC Medicine, 23(1), 244. https://doi.org/10.1186/s12916-025-04076-0
[21] Liu, P., & Guitart, J. (2021). Performance characterization of containerization for HPC workloads on InfiniBand clusters: an empirical study. Cluster Computing, 25(2), 847–868. https://doi.org/10.1007/s10586-021-03460-8
[22] Namli, T., Sınacı, A. A., Gönül, S., Herguido, C. R., Garcia-Canadilla, P., Muñoz, A. M., Esteve, A. V., & Ertürkmen, G. B. L. (2024). A scalable and transparent data pipeline for AI-enabled health data ecosystems. Frontiers in Medicine, 11, 1393123. https://doi.org/10.3389/fmed.2024.1393123
