In my work across data engineering, software systems, healthcare technology, and risk and controls, I have learned that a system is only as trustworthy as the information moving through it. This principle is especially important in healthcare artificial intelligence, where a flawed data pipeline can affect not only a dashboard or business process, but also a decision about a patient's care.

Predictive AI integrated with electronic health records is no longer experimental. In 2024, 71 percent of U.S. non-federal acute care hospitals reported using predictive AI integrated with their Electronic Health Records. (EHRs) up from 66 percent in 2023.[1] The reliability of AI depends on the quality, meaning, security, and traceability of the data.

A record can contain every required field and still be clinically misleading. Also, a billing diagnosis may not represent the full clinical picture. A laboratory value may use the wrong unit. A timestamp may record when data entered into a system rather than when the clinical event occurred.  One widely used population health algorithm used healthcare cost as a proxy for healthcare needs. The algorithm underestimated Black patients' needs, since historical spending on them had been lower than on equally sick White patients.[2]

Interoperability presents a related challenge. The Centre for Medicare & Medicaid Services interoperability and prior authorization final rule requires impacted payers to implement or enhance several health level seven fast healthcare interoperability resources (FHIR) with compliance dates for the application programming interface requirements generally beginning in 2027.[3] FHIR improves how systems represent and exchange information, but it cannot correct poor source data or reconcile conflicting definitions.[4] Two systems can exchange a value successfully and still interpret it differently. Healthcare AI needs both syntactic interoperability, which supports exchange, and semantic interoperability, which preserves meaning.

Models must also be tested where they will be used. Pneumonia detection models trained on chest radiographs from one hospital system often performed worse on external hospital data.[5] Differences in patient populations, disease prevalence, equipment, workflows, and coding practices can produce dataset shift.[6] The FDA's work on post market monitoring of AI-enabled medical devices reflects this risk by focusing on changes in model inputs, outputs, and real world performance.[7] Deployment is not the end of validation. It is the beginning of continuous operational evidence.

For healthcare organizations, trustworthy AI requires five connected capabilities: clear data contracts; patient identity and event reconciliation; lineage and provenance; automated quality and drift monitoring; and purpose-based access controls. NIST's Generative AI Profile, although not healthcare-specific, reinforces related practices such as assessing training-data risks, documenting sources, maintaining provenance, and monitoring systems over time.[8]

Evaluation must extend beyond accuracy. It should examine calibration, predictive value, performance across relevant patient groups, external and temporal validation, and effects on clinical workflow.[9] In 2024, 79 percent of hospitals using predictive AI reported some post-implementation evaluation, but fewer evaluated all or most models, and 18 percent did not know whether monitoring occurred.[1] Adoption is moving faster than consistent oversight.

The future of healthcare AI is therefore an infrastructure question as much as an algorithmic one. Interoperability without quality can spread errors faster. Quality without interoperability can create accurate but isolated systems. AI without monitoring can drift silently away from the people it is intended to serve. A healthcare AI system is only as trustworthy as the data engineering that feeds it and the controls that watch it.

References

[1] Chang, W., Owusu-Mensah, P., Everson, J., & Richwine, C. (2025). Hospital Trends in the Use, Evaluation, and Governance of Predictive AI, 2023-2024. Office of the Assistant Secretary for Technology Policy, Data Brief No. 80. https://healthit.gov/data/data-briefs/hospital-trends-use-evaluation-and-governance-predictive-ai-2023-2024/

[2] Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447-453. https://doi.org/10.1126/science.aax2342

[3] Centers for Medicare & Medicaid Services. (2024, January 17). CMS Interoperability and Prior Authorization Final Rule CMS-0057-F. https://www.cms.gov/newsroom/fact-sheets/cms-interoperability-prior-authorization-final-rule-cms-0057-f

[4] Lehne, M., Sass, J., Essenwanger, A., Schepers, J., & Thun, S. (2019). Why digital medicine depends on interoperability. npj Digital Medicine, 2, 79. https://doi.org/10.1038/s41746-019-0158-1

[5] Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018). Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLOS Medicine, 15(11), e1002683. https://doi.org/10.1371/journal.pmed.1002683

[6] Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., & Saria, S. (2021). The clinician and dataset shift in artificial intelligence. New England Journal of Medicine, 385(3), 283-286. https://doi.org/10.1056/NEJMc2104626

[7] U.S. Food and Drug Administration. (n.d.). Methods and Tools for Effective Postmarket Monitoring of Artificial Intelligence-Enabled Medical Devices. Accessed August 24, 2026. https://www.fda.gov/medical-devices/medical-device-regulatory-science-research-programs-conducted-osel/methods-and-tools-effective-postmarket-monitoring-artificial-intelligence-ai-enabled-medical-devices

[8] National Institute of Standards and Technology. (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). https://doi.org/10.6028/NIST.AI.600-1

[9] Wiens, J., Saria, S., Sendak, M., et al. (2019). Do no harm: A roadmap for responsible machine learning for health care. Nature Medicine, 25, 1337-1340. https://doi.org/10.1038/s41591-019-0548-6