Healthcare IT teams are quietly experimenting with AI-driven software development — using artificial intelligence to write code — often without executive leadership knowing about it. The practice raises specific security concerns for hospitals and clinics, where patient information and regulatory compliance are at stake.
AI-driven software development refers to generating applications through natural language prompts to AI tools. While it can speed up development, it also introduces risks that health IT managers need to address.
What’s the scope for mistakes when AI writes code?
AI coding tools can produce applications based on user direction, but the process demands a deep understanding of network security, data privacy standards, and compliance requirements. All of that context must be fed into the AI before it starts working on a problem.
These tools may appear to read minds and move quickly. But if users don’t set the stage properly, they risk creating something with inconsistent behavior. That inconsistency can then impact data security and regulatory compliance.
Related: NIH Unifies Seven Decades of Health Data
The scope for mistakes is wider than many managers assume. AI-generated code can look correct while containing subtle flaws that only surface under specific conditions — the kind of bugs that auditors and security scanners find later.
Nontechnical users are building apps — and that’s a problem
Programming remains a technical skill. Just because a nurse or administrator can generate a working application using AI doesn’t mean that app is ready for wider use. If someone builds a discrete tool to help their clinic, that’s fine for personal use — but it can’t be shared or rolled out without a programmer reviewing it first.
This concern mirrors the shadow IT issues that have plagued healthcare organizations for years. The difference is that AI-driven software development makes it even easier for nontechnical staff to create software that looks production-ready but isn’t.
AI tools expand the attack surface
Increasing the attack surface is inevitable when AI writes code. These solutions build on pre-existing open-source libraries and tools, which means your software bill of materials will lengthen considerably. That creates technical debt and dependencies on packages your team may not know well.
This expansion is almost unavoidable when AI-written software moves into production. Organizations must document every new dependency for compliance and risk management purposes. Without that documentation, auditors will have a hard time tracing what’s actually running in the environment.
Related: Healthcare innovation needs strong data base
Legacy systems don’t play well with AI-generated code
AI coding tools assume a modern development environment. They’re built on large language models trained on current code and standards, which creates friction when interacting with legacy systems like old HL7 electronic health record platforms or aging databases.
To make the best use of AI tools, organizations need to adopt modern DevSecOps practices. Going full continuous integration and deployment isn’t necessary, but teams should be able to rapidly deploy fixes for both functional and security flaws discovered downstream.
The gap between modern AI tools and legacy healthcare infrastructure is wider than many vendors acknowledge. A tool trained on contemporary Python libraries might generate code that passes general security checks but misses requirements specific to health information handling.
Writing consistent security instructions is essential
Writing a consistent set of security instructions for every AI-driven software development session is a necessary risk-reduction measure. Without that, the AI might generate code that passes general security checks but misses requirements specific to healthcare data handling.