15 Sep HIT End-Users Alliance Responds to NIST AI Standards “Zero Drafts” Pilot Project: TEVV Outline: Invitation for Input
Craig Burkhardt
Acting Director
National Institute of Standards and Technology
100 Bureau Drive
Gaithersburg, MD 20899
Submitted electronically to ai-standards@nist.gov
RE: NIST AI Standards “Zero Drafts” Pilot Project: TEVV Outline: Invitation for Input
Dear Acting Director Burkhardt:
The Health IT End-Users Alliance (the Alliance) appreciates the opportunity to provide input on the National Institute of Standards and Technology (NIST) AI Standards “Zero Drafts” Pilot Project: testing, evaluation, verification, and validation (TEVV) outline.
The Alliance brings together health information professionals, physicians, hospitals, and other front-line healthcare providers and organizations that use health information technology (IT) to ensure that policy and standards development activities reflect the complex web of clinical and operational challenges facing those who use such technologies today. By working collaboratively, the Alliance is focused on priorities for how technology can best support clinical care and operations.
The Alliance was pleased to provide feedback to NIST on the initial topics and scopes for the development of AI standards. The end-user community is well-equipped and has the expertise needed to ensure appropriate design, development, implementation, and post-market surveillance of AI tools, and TEVV methods should include methods to gather feedback from end-users on their experiences. Ensuring there are avenues for end-users to provide feedback will ensure TEVV assessments are well-rounded, accurate, and reflective of an AI tool’s true performance and the impact to patient care and administrative workflows.
As previously shared, the Alliance published a consensus statement in April 2025 reflecting on the current state of AI in healthcare including principles intended to guide policymakers as they work to ensure appropriate oversight of AI tools without hampering innovation. We continue to urge NIST to prioritize end-users and the recommendations put forward within the consensus statement as NIST and other agencies contemplate how to ensure proper oversight of AI while fostering innovation. Having a diverse set of stakeholders, including end-users, at the table will assist NIST in guiding the development of standards that accomplish the AI policy goals of the Trump Administration.
Terminology, Key Terms, and Concepts (Clauses 3 and 4). In defining terminology, end-users must be present to assist in determining comprehensive definitions of the components that are involved with and influence TEVV. Those directly involved with the implementation, use, and monitoring of AI, such as end-users, are best positioned to assist NIST in identifying the components of how AI should be evaluated and what specific considerations should be included in appropriate TEVV. End-users can also assist in identifying what adequate validity and reliability looks like and raise considerations of variances across patient populations to ensure terminology, and in turn TEVV methods, are comprehensive and appropriate for the healthcare settings in which AI tools are used. End-users can provide feedback on how TEVV methods may need to be tailored to specific healthcare organizations and patient populations, as well as the potential limitations of TEVV and how to account for those. In Table 1, end-users can help amplify the current understanding associated with each term and build out examples allowing the eventual development of the standards draft to be comprehensive and reflective of how this information is applicable to the various capabilities of different healthcare organizations. It is also important to ensure end-user perspectives and expertise are incorporated in the TEVV methods in Figure 1, as the different considerations within these methods will vary across both AI tools and healthcare organizations.
Governance (Clause 5). End-user input throughout the development of governance policies and frameworks and monitoring of these policies is paramount. It is critical to ensure that different types of end-users in a healthcare organization and their workstreams are reflected, as organizations have varying resource levels. These differences in resource levels result in a wide range of diverse approaches that organizations use to implement governance strategies, manage personnel, determine internal workstreams, and make resource decisions. We appreciate NIST acknowledging these concerns and other limitations involved with governance and assessment of AI in the outline. As indicated in Figure 2, regulatory and legal requirements and rules can both be drivers and constraints on the use of AI and TEVV. End-users best know the important intersections between quality, safety, reporting, and payment requirements and regulations, and the associated workstreams. By leveraging the roles of end-users and bringing their expertise forward, NIST can ensure healthcare organizations are well informed of how to leverage resources and manage limitations, and the best ways to incorporate TEVV activities into existing workstreams and/or create new workflows in a meaningful, yet minimally burdensome manner.
The Health IT End-Users Alliance thanks NIST for the opportunity to provide input on the TEVV outline and throughout the standards development process. We are committed to being a partner in this effort to create standards that meet the end-user community’s needs and ensure AI is developed, implemented, and used in optimal ways. The Alliance looks forward to hosting NIST for a virtual listening session to expand on our AI principles and recommendations and provide more detailed feedback and substantive comments on the TEVV outline and draft from our member organizations.