Health IT End-Users Alliance
Real-world Testing Consensus Statement
The current standards and health IT policy development process does not include a
robust real-world testing process before standards are solidified and mandated
through policy. There are opportunities for health IT end-users who use technology
tools for care, as well as for patients to guide the development of policies and
standards that meet real-world needs and reduce burden.
The development of new health IT standards and their adoption into regulatory
requirements can bring benefits by addressing specific challenges and creating
more uniformity in how health information is gathered, shared, and used. However,
over the past decade, end-users have found that new technical approaches and
regulatory requirements are not sufficiently grounded in real-world experiences and
do not adequately consider the implementation pathway before mandating use.
This includes issues such as how new approaches work with the existing
infrastructure that is deployed, workflow constraints to adopting new technology,
technology costs, engaging with and educating patients on their role in utilizing the
technology, and how new requirements will fit into the array of regulatory
requirements that health IT end-users face.
Opportunities and Challenges:
Just as we have seen the inclusion of user-centered design in the development of
health IT applications, so should we in the development of health standards and
policies. Leveraging health IT end-user input throughout the standards and policy
development process and conducting robust real-world testing can help mitigate
those potential issues. It also can better align standards and policies to support the
needs of front-line health workers and result in faster adoption and realization of
the goals for the collection, sharing and use of health information to care for
patients and support individuals on their health journeys.
Front-line health workers are busy and do not have extensive backgrounds in
standards and policy development. Providers working in small, solo, or rural
medical offices experience these issues to an even greater degree. There will be
challenges in supporting front-line workers to provide the needed input throughout
the standards and policy development process.
Since the passage of HITECH, the federal government has played an increasingly
active role in advancing health IT standards and mandating their use. By adopting
specific standards, creating forums to identify needed standards, and funding
standards development, the federal government is setting the path for standards
adoption. The results of rigorous real-world testing that considers the impact on a
range of end-users and is conducted before standards are included in policy should
be part of the determination of whether a standard is mature.
The following principles will enhance the likelihood that beneficial new standards
are adopted across the spectrum of health care providers.
- Health IT end-users must be instrumental in establishing the goals and
priorities for setting standards, across care settings and use cases, such as
- Health IT end-user input should be included in every phase of standards
development from initial standard scoping to real-world testing.
- Standards and policies to advance standards adoption should prioritize patient
care and wellness and public health; secondary or supplemental requirements
(e.g., reporting, quality measurement, compliance) should be derived from
- Real-world testing should inform/investigate whether the standard and/or
- Be implementable by health care organizations without significant effort
beyond the value incurred by adoption
- Be effective at achieving its desired goal
- Encompass a complete solution to achieve the desired goal
- Not result in unintended consequences that would harm individuals
(caregivers, patients, physicians and other clinicians)
- Respect and accommodate the privacy needs of individual patients
- Not add extraneous work to the care team
- Ensure sufficient return on investment to justify the health IT spend
- Disparately impact providers who care for communities that are underserved
- Health IT end-user engagement should be supported to maximize the end user’s ability to provide effective input
- Standards development organizations should assign resources to this purpose
- Clinical and operational informaticists can be called on to participate, while
professional membership associations can encourage participation and serve
as a liaison
- Providers caring for the underserved should be included and provided
resources to participate otherwise they will be excluded.
- The federal government should provide financial and technical support to
ensure that end-user testing is a routine part of standards development and
that testing methods are sufficiently rigorous and based on real-world needs
of clinicians and patients. Federal support should also ensure participation
by those who care for communities that are underserved or marginalized.
- Payers and health IT vendors should also support testing, while avoiding
- Health IT end-users, or their professional organizations, should commit to being
active participants in the standards development activities of several standards
development organizations and data content committees.
- Standards development organizations must take health IT end-user input
seriously and not dismiss it as “simply out of scope” or a “matter of policy”
inappropriate to address through standards development. Granularity in
standards implementation guides is increasingly necessary to ensure uniform
development across vendors, interoperability, and achievement of the
technology’s end goals.
- The testing of standards must include real-world implementations, production
pilots, and collection of metrics regarding the effort needed to implement, the
training needs for staff, the extent to which the standard achieved the stated
goal and estimates of the costs and benefits of implementation. This includes,
- Conceptual development
- Use cases
- Standards testing
- Pilot testing
- Return on investment
- Assessment of impact on different provider types and with different
- On-going monitoring
- The health IT community should work together to identify expectations for
rigorous real-world testing, such as the needed metrics, methods of
accountability, assurance that testing results are impartial, external expert
review of testing methods and results, impact on health equity, and public
reporting of the outcome.
- Standards should not be considered mature until real-world testing has been
completed and comprehensive report-outs on the testing are made public.
Maturity models should not consider inclusion in regulation as a sign of
- Once in use, standards should be monitored to understand usability and other features.
- Policy proposals should build from the results of real-world testing and consider the implementation path for a range of provider settings, from small offices to large systems and across medical specialties and including end-users caring for
diverse populations and communities with the highest needs.
- Standards that have not completed robust real-world testing are not suitable for mandated use in health policy. However, the federal government could
support real-world testing of standards that support policy goals. This should be
done in close coordination with end users and in a way that ensures than end users serving diverse populations and communities with the highest needs are included.
- Standards and health policy must ensure equity and embrace diversity,
including end-user involvement, conducting real-world testing, and the creation
of resources for standards development