Health IT End-Users Alliance

Real-world Testing Consensus Statement


The current standards and health IT policy development process does not include a
robust real-world testing process before standards are solidified and mandated
through policy. There are opportunities for health IT end-users who use technology
tools for care, as well as for patients to guide the development of policies and
standards that meet real-world needs and reduce burden.


The development of new health IT standards and their adoption into regulatory
requirements can bring benefits by addressing specific challenges and creating
more uniformity in how health information is gathered, shared, and used. However,
over the past decade, end-users have found that new technical approaches and
regulatory requirements are not sufficiently grounded in real-world experiences and
do not adequately consider the implementation pathway before mandating use.
This includes issues such as how new approaches work with the existing
infrastructure that is deployed, workflow constraints to adopting new technology,
technology costs, engaging with and educating patients on their role in utilizing the
technology, and how new requirements will fit into the array of regulatory
requirements that health IT end-users face.

As a consequence, end-users have experienced:

  • Adoption of standards that require significant workarounds by health care
    organizations to implement
  • The set of standards are incomplete or too immature to accomplish the desired
  • Policy mandate deadlines changed multiple times leading to confusion and poor
    resource allocation
  • Standards and policies that do not achieve their desired goals when deployed
  • Excessive burden added to health IT end-users
  • Wasted money on failed implementations
  • Confusion from patients with respect to technological capabilities

Opportunities and Challenges:

Just as we have seen the inclusion of user-centered design in the development of
health IT applications, so should we in the development of health standards and
policies. Leveraging health IT end-user input throughout the standards and policy
development process and conducting robust real-world testing can help mitigate
those potential issues. It also can better align standards and policies to support the
needs of front-line health workers and result in faster adoption and realization of
the goals for the collection, sharing and use of health information to care for
patients and support individuals on their health journeys.

Front-line health workers are busy and do not have extensive backgrounds in
standards and policy development. Providers working in small, solo, or rural
medical offices experience these issues to an even greater degree. There will be
challenges in supporting front-line workers to provide the needed input throughout
the standards and policy development process.

Since the passage of HITECH, the federal government has played an increasingly
active role in advancing health IT standards and mandating their use. By adopting
specific standards, creating forums to identify needed standards, and funding
standards development, the federal government is setting the path for standards
adoption. The results of rigorous real-world testing that considers the impact on a
range of end-users and is conducted before standards are included in policy should
be part of the determination of whether a standard is mature.


The following principles will enhance the likelihood that beneficial new standards
are adopted across the spectrum of health care providers.

  • Health IT end-users must be instrumental in establishing the goals and
    priorities for setting standards, across care settings and use cases, such as
    public health.
  • Health IT end-user input should be included in every phase of standards
    development from initial standard scoping to real-world testing.
  • Standards and policies to advance standards adoption should prioritize patient
    care and wellness and public health; secondary or supplemental requirements
    (e.g., reporting, quality measurement, compliance) should be derived from
    patient care.
  • Real-world testing should inform/investigate whether the standard and/or
    policy will:

    • Be implementable by health care organizations without significant effort
      beyond the value incurred by adoption
    • Be effective at achieving its desired goal
    • Encompass a complete solution to achieve the desired goal
    • Not result in unintended consequences that would harm individuals
      (caregivers, patients, physicians and other clinicians)
    • Respect and accommodate the privacy needs of individual patients
    • Not add extraneous work to the care team
    • Ensure sufficient return on investment to justify the health IT spend
    • Disparately impact providers who care for communities that are underserved
      or marginalize
  • Health IT end-user engagement should be supported to maximize the end user’s ability to provide effective input
    • Standards development organizations should assign resources to this purpose
    • Clinical and operational informaticists can be called on to participate, while
      professional membership associations can encourage participation and serve
      as a liaison
    • Providers caring for the underserved should be included and provided
      resources to participate otherwise they will be excluded.
    • The federal government should provide financial and technical support to
      ensure that end-user testing is a routine part of standards development and
      that testing methods are sufficiently rigorous and based on real-world needs
      of clinicians and patients. Federal support should also ensure participation
      by those who care for communities that are underserved or marginalized.
    • Payers and health IT vendors should also support testing, while avoiding
      sponsorship models
  • Health IT end-users, or their professional organizations, should commit to being
    active participants in the standards development activities of several standards
    development organizations and data content committees.
  • Standards development organizations must take health IT end-user input
    seriously and not dismiss it as “simply out of scope” or a “matter of policy”
    inappropriate to address through standards development. Granularity in
    standards implementation guides is increasingly necessary to ensure uniform
    development across vendors, interoperability, and achievement of the
    technology’s end goals.
  • The testing of standards must include real-world implementations, production
    pilots, and collection of metrics regarding the effort needed to implement, the
    training needs for staff, the extent to which the standard achieved the stated
    goal and estimates of the costs and benefits of implementation. This includes,
    for example:

    • Scoping
    • Conceptual development
    • Use cases
    • Standards testing
    • Pilot testing
    • Return on investment
    • Assessment of impact on different provider types and with different
    • On-going monitoring
  • The health IT community should work together to identify expectations for
    rigorous real-world testing, such as the needed metrics, methods of
    accountability, assurance that testing results are impartial, external expert
    review of testing methods and results, impact on health equity, and public
    reporting of the outcome.
  • Standards should not be considered mature until real-world testing has been
    completed and comprehensive report-outs on the testing are made public.
    Maturity models should not consider inclusion in regulation as a sign of
  • Once in use, standards should be monitored to understand usability and other features.
  • Policy proposals should build from the results of real-world testing and consider the implementation path for a range of provider settings, from small offices to large systems and across medical specialties and including end-users caring for
    diverse populations and communities with the highest needs.
  • Standards that have not completed robust real-world testing are not suitable for mandated use in health policy. However, the federal government could
    support real-world testing of standards that support policy goals. This should be
    done in close coordination with end users and in a way that ensures than end users serving diverse populations and communities with the highest needs are included.
  • Standards and health policy must ensure equity and embrace diversity,
    including end-user involvement, conducting real-world testing, and the creation
    of resources for standards development