The National Institute for Health and and Care Excellence (NICE) has updated their evidence standards framework (ESF), working with stakeholders, system partners and thought leaders to ensure that digital health technologies (DHTs) are “clinically effective and offer value to the health and care system”.
Originally published in December 2018, the August 2022 update includes evidence requirements on artificial intelligence (AI) and data-driven technologies with adaptive algorithms. NICE also notes that they have “aligned it with regulatory requirements and made it easier to use”, and “outlined a subset of early deployment standards that can be used within evidence generation programmes”.
The evidence standards framework (ESF) can be used by evaluators and decision-makers to identity the DHTs that are likely to benefit users and the health and care system.
The framework is split into four key sections. Section A focuses on technologies suitable for evaluation using the evidence standards framework; section B looks at the classification of digital health technologies; section C provides evidence standards tables; and Section D includes early deployment standards for evidence-generation programmes.
Here, we will take a look at some of the updated content of the framework, focusing on the role of AI.
Included in the evidence standards tables is the need to “consider health and care inequalities and bias mitigation”. Under this, the updated framework notes: “For data-driven DHTs (including those with artificial intelligence), the company should describe any actions taken in the design of the DHT to mitigate against algorithmic bias that could lead to unequal impacts between different groups of service users or people.”
Another listed standard emphasises the need to “show real-world evidence that the claimed benefits can be realised in practice”. The framework notes that there needs to be evidence that the DHT was acceptable to users, performed at its intended purpose to the expected level, successfully integrated into the current service provision, caused no unintended negative impacts, showed improvements in outcomes and was used in line with expectation. Here, an update to the framework notes: “For DHTs whose performance may be affected by local deployment factors (such as DHTs using artificial intelligence), this may include deploying the DHT to run offline or evaluating it ‘in silent mode’.”
Elsewhere, the framework states: “The company and evaluator should agree a plan for measuring usage and changes in the DHT’s performance over time”, which includes agreeing to a plan for ongoing data collection. It adds: “For DHTs whose performance is expected to change over time (such as DHTs that use artificial intelligence or machine-learning algorithms, or DHTs that are expected to be updated in subsequent versions), the company and evaluator should agree on post-deployment reporting of changes in performance.”
This could include:
- future plans for updating the DHT, including how regularly the algorithms need to retrain, re-version or change functionality
- the sources of retraining data and how quality will be assessed
- processes in place for measuring performance over time, to detect impacts of planned changes or environmental factors that could affect performance
- processes in place to detect decreasing performance in certain groups of people over time
- whether there is an independent overview process for reviewing changes in performance
- an agreement on how and when changes in performance should be reported and to whom
“If the intended purpose of the DHT changes,” the framework notes, “or if additional functions are added that change the intended purpose and ESF classification of the DHT, then a new evaluation should be done.”
To access the updated framework in full, please click here.
