A new study published last month has put forward a new framework intended specifically for assessing evidence from digital health interventions (DHIs).
The research entitled ‘rigorous and rapid evidence assessment in digital health with the evidence DEFINED framework’, was published by npj Digital Medicine, proposing a framework to assess Evidence in Digital health for Effectiveness of Interventions with Evaluative Depth (Evidence DEFINED).
Noting “limitations of existing frameworks for digital health-specific interventions”, the authors put forward an alternative which is based around evidence considerations which are unique to digital health interventions. They intend for the development of this kind of framework to promote the adoption of digital health interventions by allowing rigorous assessment of evidence in a more standardised manner.
“The primary goals of Evidence DEFINED are to a) facilitate standardised, rapid, rigorous DHI evidence assessment in organisations and b) guide digital health solutions providers who wish to generate evidence that drives DHI adoption.”
Although the article stopped short of a comprehensive scoping review, it did present a preliminary catalogue of existing frameworks. The paper added that: “Seventy-eight prior frameworks were identified. Some of these may be useful, though many prior frameworks are underdeveloped in the key domain of clinical outcomes assessment. Prior DHI evidence assessment frameworks are typically sections of broader DHI assessment guides, often containing just a few, superficial questions, with minimal evaluation of evidence quality or bias.”
The authors also noted a gap in “evaluation methodologies that were developed for non-digital interventions”. The Evidence DEFINED framework incorporates established evidence assessment methodologies, rather than attempting to “reinvent the wheel”.
Specifically, Evidence DEFINED offers a supplementary checklist of evidence quality criteria which are recommended for DHI evidence assessments. This includes evidence quality criteria that are unique to digital health, “as well as evidence quality considerations that may require enhanced vigilance for assessment of digital health interventions”.
The authors noted that a DHI assessment should address a number of factors in assessing clinical evidence, including patient experience, product design, and code effectiveness. Data governance and digital health equity are two other important domains, ensuring that care is taken over patient data, and that digital literacy is at a level to support DHI assessment.
The framework recognises the need for rapid assessment, first using screening items to determine whether a DHI meets a series of absolute requirements, before moving on to a streamlined approach briefly outlined below.
Step one of the framework is screening for failure to meet absolute requirements; step two is applying an established evidence assessment framework that was developed for non-digital interventions; step three is applying the Evidence DEFINED supplementary checklist to address evidence quality considerations unique to digital health interventions; and step four is making actionable and defensible recommendations around levels of appropriate adoption.
Whilst Evidence DEFINED may facilitate initial assessments of DHIs, the authors note that more work will have to be done to identify best practices for “monitoring post-trial DHI modifications (e.g., due to software updates), as well as any changes in safety or effectiveness, throughout the product lifecycle”.
The authors finish with a note on how they hope Evidence DEFINED will help to advance DHIs and promote their adoption. “To realise the potential of digital health, we need stronger, standardisd frameworks for DHI evidence assessment. We should encourage DH solutions providers to follow high standards—and hold DHSPs accountable to deliver the clinical value they promise. Evidence DEFINED may help guide DHSPs that wish to develop compelling evidence and drive adoption of digital health products. We hope this will promote evidence-based decision making, encourage adoption of effective DHIs, and thereby improve health outcomes across a range of conditions and populations.”
To view the paper, please click here.
Silberman, J., Wicks, P., Patel, S. et al. Rigorous and rapid evidence assessment in digital health with the evidence DEFINED framework. npj Digit. Med. 6, 101 (2023). https://doi.org/10.1038/s41746-023-00836-5
