Yes nice. I wonder if we can have a chat with someone from SAS or other big company to get more details. If we can follow and reference some of this we have a strong point since these tools are accepted by regulators…
Interestingly, AETION appears to have a very different take on validity. They do not mention the more software-engineering oriented approach to validity, but instead focus solely on validity as the ability to reproduce known study results. They show they have reproduced existing observational studies (not sure why those should be considered a gold standard), and claim to have ‘predicted’ the result of an ongoing RCT.
I understand why they stress this aspect of validity, and it also seems to argue we definitely should keep the results of our method evaluation against the Method Benchmark in our document. We probably even want to extend it to PLP.
Hopefully this is contributing to this conversation about software validity. NCQA has a software certification process for HEDIS quality measures. The certification process is detailed here http://www.ncqa.org/hedis-quality-measurement/data-reporting-services/quality-measure-certification this is an approach I have seen
I am aware that the VA has not yet approved OHDSI tools including Atlas and the major analytics packages because they are open source. It might be good to investigate what criteria are part of their approval decision, unless these are already well known.
there is a document that was issued by FDA just recently on the use of RWE to support regulatory decisions that is worth reading. It does cover the “reliability” and quality of the RWD as well.
Does anyone know what, if anything, this document implies regarding FDA oversight/approval of predictive algorithms used in medical devices?
Here is a summary of what I’ve discovered re new FDA regulations for software. Some of this might be useful to update or reference in the OHDSI Software Validity and Regulatory Requirements document.
The FDA’s Center for Devices and Radiological Health has a new program focusing on Digital Health that covers software that it classifies as a medical device: https://www.fda.gov/medicaldevices/digitalhealth/
As of summer 2018 it has issued some guidance on software that is particularly relevant to PLP @Rijnbeek and @jreps. One was developed in collaboration with the Software as a Medical Device Working Group of the International Medical Device Regulators Forum. So it is relevant to OHDSI members outside the US.
Here is a link to that one: https://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM524904.pdf
I assume this will replace or supersede the 2002 General Principles of Software Validation document reference in @schuemie 's splendid document which covers some of the same territory.
There is also a draft guidance document that is US-only and focused on clinical decision support software:
https://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM587819.pdf
This Guidance on clinical decision support software defines what an algorithm implemented as software is classified as a device and is therefore subject to oversight, evidence requirements for approval, etc.
The criteria for classification of an algorithm as a device hinges on whether it processes information that affects medical decisions in ways the clinician could not do without the algorithm. For example, software that delivers reminders to do something according to a practice guideline don’t meet the definition. Software that takes data and processes it to predict something relevant to a patient’s care do meet the definition. But that’s just my gloss. People should read for themselves rather than relying on my take.
The Software as a Medical Device document also includes sections on clinical and technical validation that might be useful to reflect or mention somewhere in the OHDSI software document.
Thanks @Andrew! I will thoroughly revise our document before the symposium (hopefully much sooner than that), and will incorporate these new developments you’ve highlighted.
The idea is to have a ‘formal’ first version of the document by the symposium.
Before we stray too far from the path: Apart from this not being a regulation but just a guideline, and it not even being issued by the FDA but IMDRF, it pertains to Software as a Medical Device. And that is defined as software that does one of the following:
- diagnosis, prevention, monitoring, treatment or alleviation of disease,
- diagnosis, monitoring, treatment, alleviation of or compensation for an injury,
- investigation, replacement, modification, or support of the anatomy or of a physiological process,
- supporting or sustaining life,
- control of conception.
So, patient care. We do none of that. At least not now. When we start pushing our stuff out to individual patients or providers handling individual patients, we may be getting closer to what the intention of these guidelines is. Of course, it’s always good to get good ideas and hints about how to do software validation right, if that’s what you had in mind.
Thanks Andrew,
In the EHDEN project I have created a task related to the regulatory requirements for PLP since this is indeed getting in the area of medical devices with the new upcoming regulations. This will also happen in Europe. Thanks for sharing and we will come back to this later once the EHDEN project is started.
Peter