This post is a follow up to a comment I made to @schuemie during the excellent Population Level Estimation tutorial at this year's symposium. I suggested adding questions about evidence gaps and clinical/policy usefulness to the Atlas page for specifying pop level effect estimation analyses.
Here's a stab at describing what the point would be and some draft questions that might do that. Please comment on and improve this if it seems like it might be a useful addition to an already outstanding tool. Thanks.
Goal: Define a set of questions to be added to the Atlas Population Level Effect Estimation page. The questions would be designed to promote the value of evidence generated via the Cohort Method library. Value would be defined as usefulness to decision makers (clinical or policy relevance) and reduction in existing uncertainty about the exposures (address an evidence gap). Answers to the questions could be captured and used as more boilerplate in study protocols and publication sections.
There are several good reasons to compare exposures' effect on an outcome that the cohort method library supports. In order to be useful, the questions should be general enough to apply to most or all of these use cases. This is a starter list. I'm sure there are others.
Good reasons to compare exposures' effects on outcomes
- To reduce uncertainty about which exposure ...
- is more effective as a treatment, service, diagnostic tool, etc.
- is less harmful
- has a greater net benefit given the balance of benefits and harms
- is more cost-effective given the balance of benefits, harms, and costs
- is more acceptable to patients or providers, or is adopted more widely
- To replicate a prior finding that one exposure is superior.
- or replicate prior results in a new population
Again, the questions to be added to Atlas should help to ensure that for any of these analytic use cases ...
- prior research leaves substantial uncertainty about the relationship between exposures and outcome (results address an evidence gap)
- the results will be useful to people who need to make a specific health decision (the target, comparator and outcome are clinically appropriate)
These draft questions are inspired by the guidance for research question formation from the PCORI methodology standards
- Is there a current systematic review that supports the need for this comparison? (Yes/No/Unsure)
- If Yes: What is the citation for the review? (Text box)
- If Yes: What reasons does the review give in support of the need for this comparison? (Text box)
- If No: Is there published prior research that supports the need for this comparison? (Yes/No)
- If Yes: What are the citations for this research? (Text box)
- If Yes: What reasons does this research give in support of the need for this comparison? (Text box)
- If No: The evidence produced by this analysis will be challenging to relate to an important evidence gap and this may limit its value. (Message)
- If Unsure: The results of your analysis are likely to have a greater impact if they are motivated by an important gap in the existing research. Consulting a current systematic review can be an efficient way to ensure that your results have the potential to help address an important evidence gap. (Message)
- What is the specific health decision the results are intended to inform? (Text box)
- Who faces this decision? (Checkbox: Patients, providers, care givers, policy makers)
- How will the results inform the health decision? (Text box)
- Are both the target and the comparator exposures clinically appropriate choices? E.g. are they both first-line treatments for the same patient population? (Yes/No)
- If No: What is the rationale for considering inappropriate choices? (Text box)
- Do the target and comparator correspond to the actual healthcare options for patients, providers, caregivers or policy makers who would face the healthcare decision? (Yes/No)
- If No: What is the rationale for comparing options that decision makers do not face? (Text box)
- Is the outcome one that the population of interest notices and cares about? (Yes/No)
- If No: What is the rationale for using an outcome that is unnoticed or unimportant to the decision makers? (Text box)