OHDSI Home | Forums | Wiki | Github

Strategy for promoting the value of evidence generated from population level effect estimations

Researchers
#1

This post is a follow up to a comment I made to @schuemie during the excellent Population Level Estimation tutorial at this year’s symposium. I suggested adding questions about evidence gaps and clinical/policy usefulness to the Atlas page for specifying pop level effect estimation analyses.

Here’s a stab at describing what the point would be and some draft questions that might do that. Please comment on and improve this if it seems like it might be a useful addition to an already outstanding tool. Thanks.

Goal: Define a set of questions to be added to the Atlas Population Level Effect Estimation page. The questions would be designed to promote the value of evidence generated via the Cohort Method library. Value would be defined as usefulness to decision makers (clinical or policy relevance) and reduction in existing uncertainty about the exposures (address an evidence gap). Answers to the questions could be captured and used as more boilerplate in study protocols and publication sections.

There are several good reasons to compare exposures’ effect on an outcome that the cohort method library supports. In order to be useful, the questions should be general enough to apply to most or all of these use cases. This is a starter list. I’m sure there are others.

Good reasons to compare exposures’ effects on outcomes

  • To reduce uncertainty about which exposure …
  • is more effective as a treatment, service, diagnostic tool, etc.
  • is less harmful
  • has a greater net benefit given the balance of benefits and harms
  • is more cost-effective given the balance of benefits, harms, and costs
  • is more acceptable to patients or providers, or is adopted more widely
  • To replicate a prior finding that one exposure is superior.
  • or replicate prior results in a new population

Again, the questions to be added to Atlas should help to ensure that for any of these analytic use cases …

  • prior research leaves substantial uncertainty about the relationship between exposures and outcome (results address an evidence gap)
  • the results will be useful to people who need to make a specific health decision (the target, comparator and outcome are clinically appropriate)

These draft questions are inspired by the guidance for research question formation from the PCORI methodology standards

Evidence gap

  • Is there a current systematic review that supports the need for this comparison? (Yes/No/Unsure)
  • If Yes: What is the citation for the review? (Text box)
  • If Yes: What reasons does the review give in support of the need for this comparison? (Text box)
  • If No: Is there published prior research that supports the need for this comparison? (Yes/No)
    • If Yes: What are the citations for this research? (Text box)
    • If Yes: What reasons does this research give in support of the need for this comparison? (Text box)
    • If No: The evidence produced by this analysis will be challenging to relate to an important evidence gap and this may limit its value. (Message)
  • If Unsure: The results of your analysis are likely to have a greater impact if they are motivated by an important gap in the existing research. Consulting a current systematic review can be an efficient way to ensure that your results have the potential to help address an important evidence gap. (Message)

Clinical/policy usefulness

  • What is the specific health decision the results are intended to inform? (Text box)

  • Who faces this decision? (Checkbox: Patients, providers, care givers, policy makers)

  • How will the results inform the health decision? (Text box)

  • Are both the target and the comparator exposures clinically appropriate choices? E.g. are they both first-line treatments for the same patient population? (Yes/No)

  • If No: What is the rationale for considering inappropriate choices? (Text box)

  • Do the target and comparator correspond to the actual healthcare options for patients, providers, caregivers or policy makers who would face the healthcare decision? (Yes/No)

  • If No: What is the rationale for comparing options that decision makers do not face? (Text box)

  • Is the outcome one that the population of interest notices and cares about? (Yes/No)

  • If No: What is the rationale for using an outcome that is unnoticed or unimportant to the decision makers? (Text box)

#2

I like this! Thanks for kicking this off @Andrew.

We might add to this:
‘To replicate a prior finding that one exposure is superior’ or non-inferior

and also add something to Clinical/policy usefulness
“Who faces this decision and what is the impact on persons or populations?” - something get # of person affected or seriousness of the outcomes.

#3

[quote=“schillil, post:2, topic:3522”]
I like this!
[/quote]I’m so glad.

Those are great additions.

#4

Impact should also include a concept of “action-ability” … is there something that can be done (a real treatment choice) that impacts patient outcome?

#5

[quote=“msuchard, post:4, topic:3522”]
Impact should also include a concept of “action-ability”
[/quote]I like the idea of emphasizing this concept. @msuchard

Since I think there is some overlap, maybe we should modify this question:

to something like:

If the results clearly favor one option, will they be actionable? I.e. are the target and comparator options typically available to decision makers?

What do you think?

#6

Agree @schillil that this will be very valuable. Do you think we should ask something like:

Approximately how many people are directly affected by this decision each year?

The last question is meant to do that:

But it doesn’t quite get at degree of seriousness, which I agree is important to get at.
Maybe a modified version like:

Does the outcome have a large impact on the health, functioning or quality of life of the population of interest? (Yes/No)

  • If Yes: Describe that impact. (Text box)
  • If No: What is the rationale for investigating an outcome that might not be important to the decision makers? (Text box)

Is that better?

t