OHDSI Home | Forums | Wiki | Github

Steering committee: Proposal to Standardize Network Studies

General
#14

Counter proposal: In between “Started” and “Design Finalized” should be an explicit “Feasibility” step.

Rationale: There is significant legwork between a Started projected and a Design Finalized stage that is its own legitimate stage of progress. First commit is somewhat meaningless in comparison to the first time you test your Ts, Cs, or Os and begin to understand if your design actually works. This is a bit of a hole we fall into – often this is the stage we spent a considerable amount of time in. The limitation of current labels is we run the risk of underestimating the amount of community work going into studies – because many studies may linger at Started for an extended period of time without looking like they’re making progress. Of course, the commits in GitHub provide some transparency into what’s happening. It could be useful to have a tag for studies that are actively working through feasibility. This would also be a way to understand who may need community assistance – for instance, potentially using this self-reported tag as a mechanism for @SCYou’s Study Nurturing Committee to solicit new “nurturees” and help get past feasibility into a design finalized.

Examples:

  1. After migrating @BridgetWang’s Tofacitinib RA safety study to the new OHDSI Studies GitHub, it felt a bit disingenuous to tag this as “Design Finalized”. The study protocol was written and the study code’s been generated numerous times – but there are still some lingering design issues to debug before we can really say we’ve “finalized” the overall design such that we’d run it on other databases or publish the results / Shiny app.
  2. @mattspotnitz 's IUD studies are also “started” but I’d argue the design is still being massaged as they address issues identified during feasibility tests. Hint hint: @cukarthik and @mattspotnitz – a proper study repo would be really nice :wink:

I know less labels is generally the right approach. However, I feel strongly that if we’re using this as a way to pull metadata we should at least be giving some granularity into the abyss that exists between “Started” and “Finalized”. That’s my two cents. :moneybag:

1 Like
#15

By ‘Feasibility’, do you mean the state when the study has been determined to be feasible?

Note that the stages have deliberately been named after achieved milestones, not activities (that may or may not be taking place)

#16

@schuemie thanks for pointing to this post.

This has my full support to make this much easier to manage and remove the dependency between the studies in one repo which was kind of dirty.

I like the template proposed.

Something to consider is if we like to make the conditions/drugs that have been studied searchable as well as is done on clinicaltrials.gov. Maybe we can require each study to add a file with a list of conditions/drugs etc since this is searchable in the search box on the top of the Github if you select in this organization.

We may also consider to get some information from the github using the github API (for example: https://developer.github.com/v3/repos/#list-organization-repositories) on a page on ohdsi.org. I think we should be more proud of all the work we are doing and have it prominent on the ohdsi website. I could imagine a webpage that shows things like:

  • Some explanation on the purpose of the repo and link to it.
  • Latest study by tag
  • Most active study repo
  • etc.

I agree we can post our EHDEN studies here as well with a tag. We did get some requests for the possibility of private studies as well though that may require a different solution but lets not worry about this yet. I suggest we stimulate in Europe that if studies are registered at Encepp (http://www.encepp.eu) a link is added to the study repo hosted under this organisation.

Yes this is an ongoing discussion and we will also work on this in EHDEN in first quarter of 2020 by actually using ARACHNE in use cases together with Odysseus.

#17

Yes. Feasibility complete would be when the study hypothesis has achieved a test of viability for continued study. Activities at this point may include basic characterizations to ensure adequate cohort sizes and in more sophisticated study designs, provisional testing to understand potentially issues in covariate balance – the kind of things you’d want to know before you run a full PLP or PLE package.

This would be different than design finalized where the full design is completely pre-specified and published. A study can achieve “Feasibility” when an investigator has completed the legwork to create a partial pre-specified study design. At this phase, there may still be gaps in the study design that require addressing.

A study would stay in the “Feasibility” phase until the full study package is finished (e.g. all errors related to executing the package – things like highly correlated covariates or other design flaws – are resolved). The material difference between “Feasibility” and “Design Finalized” is that one is a interim work product and the other is a final work product.

1 Like
#18

Thanks for the nudge, @krfeeney :slight_smile:

I’ve have an updated repo after working out some kinks, but I don’t know how move it to the ohdsi-studies group. As mentioned in the study post we have two repos. The first one based on the Columbia study that was published is here and the second one based on a different T/C for claims his here.

1 Like
#19

I think only owners (or members with some rights) of your two repos and the organization (ohdsi-studies) can move the repos between the two.

#20

Ok, sounds good. I added a ‘Feasibility Established’ state. But I recommend adding no more!

#21

Hmmm, why are there two study repos? Are these two independent studies?

#22

Thanks @Rijnbeek, agree we should also use the GitHub API to get more information. My idea is to start by making a Shiny app that will replace https://www.ohdsi.org/network-research-studies/ by scraping the README files, and using the GitHub API.

I agree that would be nice, but I have a hard time imagining people will keep those files up-to-date. Let me think about it some more. If someone has ideas to make this more feasible, let me know.

#23

As long as I know, the cohort definitions were different

#24

I’m sorry, but I struggle to see how this differs from methods studies. Could you give a few examples of informatics studies that are definitely not methods studies?

#25

Hi @schuemie!

Both repos are part of the same study. One repo has target and comparator cohorts that were designed on an EHR database and the other has target and comparator cohorts that were optimized for a claims database. We are recommending that participating sites run both repos. The “IUD Study Updates” thread has additional information on this topic:

Thanks!

#26

Ok, I’m making some executive decisions:

  1. @krfeeney: Based on the lengthy discussion last Friday, I removed the ‘Feasibility Achieved’ status again, because I believe it to be undefined.

  2. @Vojtech_Huser: I’m not yet willing to add an ‘Informatics Study’ category until I see clear examples of studies that do not belong to the two existing categories (Clinical Application and Methods Research).

  3. @mattspotnitz: I’ve created a single repo for the IUD study. I also created a new rule: a study can have only one repo. Within the repo you are free to do what you want, so if you insist on having two separate study packages you can add them as subfolders. However, I recommend reconciling the two packages into one.

#27

Consensus was the semantic labeling was inappropriate (so no “Feasibility”) not to dismiss the entirety of adding a step. We talked extensively about a “Drafted” phase to incorporate the first run of a study on a local database that may still have to work through design considerations for scaling to beyond that data.

I feel strongly that the lack of a label between Started and Design Finalized would fail to capture the interim work product that is provisional study design prior to the necessary finalization. Drafted, as @Patrick_Ryan defined it, would be inclusive of the first run of the package on a local database – but it would not be a fully prespecified design. The completion of a full prespecification would be the “Design Finalized” phase.

#28

I don’t recall there was consensus :wink: I recall there was a much bigger discussion going on than just what meta-data tags to have, and I do think there was consensus that we should continue that discussion in the coming months, while we move forward with the meta-data tags we have right now.

Specifically the notion of a ‘Drafted’ tag still seems horribly vague to me. If you ran your study on your local data and have unblinded yourself to those results that to me implies you have finalized the design, because I wouldn’t want you to go back and change the design after that.

#29

Hi all!

I created a first version of a Shiny app that might replace the Network Research Studies page. The Shiny app scrapes our ohdsi-studies GitHub including the README files. Interestingly, but probably not surprisingly, I think nobody completely adhered to the template :wink: I went in and modified the README’s myself, so sorry about that.

I think overall the idea seems to be working, but it is clear we need some decision regarding tags, exposures, and outcomes.

Here’s a list of the tags used so far

  • EHDEN (3x)
  • Rheumatoid arthritis
  • rheumatoid arthritis
  • Drug Utilization
  • OHDSI-Korea
  • FEEDER-NET
  • F2F
  • drug safety
  • tofacitinib
  • etanercept
  • adalimumab
  • Xeljanz
  • Enbrel
  • Humira
  • RWD
  • RWE

I highlighted in bold the ones I would have used. I understand the desire to tag the exposures and the outcomes, but

  1. I would put those in a separate place to avoid clutter. It is not uncommon for an OHDSI study to include dozens of exposures and outcomes. @Rijnbeek proposed a CSV file. However, given the lack of adherence we see for the README template, I have little hope we can pull that of without continuous policing.
  2. We’d need a standardized terminology, as everyone on these forums will understand :wink: Luckily, we have one: our own Vocabularies. But combined with (1) I wonder if this is much more trouble than its worth.

Let me know what you think of the app, and what your thoughts are on tags, exposures, and outcomes!

#30

Let the forces of Brownian movement have more time. :smile:

I like this Shiny app. I showed it to a group of new investigators this morning. They liked the idea of having a place to know if their questions are duplicative or overlapping with other investigators.

If we accept that as one of the use cases for this page – we benefit from getting a larger sample of what naturally floats to the top and what tags feel unnecessary.

This is a subset of studies. I suspect there is a phenotype of who runs these kinds of studies :wink: Make the CSVs, see what happens. Why not use the vocabularies? We have to define our Ts, Cs and Os using OMOP standard concepts.

The labels today are the start. These are people’s blind attempts at adding “tacit knowledge” that might help someone find their study to collaborate or use it for their own future study. I think this is a very easy change management cycle here. Give them better suggestions of what you want and we’ll work on improving.

BTW, why not include tags for OHDSI studies that are the output of a dedicated community led effort (e.g. Face-to-Face, Study-a-thons, etc)? Don’t we have some questions outstanding about how these projects finish?

#31

I can see that several authors posted a study. Can you please clarify the process of requesting rights to create a repo. @schuemie

#32

@schuemie
First of all, thank you for this great job, again!

Still, I cannot see the ‘ticagrelor vs clpidogrel’ study. Is there anything wrong in the README file of this study?

1 Like
#33

Ok, here’s a proposal for a CSV file format:

  • conceptId
  • conceptName (I prefer my CSV files to also be human-readable)

We could also add a column type that distinguishes between ‘exposure’ and ‘outcome’, although I dread to think of how the freedom of a text field will be abused :wink:

We could call it 'relevantConcepts.csv"?

Anyone want to create the first CSV file?

There are currently 5 ‘owners’: @krfeeney, @schuemie, @jreps, @msuchard, and @Patrick_Ryan. Please ask one of them to create the repo for you. Provide that person a name for the repo and a one-line description that will be shown on the main GitHub page.

GitHub, like most web servers, is case sensitive. Your readme file was called ‘readme.md’, but the template specifies ‘README.md’. I changed the case for you. It may take up to 24 hours to show in the Shiny app.

1 Like
t