OHDSI Home | Forums | Wiki | Github

How do you support OHDSI tools?

The research data warehouse project that I run executes on Google Big Query. Google developers have created infrastructure that allows (most of) the OHDSI tools to run against GBQ or are willing to make additional changes to make tools work.

Now the challenge is figure out how best to install and support the tools. At this time, OMOP is mostly unknown to our community so it is unknown how well it will be accepted, adopted and used. I feel the tools are a useful way to surface the potential of the underlying database so I am interested in installing them. But I am concerned about committing a full-time FTE while I don’t know if adoption will occur.

I am seeking insights into what other OHDSI sites have done to install, support, and promote the OMOP tools infrastructure (independent of the OMOP data infrastructure) and any advice you would have for staffing this. For example, I’m thinking about pursuing Comp Sci MS students to help at least spin up the infrastructure. But they could NOT help promote the tools. That seems to need a different skill set.

How have you staffed the technical and end-user support/training aspects of making the OHDSI tools useful/used in your setting?

Thanks for any insights.
Michael

My two cents:

A long time ago, before I started working on OHDSI, I had this vision/ideal
that I would lead the development of informatics solutions that performed
complex analytics on observational data, and would make them so easy to use
that thousands of people would gleefully log on and conduct observational
analyses. Back when I was at GSK, I helped build the SafetyWorks platform,
which then ultimately got commercialized by ProSanos…and while we built a
pretty nice and sophisticated tool which addressed many important safety
questions in the company, the uptake of users was low…we probably trained
100 scientists and only 15 regularly used it. When I joined JNJ, we built
a series of internal analytics tools (off OMOP CDM, but pre-OHDSI so now
with the open-source reality we had today), and each tool had a similar
story: the tools were successfully used to answer scientific/business
questions for multiple teams and always exceeded the business’
expectations, but the number of users was always much smaller than I
hoped. We built an application for safety, trained >100 folks and only saw
<40 use it more than once a quarter. We built a tool for clinical trial
feasibility, and while we have conducted >50 feasibilties in the last
couple years, our userbase for that sits at ~5. Even when we started
collaborating through OHDSI on ATLAS, I had expected to see a lot of uptake
across JNJ, but even to this day, there are probably <30 active users
(touched more than 1/week). It bothered me for quite a while, so I did
some customer interviews to try to understand what their obstacles for
adoption were. What I was surprised to find out what the customers didn’t
think there was a problem at all, as they were getting exactly what they
needed and expected! The reason: while everyone says they want to
‘generate real-world evidence’ and ‘conduct observational analyses’, what
it became apparent to me is that for the vast majority of those people,
what they really want is to ‘consume real-world evidence’ and ‘receive the
results of observational analyses’. They want to pose questions and get
answers, but they don’t want to do the work between Q & A. The difference
between being an evidence producer and an evidence consumer is quite
important in how you perceive your role and responsibility. And it has
nothing to do with the tool itself, it has to do with lack of training in
epidemiologic principles, lack of in-depth knowledge of the source data,
lack of statistical intuition for non-randomized trials, and more important
than anything else, LACK OF DEDICATED TIME. Even for the folks that are
strong advocates of observational evidence and are regular consumers of our
work (I’m thinking clinician/researcher type people, not the informatics
folks), they maybe at most are looking at observational analysis once a
week or every two weeks, and even then, it’s only for an hour at a clip.
So, it’s unreasonable to expect those people to be facile with an analytics
application if they don’t have the depth of knowledge to begin with and
aren’t expecting to touch it more than a couple minutes every once in a
while.

So, who exactly IS the target audience for the analytics tools like what
OHDSI is providing? My evolving opinion about this is: at all of our
institutions, there is a pretty large number of potential ‘evidence
consumers’ (e.g. every clinician or researcher) and a very small number of
legitimate ‘evidence producers’ (people executing the analysis to generate
the results); meaning you have likely a lot of demand but little supply.
And there are fundamentally two ways to produce evidence: 1) treat the
evidence as a custom one-off exercise, involving someone defining the
problem and developing an analysis plan (ideally written as a protocol),
someone who can cut a data extract from a repository, someone who can
manipulate that datacut into an analytical dataset, someone who can perform
statistical modeling on that analytical dataset, someone who can format the
results into human-readable tables and figures, someone who can synthesize
the results into a final report/publication; and 2) apply standardized
analytics procedures that perform specific use cases and go end-to-end from
repository to result. Even in the best of circumstances, #1 is
resource-intensive and time-consuming, both for the evidence consumer and
the evidence producer. For #2, the aim is to make the work of the evidence
producer less resource-intensive and time-consuming (since they are likely
your real bottleneck), but it doesn’t necessarily make the task of the
evidence consumer in asking the right question or interpreting the answer
much easier (not until they’ve done it enough to see a consistent,
reproducible pattern to follow).

So, my current strategy: I don’t hire people only to ‘install, support,
promote tools and infrastructure’. Rather, I hire folks to be members of
my Epidemiology Analytics team, and charge them with being responsible for
being an evidence producer. The skills I require is that they have the
technical skills to be a power user of standardized tools and have
scientific knowledge to understand the data and analysis procedures. I’ve
been extremely fortunate to build an amazing team of gifted scientists, but
recruitment is often a challenge, because usually, I can’t find people who
excel equally well with both the technical and scientific skills, so I end
up hiring folks who are stronger on one half and we train them on the other
half. For the folks who are strong on technical skills but weaker on
science, they may take the additional responsibility of tasks like
tool/infrastructure support and we train them on epidemiology/statistics.
For the folks who are strong on the science but weaker on the technology,
they serve as the testers of the tools so they can learn the tools inside
and out. But in all cases, for 100% of my folks, no matter their
backgrounds, from the most junior all the way through and including me, our
primary task is to produce evidence. Analytics tools and infrastructure is
the means to the end and a strategy for delivering evidence as efficiently
as possible, but it is not the end game in of itself. And our future
development activities are all driven by the needs identified from our
evidence consumers and by the gaps we directly face ourselves when trying
to use the standardized tools to meet the needs of our customers. I think
its important to have the people who are ‘supporting the tools and
infrastructure’ be the same people who have a vested interest in the use of
those tools, and having them BE the users has worked well in my opinion.
And specifically, we DON’T ‘promote the tools’, but rather we promote the
use of observational data to generate evidence and we promote the types of
analytic use cases we support to potential evidence consumers throughout
the organization. In many cases, the customer doesn’t even know (and
certainly doesn’t care) that we’ve used an OHDSI standardized tool to meet
their needs, they are just impressed that they asked their question and got
an answer so quickly. I’m now comfortable with the idea that adoption of
tools shouldn’t be the objective, but efficiency in evidence generation
should be the goal. And a correlate to that: you can be tremendously
successful with a small ‘tool user base’ so long as that userbase is able
to accommodate the needs of the broader ‘evidence consumer’ demands. If
demand continues to exceed supply, you have two levers that you can pull:

  1. bring on more tool users, and 2) improve the tools to make the current
    userbase more efficient. If your demand continues to outstrip our supply,
    I recommend making investments in both #1 (hiring more staff) and #2
    (investing development resources into improving the OHDSI tools).

So to your situation, if you wanted to get CS grad students to help you, I
think that can work so long as they have the aptitude and interest in
learning about the data and analytics (and you and your team have the
capacity to teach them). I anticipate that hiring someone to only do the
IT task of installation may not be as fruitful unless they are integrated
into the research process. And specifically, if that’s all you needed as
an immediate ‘win’: installation of the OHDSI toolkit in your environment,
then I’d direct you to Broadsea (https://github.com/OHDSI/Broadsea), which
is a docker instance that will provide you all the OHDSI tools, which you
can configure to point to your CDM instance, but you wouldn’t have to worry
about the other nuisances like setting up the webserver, webapi, etc.
During the OHDSI Symposium, @lee_evans showed how you can be up-in-running
with Broadsea in no time flat:). And if a docker install isn’t
straightforward or allowable in your infrastructure, members of the
community may be able to support you through contract work for
organizations to install the OHDSI tools.

2 Likes

Patrick, thanks for your thoughtful post. Getting to “why” end users want data vs insights is important and sometimes takes more probing than just determinng how they want to receive or access information.

A must read outstanding introspective account of fundamental innovation adoption ‘Blind Spots’.

My take:

  1. This high-value account underscores a classical UX (as in ‘User Experience’) design problem so pervasive in the convergent healthcare and life sciences space

  2. Efficiency in evidence generation is necessary, yet is not sufficient to drive adoption of innovative ways and means we need to compress the cycle time and economics of translating scientific discovery to real world scaleable actions - one person / patient / population at a time

  3. To “be comfortable with the idea that adoption of tools shouldn’t be the objective” runs the risk of propagating UX Blind Spots, thus continuing to handicap progress

The call to action? Hire, or otherwise augment the team w/UX expertise and leverage UCD (User Centered Design) best practice.

@mgkahn - great question. My two cents, but this time it is from the vendor side (I was on customer side as well in my previous life)

I am a believer that each business should be built by being focused on value based activities and delegating other. In your case, being on a research side, you want to make sure your focus is on conducting research and not on installing, configuring and upgrading platforms. This is where external SME(s) come in place and companies like Odysseus (Greg) or LTS Computing (Lee) could help you in managing that side of the capability.

Another option, if you want to be in the middle of driving the innovation and proactively developing the platform - you have multiple options - you could establish an internal capability like Patrick (@Patrick_Ryan) described, or - again - build a fruitful relationship with an external SME vendor. Regardless, I agree with Patrick 100% that this should not be a pure IT shop but rather companies that have a deep expertise and perspective from both IT and user sides. Those vendors have already polished this process and will be effective in both cost, speed (automation such as Docker) and quality of service perspectives.

To promote the tool - there is nothing better than a “power” user who use it on a day to day basis and deliver results/value to the organization.

You also need to validate your choices against the complexity of your environment e.g. a number of CDM instances, number of users, complexity of the research etc…

t