OHDSI Home | Forums | Wiki | Github

Organization of ShinyDeploy

Developers
#1

In the past few months the number of apps on OHDSI’s Shiny server has gone from 5 to 18. Although that is a welcome development, I’m not sure all these apps have similar life spans, and requirements over these life spans. Some apps, like SystematicEvidence and AhasHfBkleAmputation are supplements to published papers, and must be kept alive for eternity. Other apps like oxfordStudyathonData1 I wonder if they need to be kept running for very long.

I see several issues with allowing unlimited unorganized growth of the number of apps:

  1. Hard to find a specific app (although probably not a big issue because for example papers should link to a specific app).

  2. Increased resource requirements (some of these apps are really big in terms of disk space or memory requirements). I think it is even possible for one app to bring the entire server down, knocking out all others.

  3. Incompatible requirements on dependencies. All these apps run in a single R instance. If packages are updated that may require updating these apps. For example, I already had to make changes to SystematicEvidence to accommodate changes in ggplot2. If we have hundreds of apps that doesn’t seem feasible.

Perhaps we should have several Shiny servers? One for long-term apps, one for those supporting ongoing studies, and one sandbox for development purposes?

@lee_evans: any thoughts?

Multiple shiny apps fail to load
#2

I like very much that there is an OHDSI shiny server. I agree with putting some rules around it.

For app that can use the free pool tier, they can use shinyapp.io
(like this app here that did not make the cut (too much tidyverse and other problems) when I tried to use the OHDSI server)
https://omega.shinyapps.io/dxnineten/

#3

@schuemie we could implement the open source Shinyproxy solution on the OHDSI cloud to address points 2 and 3:

https://www.shinyproxy.io/#what-is-shinyproxy

Shinyproxy uses Docker containers to isolate the server resources and R/R package version dependencies for each Shiny app,

Shinyproxy requires the use of a Dockerfile for each Shiny app to specify it’s server resources & dependencies. I could create Dockerfiles for the existing OHDSI Shiny apps as working examples. The existing Shiny apps would retain their current URLs.

I would also need to update the OHDSI Jenkins build process to use the Shiny app Dockerfiles as part of the automated build/deployment from the ShinyDeploy GitHub repo.

Additional benefits to deploying Shinyproxy are the ability to support more simultaneous users for each Shiny app and optional support for LDAP/TLS authentication if we needed some future Shiny app to be limited access.

Regarding point 1 we could use the Shinyproxy landing page to provide an attractive HTML page with a brief introduction to the data.ohdsi.org site and the URL link (and short description) for each OHDSI Shiny app.

If you think implementing Shinyproxy on data.ohdsi.org is worth pursuing I can follow-up with @hripcsa and @Patrick_Ryan about funding for the additional OHDSI cloud infrastructure & the deployment/support activities.

4 Likes
#4

This is a great proposal, Lee.
Isolating the apps from each other is a good approach.

#5

I certainly like the idea of separate containers for the long-term apps. Would it make sense to have a mix? Docker containers for ‘frozen’ apps that are not expected to ever change, and a shared Shiny server (perhaps in a single docker container) for the shorter-term apps to allow for more flexibility when making changes?

#6

@schuemie

I’d say we should aim to support the following scenarios:

  • A ‘frozen’ app still requires an older version of an R package/OS dependency.
  • Different shorter-term deployed shiny apps require conflicting versions of an R package/OS dependency (e.g. java versions)
  • Shiny app developers need to install their own R packages/OS dependencies in a shiny server, for development agility, but without impacting other developers/existing apps
  • The shiny server(s) need to be able to serve multiple shiny apps to multiple users simultaneously without blocking/queuing user requests

I’ll follow-up with @hripcsa & @Patrick_Ryan on this topic after we complete the OHDSI cloud upgrade of Atlas to v2.7.0 which I expect will happen later this week.

#7

This makes a lot of sense

#8

I agree with @admin

We’ve built several Shiny apps such as AEGIS, ICARUS-WINGS, and SOCRATex based on CDM.

And now I’m thinking about adopting ShinyProxy to deploy these apps for internal use.

#9

@lee_evans : is ShinyProxy still the solution you would propose? (it has been a few years, so maybe things have changed)

#10

I recently set up shiny proxy on digital ocean. Wasn’t too bad but getting the dockerized shiny apps to work was finiky and debugging in docker is painful in my experience. I still don’t quite understand how the ports are mapped but I got it working.

RStudio/Posit Connect would be much nicer and easier to used but costs more. However there should be discounts for academic and small businesses and if it were used in a classroom setting and purchased by a university like Roux or Columbia it could even be free. Worth exploring I think. @krfeeney - What are the chances Roux could help OHDSI get a license for Posit Connect? Posit Academic Pricing | Use Posit's Tools in the Classroom

I think this might be one of those cases where it is “cheaper” to spend money (on Posit Connect) than to spend a whole bunch of time (dockerizing shiny apps for shiny proxy).

#11

Well, corporations aren’t people and same goes for institutes or universities. Not embodied. Can’t help or get anything. Also, won’t do your homework for you. :wink:

@Adam_Black, we’re not a vending machine. You can’t punch the glass and hope a bag of Doritos comes down. If you want a group to sponsor this, you’ll have to do your homework. Call the Posit Academic Sales and get some info on the Research license. Write out – at a minimum – what you’re trying to buy, what it covers (user seats, server size or whatever, etc), applicable license period and what the potential impact is to the community.

Send that my way and I’ll weigh in on what’s feasible relative to the community use cases we’re targeting for this fiscal year.

#12

I’m not sure I agree that institutions (like the OHDSI center at Roux) “can’t help or get anything.” But maybe that is true and it’s all about individual contributions. I have done my homework, and I do think that someone representing a university and using the ohdsi results in a classroom setting could get a free Posit Connect license that would drastically improve data.ohdsi.org for both users (including students) and developers (who can deploy at will and not worry about other apps breaking theirs). It doesn’t make sense for me to interact with Posit on behalf of Roux since I don’t work there and the first question I’ll be asked is which university I work for. The process is simple though but there are some requirements like allowing the university to be featured on a website for the software. If nobody in ohdsi who is affiliated with a university is interested in taking that on then we can use the ShinyProxy which will still be better than the current setup but requires dockerizing each app :+1:. Also updates will probably need to be done by an admin and not self service.

It would be nice to get an update on the repository of research results that the Roux is apparently setting up. Might be a good opportunity for collaboration. @krfeeney do you know who we should ask about the research results repository that the OHDSI Center at Roux will provide? Will it be using the ohdsi shinydeploy GitHub repo?

IMG_4078
IMG_40781170×1432 222 KB

#13

I would like to add a few things:

  • The images we build ShinyApps on are bloated. To reduce size and improve security/quality we may consider developing base images from upstream repos like debian-slim that are controlled by OHDSI and released under the ghcr.io namespace. This also allows signing and access control via GitHub.

  • Docker for the most part follows OCI container standards. However, to broaden the deployment of tooling to other container runtimes like Podman, Runc, k0/k3/k3s, we may want to consider focusing on OCI standards.

  • While there are not currently any Python based Shiny apps, but the Python Shiny package has had decent speed of development and good ongoing support. I could see it becoming a popular option — making separate python oriented containers important.

  • Nix and NixOS is fantastic for development and testing purposes. They can also generate docker images which in turn can be exported as OCI Standard images.

#14

OHDSI is all about evidence generation but our main evidence dissemination mechanism aside from publication isn’t working very well currently. Can we take a step back and simply create a list of requirements for an upgraded OHDSI results dissemination component? Perhaps someone in the 2023 Kheiron cohort would be interested in helping build a better system.

My suggested requirements to start the list:

  • The system works as intended by developers for users (apps start up and work as intended)
  • The system is highly available on the public internet
  • The system can scale up to host large results sets and support multiple concurrent users
  • The system is easy to maintain and does not require a lot of manual work

Perhaps an API or some kind of database layer would be nice. @Paul_Nagy mentioned integration with the community dashboard and linking results to the study code and publication.

Just trying to get a productive conversation going here.

1 Like
#15

Thanks @Adam_Black for continuing to drive this forward. Just for everyone’s awareness, @lee_evans is working on a proposal for a solution on behalf of the OHDSI central coordinating center that we can support for the community, which @hripcsa and I will review with the Steering WG before implementing. I’ll let Lee share additional details on this thread so we can gather more input from the community.

1 Like
#16

Thanks @Patrick_Ryan. Good to know there’s plan in the works!

#17

(Let me just point out that I am not aware of any other effort actually sharing intermediate, diagnostic, and final results at large scale over the Web. Unless I have missed one. So while we have a lot of work to do, as far as I can tell, we are still infinitely better than the alternatives.)

t