OHDSI Home | Forums | Wiki | Github

New Broadsea implementation

I am doing a clean Broadsea implementation on my windows laptop using the docker toolbox. I have followed instructions on the github but I am running into some issues. I have edited the configuration files docker-compose.yml and "source_source_daimon.sql as per the instructions although I cannot find any reference to the Atlas file config-local.js in the Broadsea repository!!

When I attempt to execute the docker-compose command (docker-compose up -d ) I get the error
ERROR: Couldn’t connect to Docker daemon - you might need to run ‘docker-machine start default’.
when I run the start command I get the response
Starting “default”…
Machine "default is already running.

Am I missing a step here?

Thanks

@ColinOrr the Broadsea github repo README.md file references separate repos for the Broadsea Atlas / WebAPI (Broadsea-WebTools) and RStudio/OHDSI R packages (Broadsea-MethodsLibrary) Docker containers.

The example config-local.js file is in the Broadsea-WebTools repo:

The Docker commands need to be executed in the Docker toolbox terminal instead of the regular Windows command shell.

I believe you can start the Docker toolbox terminal on Windows by clicking on the ‘Docker quickstart’ icon. In the Docker toolbox terminal you can run the following Docker command, to verify that Docker toolbox installed successfully.

docker run hello-world

@lee_evans, thanks for your response to this. In relation to the README.md it states;

This repository contains the example Docker Compose files used to launch the Broadsea Docker containers:

  • OHDSI R Methods Library (in RStudio) - maintained by Marc Suchard

  • Methods Library GitHub repository

  • Methods Library Docker Hub container image

  1. OHDSI Web Applications e.g. Atlas & Calypso (in Apache Tomcat) -
    maintained by Lee Evans LTS Computing LLC

    1. Web Applications GitHub repository
    2. Web Applications Docker Hub container image

I understood from opening statement that all these components are contained within this repository or accessed directly as part of the build. When I look in the other repositories I am referred back to this repository. There are no further instructions in the other repositories. So I am not sure where to find or create the Atlas files.

Regarding starting docker, all the commands I used where used within the docker toolbox terminal. I did run docker run hello-world and it confirmed that it had started, hence my head scratching…

Just to update this thread, which may be useful for others doing this implementation; I have documented what steps I have executed and entered comments on what is confusing to me. The instructions come from the Broadsea readme document

  1. Download and install Docker. --> COMPLETED

  2. Copy the example “docker-compose.yml” file for your database from this GitHub repository to a
    directory on your machine. --> COMPLETED

  3. Copy the example “source_source_daimon.sql” file from this GitHub repository to a directory on your machine. --> COMPLETED

  4. Edit the example “source_source_daimon.sql” file to specify the database connection strings and database schema prefixes for your database(s). --> COMPLETED

  5. Edit the example Atlas “config-local.js” file to specify the WebAPI URL, otherwise the default is localhost & port 8080. --> NOT COMPLETED. There is one in the Atlas project. Am I supposed to download that and store it somewhere on my local machine?

  6. Edit the docker-compose.yml file to specify the following:
    – set the WEBAPI_URL environment variable to your Docker host machine IP address. --> COMPLETED, I am using Docket Toolbox, I used the command below in parenthesis to get the ip address and set the WEBAPI_URL variable in the docker-compose.yml (docker-machine ip) WEBAPI_URL=http://192.168.99.100:8080

  7. specify the database connection info for your database --> COMPLETED this for both the “datasource” elements and the “flyway” elements. Use the same database credentials for both.

  8. Start the Broadsea Docker Containers: docker-compose up –d --> COMPLETED. First time I did this (successfully) it did a load of downloads. Earlier attempts failed due to me being on my work network which has every popularly used website on the planet blocked…

Only as part of initial configuration:

  • stop the containers (docker-compose down)
  • run your edited “source_source_daimon.sql” file in your database -->
    NOT COMPLETED. The tables SOURCE and SOURCE_DAIMON do not exist in my ohdsi schema. Is there an earlier process that should have created these?
  • start the containers again (docker-compose up -d)

View the status of the containers: docker-compose ps

This is the output from above command

$ docker-compose ps
Name Command State Ports
----------------------------------------------------------------------------------------------------------------------------------
dockertoolbox_broadsea-methods-library_1 /usr/bin/supervisord -c /e … Up 0.0.0.0:6311->6311/tcp, 0.0.0.0:8787->8787/tcp
dockertoolbox_broadsea-webtools_1 /usr/bin/supervisord Up 0.0.0.0:8080->8080/tcp

Non of these following links work.
• Wait up to a minute for the Broadsea containers to start.
• Open the OHDSI RStudio web interface in a web browser at http://your-docker-host-IP-address>:8787
• Open the Atlas OHDSI web application at http://your-docker-host-ip-address:8080/atlas
• Open the Calypso OHDSI web application at http://your-docker-host-ip-address:8080/calypso

Just as an fyi also.

$ eval $(docker-machine env default)

ICON-EU+OrrC@DUBL-8S83TC2 MINGW64 /c/Program Files/Docker Toolbox

$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
default * virtualbox Running tcp://192.168.99.100:2376 v1> 7.11.0-ce

ICON-EU+OrrC@DUBL-8S83TC2 MINGW64 /c/Program Files/Docker Toolbox

@ColinOrr I’ve now updated the docker-compose.yml file to work correctly with the latest ATLAS/WebAPI version 2.2.0,

Please download the latest version of the docker-compose.yml file for your DBMS from the Broadsea GitHub repo and update it with your database connection info & WEBAPI_URL again.

FYI. I’ve also added a copy of the example config-local.js file to the Broadsea GitHub repo. It’s the same version of the file that is in the Atlas repo so you don’t need to download it again.

After you’ve downloaded the latest version of the docker-compose.yml and re-applied your database connection changes please verify that you have that file and the config-local.js file in the current directory. Also verify that the WEBAPI_URL host in docker-compose.yml and the url host in the config-local.js are set to use the docker toolbox IP address.

For example, when I tested, my docker toolbox IP address was 192.168.99.100 so I used WEBAPI_URL=http://192.168.99.100:8080 in docker-compose.yml and url: ‘http://192.168.99.100:8080/WebAPI/’ in config-local.js

Next run the following commands to restart the docker containers - wait a couple of minutes after running these commands so the containers have time to restart:
docker-compose down
docker-compose up -d

The next step is to troubleshoot and see what errors are being generated inside the running Docker container. Please use the following commands:

Viewing The Broadsea Web Tools Log Files

Find the name of the running Broadsea Web Tools Docker container (the value in the “Names” column):

 docker-compose ps

Connect to the Broadsea Web Tools container:

docker exec -it <broadsea-web-tools-container-name> bash

Change directory to the log directory in the running container and view the stderr and stdout log files:

(The “*” char is the bash shell file name wild card character).

  cd /var/log/supervisor
  tail -1000 *stderr*
  tail -1000 *stdout*

When you are finished troubleshooting you can exit out of the running docker container (it will continue to run) the command is:

exit

The last troubleshooting command shown above : tail -1000 *stdout* command will show the error messages when the Docker container starts up (you will need to scroll back to find the error message of interest). You mentioned that the source and source_daimon tables are not being created (they should be automatically created when the container starts) so I suspect that the database user specified in the docker-compose file doesn’t have the necessary database permissions to create those tables in the database. The tail -1000 stdout command should show you a sql error message if that is the case.

@lee_evans, thanks for those details. I updated the with the new docker-compose.yml file and included in same directory as the config-local.js. After the restart I get the following. It seems to be a common problem but not any obvious (to me) solutions. Any insights or suggestions appreciated.

$ docker-compose up -d
Creating network “dockertoolbox_default” with the default driver
Creating dockertoolbox_broadsea-webtools_1 …
Creating dockertoolbox_broadsea-webtools_1
Creating dockertoolbox_broadsea-methods-library_1 …
Creating dockertoolbox_broadsea-webtools_1 … error

ERROR: for dockertoolbox_broadsea-webtools_1 Cannot start service broadsea-webtools: OCI runtime create failed: container_linux.go:295: starting container process caused “process_linux.go:399: container init caused “rootfs_linux.go:57: mounting \”/c/Program Files/Docker Toolbox/config-local.js\” to rootfs \"/mnt/sda1/var/lib/docker/aufs/mnt/92d2ab0f486fb2425a604d31d2d906e762fc80e24dc63a63b
61b3e9cd7e026a9\" at \"/mnt/sda1/var/lib/docker/aufs/mnt/92d2ab0f486fb2425a604d31d2d906e762fc80e24dc63a63b61b3e9cd7e026a9/usr/local/tomcat/webapps/atlas/js/config-local.js\" caused \“not a directory\”"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host pat
Creating dockertoolbox_broadsea-methods-library_1 … done

ERROR: for broadsea-webtools Cannot start service broadsea-webtools: OCI runtime create failed: container_linux.go:295: starting container process caused “process_linux.go:399: container init caused “rootfs_linux.go:57: mounting \”/c/Program Files/Docker Toolbox/config-local.js\” to rootfs \"/mnt/sda1/var/lib/docker/aufs/mnt/92d2ab0f486fb2425a604d31d2d906e762fc80e24dc63a63b61b3e9cd7e026a9\" at \"/mnt/sda1/var/lib/docker/aufs/mnt/92d2ab0f486fb2425a604d31d2d906e762f
c80e24dc63a63b61b3e9cd7e026a9/usr/local/tomcat/webapps/atlas/js/config-local.js\" caused \“not a directory\”"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
ERROR: Encountered errors while bringing up the project.

Both the docker-compose.yml and config-local.js are in the same directory C:\Program Files\Docker Toolbox. I am in this directory when I try to create and start the container.

@ColinOrr it looks like docker can’t map the config-local.js file correctly.

Please create a new directory that doesn’t have a space in the directory path and copy your docker-compose.yml and config-local.js files there and cd to that directory and do docker-compose up -d from there. Docker may not like the space in the ‘Program Files’ and ‘Docker Toolbox’ parts of the path.

e.g. create a directory like ‘C:\users<your windows user name>\broadsea’.
copy your two files to that directory
cd to that directory
docker-compose up -d

@lee_evans, From within the docker terminal it appeared that the spaces were tolerated but as you have suggested I moved the files to separate directory and shutdown and restarted. This has allowed the two containers to start (and stay up). yah…

There is an issue connecting to the database and creating the tables however. The stdout contains

----------> Parent Classloader:
java.net.URLClassLoader@3af49f1c at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:283)
at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:203)
at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:732)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:664)
at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:479)
at org.apache.tomcat.jdbc.pool.ConnectionPool.(ConnectionPool.java:154)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:107)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131)
at org.flywaydb.core.internal.util.jdbc.JdbcUtils.openConnection(JdbcUtils.java:50)
… 54 more
Caused by: java.lang.ClassNotFoundException: Unable to load class: oracle.jdbc.driver.OracleDriver from ClassLoader:WebappClassLoader
context: /WebAPI
delegate: false
repositories:
/WEB-INF/classes/

@ColinOrr good we’re making progress.

It seems to be looking for the Oracle jdbc driver and can’t find it. Are you using an Oracle database?

If so, follow the below info to install the oracle jdbc driver (it will be a ‘jar’ file). In summary you would need to download the oracle jdbc driver ‘jar’ file and put it in the same directory as your docker-compose.yml file and config-local.js file and restart the docker container.

Deploy Proprietary Database Drivers

The PostgreSQL jdbc database driver is open source and may be freely distributed. A PostgreSQL jdbc database driver is already included within the OHDSI Broadsea webapi-web-apps container.

If you are using a proprietary database server (e.g. Oracle or Microsoft SQL Server) download your own copy of the database jdbc driver jar file and copy it to the same host directory where the docker-compose.yml file is located.

When the OHDSI Web Tools container runs it will automatically load the jdbc database driver, if it exists in the host directory.

@lee_evans I think we are on the home straight. Finally getting a database error.

Unable to create schema “globe1”.

The schema already exists in the database with all my data and tables.

  • datasource.cdm.schema=globe1
  • datasource.ohdsi.schema=globe1
  • datasource.username=globe1

it have privs to create objects (tables, procedures etc). It does not have privs to create other schemas but not clear why it is trying to create a new schema.

Thanks for your patience :smile:

@ColinOrr OK, Oracle schema and user names are case-sensitive. Try changing all those references to ‘GLOBE1’ (upper-case) and restart the Docker containers.

thanks for that @lee_evans. I seem to have a working version of atlas :smile:
I did experience some additional errors but that were resolved by updating the docker-compose.yml file

I needed to update the repository table prefix line. I changed the OHDSI to the schema my CDM is in.

 - spring.batch.repository.tableprefix=OHDSI.BATCH_  

changed to

 - spring.batch.repository.tableprefix=globe.BATCH_

Its working now…

@ColinOrr Excellent, I’m glad to hear it’s working now!

I’ll make a couple of updates to the instructions on the Broadsea github README.md to help the next person.

t