Problem with starting a new BROADSEA implementation

mjoss · August 13, 2018, 8:03pm

Hello, we are trying to setup broadsea to run against our MSSQL database and I am experiencing a problem that I hope someone here could assist with. I found that after downloading and configuring the files for MSSQL on github, I executed a ‘docker-compose up -d’ (apparently without error, we think) and then I ran ‘docker-compose down’ as described in the ‘Quickstart BROADSEA deployment’ instructions on github. After this I tried running my edited source-source-daimon.sql script in aqua data studio against our OMOP database, and we discovered that the schema ohdsi and the tables ‘source’ and ‘source-daimon’ were not successfully created when I ran the ‘docker-compose up -d’ command previously. We are unsure why these were not created due to the lack of an error in the log output.

This is what we see when I run the ‘docker-compose up -d’ and then the corresponding docker-compose down:
-bash-4.2$ docker-compose up -d
sqlserver_broadsea-webtools_1 is up-to-date
sqlserver_broadsea-methods-library_1 is up-to-date
-bash-4.2$ docker-compose down
Stopping sqlserver_broadsea-webtools_1 … done
Stopping sqlserver_broadsea-methods-library_1 … done
Removing sqlserver_broadsea-webtools_1 … done
Removing sqlserver_broadsea-methods-library_1 … done
Removing network sqlserver_default

Jeff Klann suggested I run a ‘docker-compose up’ (and then ctrl+c to exit) and this is the log that we see:
-bash-4.2$ docker-compose up
Creating network “sqlserver_default” with the default driver
Creating sqlserver_broadsea-webtools_1 … done
Creating sqlserver_broadsea-methods-library_1 … done
Attaching to sqlserver_broadsea-methods-library_1, sqlserver_broadsea-webtools_1
broadsea-webtools_1 | /usr/lib/python2.7/dist-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a “-c” argument specifying an absolute path to a configuration file for improved security.
broadsea-webtools_1 | 'Supervisord is running as root and it is searching ’
broadsea-methods-library_1 | 2018-08-13 19:07:22,609 CRIT Supervisor running as root (no user in config file)
broadsea-methods-library_1 | 2018-08-13 19:07:22,644 INFO supervisord started with pid 1
broadsea-webtools_1 | 2018-08-13 19:07:22,745 CRIT Supervisor running as root (no user in config file)
broadsea-webtools_1 | 2018-08-13 19:07:22,745 INFO Included extra file “/etc/supervisor/conf.d/supervisord.conf” during parsing
broadsea-webtools_1 | 2018-08-13 19:07:22,887 INFO RPC interface ‘supervisor’ initialized
broadsea-webtools_1 | 2018-08-13 19:07:22,887 CRIT Server ‘unix_http_server’ running without any HTTP authentication checking
broadsea-webtools_1 | 2018-08-13 19:07:22,889 INFO supervisord started with pid 1
broadsea-methods-library_1 | 2018-08-13 19:07:23,649 INFO spawned: ‘RStudio’ with pid 7
broadsea-methods-library_1 | 2018-08-13 19:07:23,661 INFO spawned: ‘Rserve’ with pid 8
broadsea-webtools_1 | 2018-08-13 19:07:23,895 INFO spawned: ‘deploy-script’ with pid 7
broadsea-methods-library_1 | 2018-08-13 19:07:24,283 INFO reaped unknown pid 38
broadsea-methods-library_1 | 2018-08-13 19:07:24,283 INFO reaped unknown pid 39
broadsea-methods-library_1 | 2018-08-13 19:07:24,500 INFO reaped unknown pid 97
broadsea-methods-library_1 | 2018-08-13 19:07:24,500 INFO reaped unknown pid 98
broadsea-webtools_1 | 2018-08-13 19:07:25,133 INFO success: deploy-script entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
broadsea-methods-library_1 | 2018-08-13 19:07:25,517 INFO success: RStudio entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
broadsea-methods-library_1 | 2018-08-13 19:07:25,518 INFO success: Rserve entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
^CGracefully stopping… (press Ctrl+C again to force)
Stopping sqlserver_broadsea-webtools_1 … done
Stopping sqlserver_broadsea-methods-library_1 … done

we thought, perhaps our configuration is incorrect, however for some reason we are not seeing any errors regarding whether or not we are successfully connecting to our server. When we run docker compose up -d we expect that it is going to try to connect to our server, but the log output has no indication of that.

For reference, this is how our 3 files are currently configured (btw they are all in the same directory, with a path that does not have any spaces.):

-bash-4.2$ ls
config-local.js docker-compose.yml source_source_daimon.sql
-bash-4.2$ pwd
/home/maj60/Broadsea/sqlserver
-bash-4.2$

“config-local.js”:

define([], function () {
var configLocal = {};

    // clearing local storage otherwise source cache will obscure the override settings
    localStorage.clear();

    // WebAPI
    configLocal.api = {
            name: 'OHDSI',
            url: 'http://localhost:8080/WebAPI/'
    };

configLocal.cohortComparisonResultsEnabled = false;
    configLocal.userAuthenticationEnabled = false;
    configLocal.plpResultsEnabled = false;

    return configLocal;

});

“docker-compose.yml” : (FYI, User and Password for our db is omitted for security. Also note that we tried our server path with and without the ‘.partners.org’ )

version: ‘2’

services:

broadsea-methods-library:
image: ohdsi/broadsea-methodslibrary
ports:
- “8787:8787”
- “6311:6311”

broadsea-webtools:
image: ohdsi/broadsea-webtools
ports:
- “8080:8080”
volumes:
- .:/tmp/drivers/:ro
- ./config-local.js:/usr/local/tomcat/webapps/atlas/js/config-local.js:ro
environment:
- WEBAPI_URL=http://localhost:8080
- env=webapi-mssql
- datasource.driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver
- datasource.url=jdbc:sqlserver://phsqlrpdr322.partners.org:1433;databasename=eMERGE_OMOP_Mart
- datasource.cdm.schema=dbo
- datasource.ohdsi.schema=ohdsi
- datasource.username=OMITTED
- datasource.password=OMITTED
- spring.jpa.properties.hibernate.default_schema=ohdsi
- spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.SQLServer2008Dialect
- spring.batch.repository.tableprefix=secret-database-name.ohdsi.BATCH_
- flyway.datasource.driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver
- flyway.datasource.url=jdbc:sqlserver://192.168.99.100;databasename=secret-database-name
- flyway.schemas=ohdsi
- flyway.placeholders.ohdsiSchema=ohdsi
- flyway.datasource.username=secret-user-name
- flyway.datasource.password=secret-user-password
- flyway.locations=classpath:db/migration/sqlserver

“source-source-daimon.sql” : ( FYI, User and Password for our db is omitted for security.)

– remove any previously added database connection configuration data
truncate ohdsi.source;
truncate ohdsi.source_daimon;

– OHDSI CDM source
INSERT INTO ohdsi.source( source_id, source_name, source_key, source_connection, source_dialect)
VALUES (1, ‘OHDSI CDM V5 Database’, ‘OHDSI-CDMV5’,
‘jdbc:sqlserver://phsqlrpdr322:1433;database=eMERGE_OMOP_Mart;user=OMITTED;password=OMITTED’, ‘sql server’);

– CDM daimon
INSERT INTO ohdsi.source_daimon( source_daimon_id, source_id, daimon_type, table_qualifier, priority) VALUES (1, 1, 0, ‘dbo’, 2);

– VOCABULARY daimon
INSERT INTO ohdsi.source_daimon( source_daimon_id, source_id, daimon_type, table_qualifier, priority) VALUES (2, 1, 1, ‘dbo’, 2);

– RESULTS daimon
INSERT INTO ohdsi.source_daimon( source_daimon_id, source_id, daimon_type, table_qualifier, priority) VALUES (3, 1, 2, ‘ohdsi’, 2);

– EVIDENCE daimon
INSERT INTO ohdsi.source_daimon( source_daimon_id, source_id, daimon_type, table_qualifier, priority) VALUES (4, 1, 3, ‘ohdsi’, 2);

Any help in resolving this issue would be greatly appreciated.
Thank you.

lee_evans · August 15, 2018, 9:43pm

@mjoss take a look at the troubleshooting section of the Broadsea README page:

It describes how to view the Broadsea-webtools log files which may provide some helpful error messages.

t_abdul_basser · August 23, 2018, 4:22pm

@mjoss once you follow @lee_evans’s excellent advice please feel free to post the relevant excerpt from the tomcat logs here.

I would just add that this looks very much like a flyway configuration issue. WebAPI uses a package called flyway to manage and execute installation and subsequent modifications to the WebAPI application database. It appears that you did not set the flyway settings in the YAML file and therefore WebAPI does not know how to connect to the database, what schema the application tables should be created in, what the credentials for connecting are, etc. Based on your setting it looks like they should be something like

- flyway.datasource.url=jdbc:sqlserver://phsqlrpdr322.partners.org:1433;databasename=eMERGE_OMOP_Mart
- flyway.schemas=ohdsi
- flyway.placeholders.ohdsiSchema=ohdsi
- flyway.datasource.username=OMITTED
- flyway.datasource.password=OMITTED

Try adding these settings, restarting with docker-compose and tailing the tomcat logs as per @lee_evans advice.

mjoss · September 13, 2018, 6:07pm

Hello,

Thank you for your responses. We configured the flyway lines in the YAML file, and discovered a slight error in our prior configuration. We also put the jdbc driver jar files in with docker-compose file. Then I ran the docker-compose up -d, and I see the OHDSI schema and tables are created in our DB. This is good. I was then able to run our modified source source daimon script on our DB, which then inserted values into the source and source-daimon tables. However, when I go to check the status of the containers using a web browser, we see that Atlas and calypso are not working correctly.

Namely when I try these 2 steps, we get errors:
Open the Atlas OHDSI web application at http://your-docker-host-ip-address:8080/atlas
Open the Calypso OHDSI web application at http://your-docker-host-ip-address:8080/calypso

I am using the IP address of the Linux instance that I am using to run docker, where I stored my YAML file etc. from above.

The ATLAS page has the following error:
application initialization failed
unable to connect to an instance of the webapi.
please contact your administrator to resolve this issue.

and Calypso has the following error:
HTTP Status 404 - /calypso
type Status report
message /calypso
description The requested resource is not available.
Apache Tomcat/7.0.88

We tried troubleshooting by following the instructions in the broadsea quickstart page:
We extracted the tails from stderr and stdout. These did have some vague errors, copid below, however we are not sure how to proceed this this information.

the first notion of an error in stdout:
Unable to translate SQLException with Error code ‘102’, will now try the fallback translator
//
2018-08-30 18:58:49.199 ERROR localhost-startStop-1 org.springframework.boot.SpringApplication - - Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘webApi’: Invocation of init method failed; nested exception is javax.ws.rs.WebApplicationException: HTTP 500 Internal Server Error

we also noticed this error:
Error starting ApplicationContext. To display the auto-configuration report re-run your application with ‘debug’ enabled.
2018-09-10 17:09:03.460 ERROR localhost-startStop-1 org.springframework.boot.SpringApplication - - Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘webApi’: Invocation of init method failed; nested exception is javax.ws.rs.WebApplicationException: HTTP 500 Internal Server Error

And here was an error we noticed in stderr:
org.apache.catalina.startup.HostConfig deployWAR
SEVERE: Error deploying web application archive /usr/local/tomcat/webapps/WebAPI.war
java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/WebAPI]]

If the entire tails of each of these would be useful, or if you would like to check our updated YAML file, please let me know and I can copy them into the thread.

Any help with this problem would be greatly appreciated. Thank you.

t_abdul_basser · September 13, 2018, 10:18pm

Please post relevant excerpts from the supervisor logs as per BroadSea instructions.

admin · September 15, 2018, 3:45pm

@mjoss all the Calypso OHDSI web app functionality was incorporated into ATLAS (a couple of years ago), so you can just focus on getting ATLAS up and running.

To debug we need to see the earlier errors in the docker container supervisor log files. Start with the error messages around the point where it tries to load the jdbc drivers.

mjoss · September 20, 2018, 5:48pm

Hi everyone,

thanks for the tips. Below I have copied the stderr and stdout log files as requested. If you have any idea on how to get ATLAS working from this, any help would be appreciated. Thank you.

root@2e60267eaf26:/var/log/supervisor# tail -1000 stderr