OHDSI Home | Forums | Wiki | Github

March to the Broadsea

Implementers
#1

We have more-or-less successfully composed Broadsea on CentOS 7 (the Hades container not withstanding). We are having problems loading additional vocabularies to the system.

It starts out looking good:

[drando2@toolbox Broadsea]$ docker compose --profile omop-vocab-pg-load up -d
[+] Running 2/2
Container omop-vocab-load  Started  0.0s 
Container traefik          Running          0.0s

But then the container does not materialize. I expect it to spawn a container that will load the vocabularies stored locally:

[drando2@toolbox files]$ ls -l ~/repo/Broadsea/omop_vocab/files
total 5736216
-rw-r--r--. 1 drando2 domain users 1747031305 Dec  5 14:48 CONCEPT_ANCESTOR.csv
-rw-r--r--. 1 drando2 domain users      17697 Dec  5 14:48 CONCEPT_CLASS.csv

…

But nothing happens. Has anyone else run into this?

Thanks,
Dave

#2

We made a few fixes in the develop branch, have you tried that?

The 3.1 release has been a few times delayed due to our day jobs :slight_smile:

1 Like
#3

Hi David,

These .csv loads have been notoriously finicky for me. What methods have you tried so far? Only just the Broadsea profile?

Also, what issues are you having with Hades?

-S

#4

Hi David,

Have you tried to show docker logs from omop-vocab-load container in order to see what is happening ?

docker logs omop-vocab-load
#5

Yes, I have just used the profile at this point.

Regarding HADES, I see this:

[drando2@toolbox Broadsea]$ docker compose pull && docker compose --profile default up -d
[+] Pulling 1/1
 \u2714 traefik Pulled                                                                                                          0.4s 
[+] Running 5/6
 \u2714 Container broadsea-content  Started                                                                                     0.0s 
 \u2714 Container ohdsi-atlas       Started                                                                                     0.0s 
 \u2714 Container broadsea-atlasdb  Started                                                                                     0.0s 
 \u2714 Container ohdsi-webapi      Started                                                                                     0.0s 
 \u2714 Container traefik           Started                                                                                     0.0s 
 \u2826 Container broadsea-hades    Starting                                                                                    1.6s 
Error response from daemon: driver failed programming external connectivity on endpoint broadsea-hades (7dadc137a10100090e3e4fb194c1d6acf30d359974e35bed0da22fefd4b8b0b5): Error starting userland proxy: listen tcp4 0.0.0.0:8787: bind: address already in use
[drando2@toolbox Broadsea]$
#6

10-0-0-2:Broadsea $ docker logs omop-vocab-load
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/aarch64/APKINDEX.tar.gz
(1/8) Installing postgresql-common (1.1-r3)
Executing postgresql-common-1.1-r3.pre-install
(2/8) Installing lz4-libs (1.9.4-r1)
(3/8) Installing libpq (15.5-r0)
(4/8) Installing ncurses-terminfo-base (6.3_p20221119-r1)
(5/8) Installing ncurses-libs (6.3_p20221119-r1)
(6/8) Installing readline (8.2.0-r0)
(7/8) Installing zstd-libs (1.5.5-r0)
(8/8) Installing postgresql15-client (15.5-r0)
Executing busybox-1.35.0-r29.trigger
Executing postgresql-common-1.1-r3.trigger
* Setting postgresql15 as the default version
OK: 13 MiB in 23 packages
psql: error: could not translate host name “broadsea-atlasdb” to address: Name does not resolve
10-0-0-2:Broadsea $

#7

Ah, the broadsea-atlasdb container wasn’t running because a local PostgreSQL instance was squatting on the default port. So everything seems to be in order on my Mac.

My original problem is on Linux, and it appears to be related to a networking problem that prevents the Postgres client from being installed:

[drando2@toolbox Broadsea]$ docker logs omop-vocab-load
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.17/main: network error (check Internet connection and firewall)
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.17/main: No such file or directory
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.17/community: network error (check Internet connection and firewall)
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.17/community: No such file or directory
ERROR: unable to select packages:
  postgresql-client (no such package):
    required by: world[postgresql-client]
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.17/main: network error (check Internet connection and firewall)
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.17/main: No such file or directory
ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.17/community: network error (check Internet connection and firewall)
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.17/community: No such file or directory
ERROR: unable to select packages:
  postgresql-client (no such package):
    required by: world[postgresql-client]
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.17/main: network error (check Internet connection and firewall)
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.17/main: No such file or directory
ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.17/community: network error (check Internet connection and firewall)
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.17/community: No such file or directory
ERROR: unable to select packages:
  postgresql-client (no such package):
    required by: world[postgresql-client]
#8

The development branch seems to behave the same way. Thanks.

#9

On the Mac, I was surprised not to see the updated vocabularies in ATLAS. This is because by default omop_vocab_load populates a separate schema called omop_vocab, which is not accessed by ATLAS. ATLAS keeps looking at the demo_cdm schema, which has its own set of vocabulary tables. You need to change the setting in Section 9 of the .env file:

VOCAB_PG_SCHEMA="demo_cdm"

and then rerun the

docker compose --profile omop-vocab-pg-load up -d

command. This makes the new vocabularies visible to ATLAS.

Another key point is you can access the broadsea-atlasdb PostgreSQL database running in the container through localhost:5432 on a Postres client. I am using DbVisualizer. The userid is postgres, and password is mypass.

#10

One more tip: to access the WebAPI, you need to use the IP address. localhost doesn’t work. To wit,

http://127.0.0.1/WebAPI/vocabulary/descendantofancestor/

Unfortunately, some of the API calls are disabled by default, including this one. I get this JSON:

{
  "payload": {
    "cause": null,
    "stackTrace": [
      
    ],
    "message": "An exception occurred: javax.ws.rs.NotAllowedException",
    "localizedMessage": "An exception occurred: javax.ws.rs.NotAllowedException",
    "suppressed": [
      
    ]
  },
  "headers": {
    "id": "5c91c454-b9dc-dc3f-44e5-a95010becd42",
    "timestamp": 1703897364246
  }
}

It seems that ATLAS has access to parts of the WebAPI that Joe User is not. Is there a way around this? I am trying to compile a list of source codes that are included in a concept set, and I would like to do this through the WebAPI.

Thanks to all of you for helping me get this far.

#11

Hello @rndlph,

So let’s summarize:

  1. The Linux install of PG Client failed, but on Mac, it worked fine.

    • In develop, we’ve changed to Alpine 3.18.2, but it looks like pg client should be available in 3.17.
    • I’ve added a call to apk update to hopefully resolve this.

  2. While the omop-vocab-pg-load profile will load whatever batch of Athena downloaded OMOP Vocab files into a PG schema of your choosing, in Atlas, you still need to configure the app to use this vocab. The default source/source daimon setup that “ships” with Broadsea will indeed use an outdated vocab from the Atlasdb container.

  3. Indeed the atlasdb PG instance is accessible on the host, we’ll look to add a section to the Readme on this.

  4. Not all API calls are open, especially if you have security enabled, you would need to authenticate and have a user role that can access it.

  5. To get all source codes in a concept set, I’d highly recommend using the ROhdsiWebApi package, as that nicely wraps all the API calls into functions, and even supports authentication.

t