OHDSI Home | Forums | Wiki | Github

How to add a custom vocabulary to the OMOP vocabulary table?

CDM Builders
#4

Hello, Tim @quinnt

Yes, there still would be an outcry, not mainly because you are not an authoring organization for the ICD10CM vocabulary, but because the OHDSI goal is to maintain vocabulary consistency and uniformity across the globe. ICD10CM should be the same in the USA, China, all over Europe, etc. Thanks to this uniformity, you can easily expand your researches and collaborate with the community.

What you want to do: you want to create your own vocabulary, map it’s terms to Standard concepts (preferred way), or create them as standard concepts (still possible) to create cohorts later. To build your own vocabulary, you need to populate

  • vocabulary: all the information on your custom vocabulary, done similar to other records, according to CDM specifications
  • concept: try to see everything as a concept in CDM as we do in object-oriented programming
    *concept_relationship: put your relationships from your concepts to standard using ‘Maps to’ relationship. Remember, if you can’t find a corresponding concept in Standard vocabulary and create your concept as standard, you still need to create ‘Maps to’ relationship from concept to itself.

2 bil something, the concept_id of concept you would create for your vocabulary ‘social_hx’

To illustrate what I’m talking about, try to check concepts of other vocabularies (ICD10CM is perfect example)

SELECT * FROM vocabulary WHERE vocabulary_id = 'ICD10CM';
SELECT * FROM concept WHERE concept_id = 44819098;

The book of OHDSI, Standardized Vocabularies chapter may be very helpful

Advice on how to map oral glucose tolerance test measurements at specific time points that do not have standard ids
#5

I would like to add my own custom vocabulary to the OMOP vocabulary table.

Why would I want to do this? Because the column concept.vocabulary_id in the OMOP concept table is required to be not NULL, so I need one to load concepts from my custom vocabulary.

Okay, let’s add a new row to the OMOP vocabulary table. I give it a vocabulary_id = “social_hx” and a vocabulary_name = “Social History”. Hmm, vocabulary_reference is also required, so I put in a URL to our internal wiki documentation. The column vocabulary_version is not required, so I skip it.

Now I get to the column vocabulary_concept_id. What is this? Seems a bit confusing. Let’s check the OMOP data dictionary: “A Concept that represents the Vocabulary the VOCABULARY concept belongs to.” Still confusing.

Okay, let’s see how the smart people of OHDSI actually use this: SELECT * FROM vocabulary; Okay, I see 96 rows, all of which have a concept ID populated in the vocabulary_concept_id column.

Let’s look at the one for ICD10CM. It’s concept_id = 44819098. Another query: SELECT * FROM concept WHERE concept_id = 44819098;

Aha! Now I see that each row in the OMOP vocabulary table also has a corresponding row in the OMOP concept table. The keepers of ATHENA have defined a “vocabulary catalog” of sorts, where the list of vocabularies is itself a vocabulary. This is the source of the confusion, because a “vocabulary of vocabularies” is hard to wrap your brain around.

Okay, so I want to add my custom vocabulary to the ATHENA vocabulary catalog. Can I simply add my custom vocabulary as a new concept in this “Vocabulary” vocabulary (which is in the “Metadata” domain)?

Or should I create a new catalog for my list of custom vocabularies called “Custom Vocabularies”?

#6

Yes, you can do it and keep all your vocabulary together.

1 Like
#7

A quick side-note as this topic might show up for a lot of people searching how to deal with custom concepts.

In my view, custom concepts should NEVER be marked as standard concepts. And only be used in the _source_concept_id fields. The source concept can still be used (in the local Atlas instance) to create cohorts.

If someone disagrees (which is fine), please reach out as this is an important principle that we need to agree on. We might need to start a separate topic or take this up with the CDM WG.

2 Likes
#8

@MaximMoinat,
I agree with your stance 1,000% and it matches exactly what we’re doing.

In our OMOP instance, we are loading our EHR’s master files (diagnoses, medications, procedures, lab tests, etc.) and category lists/lookup tables to the CONCEPT table with concept_id values above the 2 billion threshold.

We do this for two reasons:

  1. The ability to construct cohorts using any combination of “standard” and/or “source” concepts
  2. Transparent data lineage from our OMOP instance back to our EHR system

As you said, we NEVER mark these “source” CONCEPT records as “standard” and we ONLY use them within the _source_concept_id columns.

Additionally, we map them to “standard” concepts in the CONCEPT_RELATIONSHIP table with relationship_id = 'Maps to' (and 'Mapped from').

This thread was prompted by my questions about whether we could add a “source vocabulary” to ATHENA’s “Vocabulary Catalog”. What is ATHENA’s Vocabulary Catalog?

It exists in 2 places:

  1. All records in the VOCABULARY table
  2. All records in CONCEPT where domain_id = 'Metadata' and vocabulary_id = 'Vocabulary'.

We created our own “source” vocabularies and added them to ATHENA’s “Vocabulary Catalog”.

We also created our own “source vocabularies” concept class so that we could differentiate between our “source vocabularies” and ATHENA’s “standard” vocabularies within the “Vocabulary Catalog”.

1 Like
#9

Tangential to this topic, is there any standard guidance for using the Concept table for local codes? I created a separate topic.
Move local mappings from Source_to_concept_map to Concepts - CDM Builders - OHDSI Forums

#10

In my view, there is little added value of adding a concept for your custom vocabulary into the concept table. Just adding a record in the ‘vocabulary’ table with vocabulary_concept_id set to 0 works fine.

#11

The process explanation (attached link) is in line with what you posted in your tangential topic post:

#12

Hi Maxim,

Can you provide a brief explanation on how setting vocabulary_concept_id to 0 allows Achilles to characterize these custom concepts?

-S

#13

@Sanjay_Udoshi As far as I know, Achilles does not characterize the vocabulary_concept_id. But maybe I misunderstood the question.

#14

To add a custom vocabulary to the OMOP Vocabulary table, you can follow these general steps:

Prepare your custom vocabulary: Define the concepts, codes, and relationships for your custom vocabulary. Make sure you have the necessary information, such as concept IDs, concept names, domain IDs, relationships, and any additional metadata.

Create a concept table: The concept table is the main table in the OMOP Vocabulary database. It contains information about all the concepts, including your custom ones. You’ll need to insert your custom concepts into this table. Ensure that you have the required columns, such as concept_id, concept_name, domain_id, vocabulary_id, etc.

Define relationships: If your custom vocabulary has relationships with existing concepts in the OMOP Vocabulary, you’ll need to establish those relationships. Determine which existing concepts your custom concepts are related to (e.g., is-a relationship, maps-to relationship), and insert the appropriate rows into the relationship tables (e.g., concept_relationship table).

Load the data: Once you have prepared the concept and relationship information, you need to load the data into the OMOP Vocabulary table. This can be done using SQL statements or by importing data from a file, depending on the method you prefer.

Validate and test: After loading the data, it’s essential to validate the entries and ensure they align with the OMOP Vocabulary specifications. Perform checks for consistency, accuracy, and adherence to the OMOP conventions.

Update the metadata: Update the metadata of your custom vocabulary in the metadata tables, such as the vocabulary table, to provide additional details about your custom vocabulary.

Hope this helps.

#15

This perfectly works for concepts that sufficiently define the events. But people who build their highly customed CDMs mostly out of 2 bil+ concepts would argue with you. They prefer to look into the standard concept_id fields for the sake of convenience.

What about the survey data problem? People prefer to keep the questions and answers as separate concepts and do not pre-coordinate the whole vocabulary creating tons of new instances. Unless we have the value_source_concept_id field introduced, such sort of analytics on the survey data will not work, unfortunately.

I agree there’s not much value unless people submit their local vocabularies (at least the metadata) to the official OMOP vocabularies to get the concept_id assigned, and then leverage this across the data networks.

But how do Achilles and DQD react to the fact there’s no matching concept? Also, we usually explicitly allow this in the documentation if that’s allowed. Do we?

@Agnes_Wojciechowski Please chime in :wink:

1 Like
Mapping BI-RADS related terms to SNOMED CT
#16

Since DQD is paramount as standard tool for checkinfg OMOP it would be nice to add the guidance saying when creating custom vocab we are putting 0 as vocabulary_concept_id for it as a default. We can conjure some specific unique scenarios allowing it to be diffenerent, but from my experience it is not really necessairy in most cases. The reason for the explicit guidance is for people to adhere to requirement for not being NULL - OMOP CDM v5.4 - and DQD check to not to indicate the error which in this case is in my opinion really superficial.

#17

This post got me thinking about the utility and necessity of the vocabulary_concept_id.

In my view, the record in the Concept table for a vocabulary does not provide any necessary, additional information about the vocabulary. The record in the Vocabulary table for the vocabulary contains all necessary attributes. I am unaware of any use case for the vocabulary_concept_id. The most common use of the Vocabulary table is identifying the date of the Vocabulary batch being use in the CDM and that query uses vocabulary_name = ‘None’. Another common use of the Vocabulary table is to identify which vocabularies are being used in the CDM.

How does the OHDSI Vocab team use the vocabulary_concept_id field? Is it used by Atlas in some way I don’t know? Do we even need the vocabulary_concept_id field in the Vocabulary table?

@Alexdavv @aostropolets @Christian_Reich @Chris_Knoll @clairblacketer @MaximMoinat

#18

Any time we show you a concept in Atlas, we tell you which vocabulary it came from…which might be useful if you find a concept code that conflicts between codes between vocabularies.

If you are building a drug list, you should usually find it in RxNorm…if you pick something from another vocabulary you might want to check your work.

Likewise, if you are building a condition list, usually that comes from SnoMED, and if you choose from a different vocabulary you may want to check your work.

#19

I think @MPhilofsky means something else, @Chris_Knoll. She is challenging why we need a concept for each vocabulary, while we already have a record in the VOCABULARY table with vocabulary_id as foreign key to it. Only one should suffice, and we never seem to make use of the concepts for anything.

The answer is this, @MPhilofsky: There is a population in our community who thinks of the clinical facts and vocabularies as one continuous information space. Instead of fixed pre-defined queries, you could navigate and detect connections and knowledge. You use technologies like RDF, OWL and Triple Stores to do that efficiently: here, here, here, here and probably in more places.

The problem is that none of these have produced a successful application to a use case, yet, as far as I can tell. We put all those extra concepts in, with a certain burden to maintain them (we also maintain all tables and fields as concepts and relationships, for all versions), but I am not seeing the stream of tools or results.

If you want to propose to get rid of all this, go ahead, but make sure you talk to the sponsors first.

#20

Correct.

I don’t want to get rid of it. But we should give guidance for folks who map non-OHDSI supported source values/codes to standard concept_ids, aka custom mapping. It sounds like there isn’t an OHDSI network or clinical/medical research use case for the vocabulary_concept_id. So, concept_id = 0 would suffice for custom vocabularies. Correct? Of course the ETL can assign a concept_id > 2 billion for the vocabulary and create a record for it in the concept table if desired.

@MaximMoinat is presenting a proposal on “guidance for ETLers when mapping non-OHDSI supported source values/codes to standard concept_ids, aka custom mapping” to the Themis WG, Thursday October 5th at 9:30 Eastern Time. We encourage those who do custom mapping work to join us. The more eyes on the documentation proposal, the better the final documentation will be.

EDIT: Fixed the time for the meeting.

1 Like
#21

Please take a look and comment on the new documentation proposal created by @MaximMoinat

Maxim did a lot of great work on illustrating the custom vocab construction. There are 3 final details waiting on community input

2 Likes
#22

This is a side topic: how can this be done? Is there a downloadable Atlas which is slightly different from the online Atlas? Sometimes, we would like to use the source codes for local studies, but in the online Atlas, we could not find a way to do it.

#23

Great question. In Atlas Cohort Definition, you do the following:

  • Add initial event, do not specify a concept set (i.e. leave it at ‘Any Condition/Observation/Procedure’)
  • ‘Add Attribute’ → ‘Add … Source Concept’
    image
    image2674×1442 451 KB
  • Add your concept set containing source concepts
    image
    image1248×538 45.3 KB

This of course does rely on population of the _source_concept_id fields. Note that, as far as I know, you cannot combine using the standard concept and source concept (i.e. the standard concept set should be ‘Any …’). You then need to add an another initial event.

t