Methodology for Converting Source-To-Concept-Map to Concept/Concept_Relationship (2 billionaires)

[roger.carlson]

One of the OKRs for the Healthcare System Interest Group (HSIG) is to create a methodology for moving the data in the SOURCE_TO_CONCEPT_MAP (STCM) table into the CONCEPT and CONCEPT_RELATIONSHIPS (C/CR) tables in the 2 billionaire range.

The STCM was originally created for OMOP v4 for the purposes of allowing the mapping of local codes to OMOP Standard Concepts. With the advent of OMOP v5, however, that functionality was transferred to the C/CR tables. However, for backward compatibility reasons, the STCM was not deprecated. The idea was (presumably) that folks would gradually migrate to the newer method.

Unfortunately, the STCM still remains in wide use for several reasons:

1. The Book of OHDSI recommends its use.
2. It is relatively simple to implement.
3. USAGI (a standard tool that helps with mapping) works with the STCM format.
4. Local mapping can be maintained by a separate team unfamiliar with OMOP.
5. There is no standard or recommended way to maintain the C/CR method.
6. Moving from the STCM to C/CR can involve a lot of ETL code modification.

Also unfortunately, there are several reasons to stop using SCTM:

1. Codes mapped in SCTM are not visible in ATLAS (OHDSI’s cohort creation tool) and other standard tools.
2. SCTM is not flexible enough to map the more subtle relationships available with C/CR like “Maps To Value”.
3. Hierarchies are not supported using STCM.
4. As an organization’s mapping becomes more robust, the SCTM will remain limited.

Additional questions:

1. What are the costs, benefits, and ROI (return on investment) with this move?
2. What is the recommended course for organizations new to OMOP to implement local codes? SCTM or C/CR?
3. How can organizations using STCM best convert their process to C/CR?
4. Is a hybridized method possible?
5. Should the STCM table be deprecated in future major OMOP releases (>6.0)?
6. Whatever happened to the Wide Mapping Table?

The HSIG would like feedback, stories, problems, solutions, opinions, etc. around this issue.

Thanks.
@MPhilofsky @Eduard_Korchmar @Daniel_Smith @Yacob_Tsegay_Gebrete @roger.carlson

Pertinent links:

• Chapter 6 Extract Transform Load | The Book of OHDSI
• Creating a registry of custom concept_ids (2-Billionaire Club) to avoid collisions across networks - Vocabulary Users - OHDSI Forums
• Wide MAPPING table (in vocabulary) (problems with relationship) - OHDSI Forums
• THEMIS Question: What do people put in the source_value fields - CDM Builders - OHDSI Forums
• Move local mappings from Source_to_concept_map to Concepts - CDM Builders - OHDSI Forums
• Data and use of source_to_concept_map table - Implementers - OHDSI Forums

[Daniel_Smith]

Cross posting this issue, pertinent to the adoption of a C/CR process in light of coordination with custom concepts across several institutions:

Creating a registry of custom concept_ids (2-Billionaire Club) to avoid collisions across networks - Vocabulary Users - OHDSI Forums

[aostropolets (Anna Ostropolets)]

Melanie presented a poster back in 2020 on this exact topic and I feel it answers all of your questions: https://www.ohdsi.org/wp-content/uploads/2020/10/Melanie-Philofsky-Philofsky-Mapping-Source-Codes-Poster.pdf

TLDR: if you can do C/CR do it (you’ll need to learn some vocab rules so that your instance doesn’t violate basic conformance checks like having ‘Maps to’ only to standard concepts).

[Eduard_Korchmar]

While Melanie’s paper a useful basis for creation of STCM to C/CR conversion model, the actual process has never been formally described. And QA SQL script, although, once again, very useful, is not a good basis to base the logic on:

1. Nor design decisions, nor implementation is explicitly documented;
2. Uses concept_*_stage data model, which is not a part of OMOP CDM, and just used by Vocabulary Team as implementation detail for authoring.
3. Does not cover modern cases – like “Maps to” and “Maps to value” targets domain consistency (“Meas Value” for “Measurement”)

What was discussed on HSIG call is a proposal to formally model the process of conversion, in documentation and/a as a conversion script. There is institutional inertia that keeps people using STCM – and if we want people to switch to C/CR, we need to answer some very crucial questions, not the least of which is:

@roger.carlson

What are the costs, benefits, and ROI (return on investment) with this move?

The conversion is not a trivial task. I am familiar with OMOP, so I can imagine what a process would look like, but every OMOP instance has it’s own established solutions and conversion scripts, often oriented to target STCM to store and update mappings. Until we have a clear model in place, we can not convince people, how easy or difficult it is for any OMOPized dataset to make the jump to C/CR.

[aostropolets (Anna Ostropolets)]

Absolutely agree, we do need a formal description/convention. @Eduard_Korchmar, given your expertise with the vocabularies and OMOP CDM, would you like to initiate and lead such an endeavor? That would be much appreciated. Or, if somebody already volunteered, it would be good to know their name to send help and questions their way.

[Eduard_Korchmar]

I do intend to do that. As a part of a planned OHDSI Python package, I want to include a module for a framework for converting data between STCM and 2billion concept space; this will by necessity include writing design documentation for the reference implementation. Once this exists, both the documentation and the library can be iterated upon following results of pilot projects. I plan to present a development proposal for this on the upcoming Open-Source Workgroup call on April 19th.

I do not have a proper announcement to link until then, but there is a long form forum message:

Implement OMOP CDM Bioimage Extension + Python Backend/Frontend interface

The normal mode of working with OMOP CDM “backend” is OHDSI family of R packages. Which is not ideal – R is a wonderful language when it comes tabular data operation and statistical analysis, but it is not a good general purpose language, like Python is. And in 2024, it is way easier to find a Python developer than an R one. I would imagine that image data is also not a prime use-case by R libraries. I am currently gathering use-cases for a Python toolset for purposefully interacting with an OM…

[roger.carlson]

I am also currently working on an approach. Mine is not so comprehensive as Eduard’s, which I’ve seen and is quite impressive.

It seems to me, however, that there is no single best approach because there are multiple use cases.

1. New to OMOP: Network Study
   An institution may be looking at building an OMOP instance only to contribute to a specific network study. They may never load the data into the standard OMOP tool stack at all. The Source_To_Concept_Map (STCM) is by far the simplest and fastest method. It is also the method detailed in the Book of OHDSI and works with USAGI.

2. New to OMOP: Enterprise-Wide Use
   An institution may be interested in OMOP for enterprise-wide use including implementing the full OMOP tool stack. These will want data to display in Atlas, but may also want to use STCM in the early phase development.

3. Expanding Single Purpose OMOP to Enterprise
   An institution may have a limited OMOP instance with already build ETL that uses the STCM. They will need to use Atlas and the other OMOP tools, but re-writing all of their ETL may be a hurtle. These may want to leave their ETL in STCM, but also write 2billionaire values to Concept and Concept Relationship (C/CR). There may or may not be ROI on re-writing the ETL.

4. Common Shared Mapping
   Institutions part of a network group may want to share common mappings, which can produce conflicts (Creating a registry of custom concept_ids (2-Billionaire Club) to avoid collisions across networks)

5. Jackalope+ Users
   If an institution is using Jackalope+ for mapping their only choice of export format is C/CR.

There are no doubt others. The point is that using STCM may be all someone needs. Or STCM may be a good place to start and then convert to C/CR. Using both STCM and C/CR may make sense for some. And others may decide to go directly to C/CR.

[roger.carlson]

So, when it comes down to the brass tacks of actually coding a solution, for the simple case of mapping local codes to standard codes, and ignoring reserved code blocks, I haven’t been able determine the following:

1. When a vocabulary/mapping is updated, is it permissible to delete >= 2Billion and re-assign new concept_ids

or

2. Do concept_ids need to be preserved and maintained?

The simplest method, by far, is to delete and re-write from either the USAGI csv files or STCM. Would this cause a problem with Atlas? I don’t think it would, but I’m not certain.

@Eduard_Korchmar @MPhilofsky @Daniel_Smith

[Eduard_Korchmar]

This completely depends on whether local 2billion concepts are explicitly used to define cohorts or not. If cohorts are defined by non-standard 2billion *_source_concept_id instead of their standard mapping targets, or if 2billion concepts themselves are made standard and used in concept sets, then it may be required to “lock” these concept_ids to preserve integrity of cohort definitions between versions.

[MPhilofsky (Melanie Philofsky)]

The best practice is to maintain the original 2billion concept_id when moving mappings from the STCM to the C/CR table. Since it may be hard to know all downstream uses of a concept_id, I would keep it the same.

@roger.carlson

Would this cause a problem with Atlas?

If it was used in an Atlas concept set or cohort, it would break if you changed it.