OHDSI Home | Forums | Wiki | Github

Quality of Mapping Process - Peer review on vocabulary mapping

Vocabulary Users
,
#1

We are conducting a project to review and add new concepts to ‘SUS’ [Table of Procedures, Drugs, Orthoses, Protheses and Special Materials (Brazilian Unified Health System)] vocabulary.
We will use Usagi to map the concepts and we would like to estabilsh a peer review in mapping process to standard vocabularies.
I would like to know if there is a standard process or application in OHDSI community to evaluate mapping quality (peer review etc).
If we do not have a standard process currently, I think it could be a good contribution to OHDSI community in the future.

Regards,

Carlos Campos

#2

Hello! Сurrently all vocabulary additions and updates are centralized at https://github.com/OHDSI/Vocabulary-v5.0. However, there may be new proposed changes tomorrow at 2022 European OHDSI Symposium

#3

There is a way to measure agreement in a task like mapping. You could use the standards developed by the clinical NLP community. The process is this

  • Create an annotation guideline
  • Training session with the annotators. This could be a video explaining the guidelines plus a small test to see if the person can correctly map.
  • Give each text to at least two annotators. In this way, you could measure inter annotation agreement.
#4

Great, @jposada ,
We are thinking to do exactly this. Probably we will use 3 annotators and measure inter annotation agreement.
Are there any R or Python script developed by NLP community to facilitate the analysis of agreement? → Aggregation of files, measure inter annotation agreement, list which terms had full agreement and which not…

#5

Hi @carlosalcampos - if you are using Usagi, you can use its built in approval function. The newer version of Usagi (1.4.3) asks you who the author on the respective machine is. You can also assign portions of an Usagi file to respective authors for mapping and can then agree on a 4-eyes principle of a reviewer looking at another user’s mappings and going through the approval process one by one. @Rijnbeek and @MaximMoinat had defined an approach where

  1. you create a completely working Usagi directory with vocabulary and Usagi file ready for mapping, then package this as a zip file for distribution
  2. the Usagi file can be worked on in different locations (make sure every contributor knows what they are supposed to work on)
  3. Use github to synchronize the Usagi file by having people pushing their file to their respective branch and have the project lead merge the branches to the master
  4. After the mapping add a second round for reviewing and again assign mappings to be reviewed to individual users
    Maybe Maxim can provide even better instructions.
    Cheers - Mik
2 Likes
#6

Looks great, @mik
Do you have any documentation of this process?

Regards

#7

Hi @carlosalcampos,
I am afraid we do not have formal documentation of this process. I think Michael explained it very well. With the addition that Usagi also supports ‘equivalence’ statuses to indicate how good the mapping captures the the original meaning.

1 Like
t