MindMeetsMachines Vocabulary Edition @OHDSI EU Symposium

General
1

Please join us at the MindMeetsMachines Vocabulary Edition challenge at the OHDSI EU Symposium (Sunday April 19th 1pm-3pm)!

Motivation
Mapping source codes to Standard Concepts in the OHDSI Standardized Vocabularies is hard, but essential if we want to collaborate. Let’s therefore test the current level of our human concept mappers and our AI concept mappers and learn from each other!

What will happen
We will be mapping (samples of) source codes related to Procedures and Measurements from all over Europe and see where OHDSI’s best human mappers outperform the artificial ones and vice versa.

  • The AI teams that want to participate are expected to prepare their systems upfront. Additional details of the task the systems should be able to perform is attached to this post.
  • Human experts who want to participate do not need to prepare and should just bring their experise and themselves :slight_smile:)

The goal is to get a better understanding of what is required to do good mappings both manually and through LLM orchestration.

Who should come

  • AI/GenAI mapping tool developers
  • Manual mapping experts (those who map codes)
  • Anyboby interested in mappings

Instructions and mapping assumptions:
procedures_labs_mapping.docx (18.4 KB)

Tagging the Challenge Team: @schuemie, Tom Seinen, Matthijs Otten, @Dymshyts, @m-khitrun and @zhuk

2 Likes
2

I’ll be there as a contestant! @aostropolets , it sounds like just show up with our kit, and then we’re entered?

1 Like
4

Yay!!! Yes, please just come :slight_smile: For those who can’t come in person we will have an asynchronous option. The post above has description of the task (in case your current approach doesn’t quite fit it)

5

Hi @aostropolets Do we need to pre-register somewhere if we attend as expert or curious person?

6

There is no separate registration. Would be great to see you! :slight_smile:

7

Link doesnt work @aostropolets :slight_smile: I would love to be there virtually.

Question - are all source-codes also OMOP concept_id’s (i.e. they are vocabularies recognized by OMOP vocabulary tables or they source codes that do not have concept_id)?

8

Hi @aostropolets

I’m based in Japan and would love to participate asynchronously. Is there any registration or sign-up process required to join, or can we just get started with the materials once they’re shared?

Thanks!

9

Hi Friends!

Looking forward to see you next Sunday!

For those, who participate as Machine arm: we have prepared for you a train set that you can use to test/adjust your tools as well as updated description of the task.

For those, who want to participate as Human arm: please-please-please come and map the codes manually at the Workshop (you do not need to have tools or LLM experience for that, just some mapping experience :slight_smile: )!

Additional next steps for Machine arm:

  • We will share the test set next Friday for you to run your tools/approaches on

  • If you would like to be included in the assessment at the Workshop at the Symposium, we ask you to send us your results by Sunday April 19th 1pm Rotterdam time (CET).

  • If you need more time, your results will be included in publication but not the workshop

  • For those in-person, please come to the Workshop :blush: If you would like to highlight your tool/approach, please prepare and send us 1-2 slides about the tool.

  • For those virtual, we will create a Teams meeting for you to join. Please also send us slides if you like to talk about your tools.

We also sent you an email if we know you’d like to participate. If you have not recieved an email but would like to be included, please shoot us a message.

2 Likes
10

@Gowtham_Rao source codes are not in the Vocabularies, so no concept_ids.
@Maiko_Ishikawa very exciting to have you :slight_smile: Aside from the additional information above, please send me (ostropolets@ohdsi.org) your email so that I can include you in email chains.

1 Like
11

The workshop is just around the corner, and we are excited to see you in person/virtually!

Machine arm:

  • We published the test set for you (link). Hope you will have fun with it. Please send us your results by Sunday April 19th 1pm CET.

  • If you would like to talk about your tools/approaches at the workshop, please send us 1-2 slides by Sunday.

Human arm:

  • Reminder to please come on Sunday 1pm CET to manually map the codes with us!

We also created a meeting for those virtual to join (Meeting link).

12

test_codes_team_1.xlsx (14.6 KB)
test_codes_team_2.xlsx (14.2 KB)
test_codes_team_3.xlsx (14.7 KB)
test_codes_team_4.xlsx (14.1 KB)
test_codes_team_5.xlsx (14.4 KB)

13

Hello @aostropolets,

I’m curious about this session. Was it recorded?

14

Hi @aostropolets , @Dymshyts , and @schuemie ,

Thanks so much for holding a wonderful session and then quickly analyzing the results. It was highly coordinated and quite fun to be a part of. I spoke with one of the other competitors, Mathias, and he had the great idea that all 8 teams could try to combine forces and their specific tooling toward a central method, taking some of the key insights from each. I’m definitely onboard! Curious about next steps for this group, as well as a posting of results to the community if possible. Was a very insightful experience.

Thanks again

1 Like
15

Many thanks to everybody who participated!!
Please come to the next Generative AI and Foundational Models in Healthcare workgroup meeting on June 9th noon EST (link) to learn more about the results of the challenge and discuss what we as a community can learn from it and where we can go from here.

@MPhilofsky the original session wasn’t recorded but the WG meeting is usually recorded.