@dlrubin, Here's my model how to use unsupervised deep learning for medical image combined with structured clinical data to predict clinical outcome.
I'm trying to convert medical image into small-size vector representing the the original image, and combine it with other clinical features.
I uploaded the code to github for building encoder for medical image, and make feature vector from medical image based on radiology CDM. This is totally compatible with other OHDSI tool ecosystem, such as FeatureExtraction package and PatientLevelPrediction package.
I know i need more explanation to show entire process. I can show it in the poster at the OHDSI symposium.