Multimodal integration of radiology, pathology and genomics for prediction of response to PD-(L)1 blockade in patients with non-small cell lung cancer

Authors

Vanguri RS

Luo J

Aukerman AT

Egger JV

Fong CJ

Horvat N

Pagano A

Araujo-Filho JAB

Geneslaw L

Rizvi H

Sosa R

Boehm KM

Yang SR

Bodd FM

Ventura K

Hollmann TJ

Ginsberg MS

Gao J

Hellmann MD

Sauter JL

Shah SP

Doi

10.1038/s43018-022-00416-8

PMID: 36038778 · DOI: 10.1038/s43018-022-00416-8 · Journal: Nature Cancer (2022)

TL;DR

This study developed a machine-learning framework called DyAM (dynamic deep attention-based multiple-instance learning with masking) to integrate CT radiomics, digitized PD-L1 immunohistochemistry texture features, and genomic alterations from MSK-IMPACT sequencing to predict response to PD-(L)1 blockade in 247 patients with advanced NSCLC at Memorial Sloan Kettering. The multimodal model (AUC = 0.80, 95% CI 0.74–0.86) significantly outperformed unimodal biomarkers including TMB (AUC = 0.61) and PD-L1 TPS (AUC = 0.73), demonstrating that integrating routinely collected clinical data improves immunotherapy response prediction.

Cohort & data

  • Multimodal cohort: 247 patients with advanced NSCLC treated with PD-(L)1 blockade at MSK between 2014 and 2019, with baseline CT scans, digitized PD-L1 IHC slides, and MSK-IMPACT genomic data.
  • Radiology validation cohort: 50 patients with expert CT segmentations (did not meet multimodal inclusion criteria due to missing data from other modalities).
  • Pathology validation cohort: 52 patients with PD-L1-positive IHC slides.
  • Histology breakdown: 79% adenocarcinoma (LUAD), 15% squamous (LUSC), 3% large cell, 3% NSCLC NOS.
  • Response was binarized as responders (CR/PR, 25%) vs. nonresponders (SD/PD, 75%) per RECIST v1.1.
  • Median PFS 2.7 months (95% CI 2.5–3.0); median OS 11.4 months (95% CI 10.3–12.8).
  • Dataset: lung_msk_mind_2020.

Key findings

  • The multimodal DyAM model integrating radiology, pathology, and genomics achieved AUC = 0.80 (95% CI 0.74–0.86), significantly outperforming all unimodal and bimodal approaches.
  • Unimodal AUCs: TMB alone AUC = 0.61 (95% CI 0.52–0.70); PD-L1 TPS alone AUC = 0.73 (95% CI 0.65–0.81); CT radiomics averaging AUC = 0.65 (95% CI 0.57–0.73); genomic alterations + TMB AUC = 0.65 (95% CI 0.60–0.80).
  • Best bimodal combination was radiology + genomics (AUC = 0.76, 95% CI 0.69–0.83).
  • Fully automated trimodal approach (using IHC-G features instead of pathologist-scored TPS) achieved AUC = 0.78 (95% CI 0.72–0.85).
  • DyAM multimodal Kaplan–Meier stratification separated high- and low-risk patients more effectively than PD-L1 TPS or TMB alone: progression at 4 months was 21% for the lowest (protective) quartile vs. 79% for the highest (risk) quartile.
  • Multivariable Cox regression with clinical covariates yielded c-index = 0.74; DyAM risk score was independently significant (HR = 13.65, 95% CI 6.97–26.77, P < 0.005).
  • CT-based predictions were validated in the radiology cohort (lung parenchymal lesion AUC = 0.66, 95% CI 0.48–0.84, consistent with discovery cohort AUC = 0.64).
  • IHC texture features validated in the pathology cohort (AUC = 0.62, 95% CI 0.51–0.73, consistent with multimodal cohort AUC = 0.61).

Genes & alterations

  • EGFR mutation: 22/247 (8.9%); associated with worse PFS on immunotherapy (aHR = 2.14, 95% CI 1.06–4.31, P = 0.03); independent predictor in multivariate analysis.
  • STK11 mutation: 44/247 (17.8%); strongly associated with immunotherapy resistance (aHR = 2.53, 95% CI 1.71–3.74, P < 0.005); independent predictor.
  • ERBB2: assessed for both mutation and amplification; neither reached significance (P = 0.48 and P = 0.47, respectively).
  • ARID1A mutation: not significant (P = 0.44).
  • ALK fusion: 6% of cohort; included in multivariate model.
  • ROS1 fusion: 7% of cohort; included in multivariate model.
  • RET fusion: 5% of cohort; included in multivariate model.
  • MET: assessed for mutation and amplification; neither significant (P = 0.81 and P = 0.42).
  • BRAF mutation: not significant (P = 0.80).
  • TMB (median 7 mt/Mb, range 0–90): significant in multivariate analysis (aHR = 0.14, 95% CI 0.02–0.88, P = 0.04); higher TMB associated with better outcomes. TMB was uncorrelated with EGFR (r = -0.03) and STK11 (r = -0.01) mutations.

Clinical implications

  • Integrating routine clinical data modalities (CT scans, PD-L1 IHC, and targeted sequencing) into a single multimodal risk score substantially improves immunotherapy response prediction over any single biomarker, including the two FDA-approved biomarkers (PD-L1 TPS and TMB).
  • The DyAM attention mechanism handles missing modalities gracefully and provides modality-specific risk scores, enabling interpretability in the clinical setting.
  • EGFR mutation and STK11 mutation are confirmed as independent negative predictors of PD-(L)1 blockade response in this cohort, consistent with prior literature.
  • Quartile-based risk stratification using the multimodal score could inform early treatment decisions within the first 4 months.

Limitations & open questions

  • Single-institution study (MSK) without external validation of the full multimodal model; only unimodal radiology and pathology components were validated in separate hold-out cohorts.
  • Relatively small sample size (n = 247 multimodal, n = 50 radiology validation, n = 52 pathology validation).
  • Only 25% of patients responded (consistent with real-world rates), requiring class-balancing techniques.
  • Requires expert annotation for CT lesion segmentation and PD-L1 IHC tumor masking, limiting scalability without automated segmentation tools.
  • Commercial and institution-specific sequencing panels differ in coverage and germline filtering, posing challenges for multi-site generalization.
  • The study focused on baseline data; longitudinal or on-treatment data were not incorporated.
  • The DyAM model was not compared with simpler ensemble methods beyond logistic regression averaging.

Citations from this paper used in the wiki

  • “Our multimodal model (area under the curve (AUC) = 0.80, 95% confidence interval (CI) 0.74–0.86) outperformed unimodal measures, including tumor mutational burden (AUC = 0.61, 95% CI 0.52–0.70) and programmed death ligand-1 immunohistochemistry score (AUC = 0.73, 95% CI 0.65–0.81).” (Abstract)
  • “alterations of EGFR (n = 22/247, 8.9%; adjusted hazard ratio (aHR) = 2.14, 95% CI 1.06–4.31, P = 0.03), STK11 (n = 44/247, 17.8%; aHR = 2.53, 95% CI 1.71–3.74, P < 0.005) and tumor mutation burden (TMB) (median 7 mt per mb, range 0–90; aHR = 0.14, 95% CI 0.02–0.88, P = 0.04) exhibited significant aHR in a multivariable analysis” (Results, Genomic predictors section)
  • “The resulting c-index was 0.74 with several significant features: dNLR (hazard ratio (HR) = 6.87, 95% CI 1.76–26.77, P < 0.005), DyAM risk (HR = 13.65, 95% CI 6.97–26.77, P < 0.005)” (Results, Multimodal integration section)
  • “Progression at 4 months was 21% for the lowest (protective) quartile and 79% for highest (risk) quartile” (Results, Multimodal analysis section)

This page was processed by crosslinker on 2026-05-06.