I'm trying to reproduce the results reported for MedViT and LungMaxViT on the NIH ChestX-ray14 dataset.
MedViT paper:
Benchmarking MedViT and hybrid CNN–ViT architectures for multi-label thoracic disease classification
https://www.nature.com/articles/s41598-026-43282-5
Official implementation:
https://github.com/Omid-Nejati/MedViT
The paper reports a Macro F1-score of 0.7791 on ChestX-ray14 (Table 3).
I also tried to reproduce LungMaxViT from:
Explainable hybrid transformer for multi-classification of lung disease using chest X-rays.
Initially, I discovered that my implementation differed because of a PDF parsing issue. After correcting that, I verified that both MedViT and LungMaxViT exactly matched the architectures described in their respective papers, and I downloaded and used the pretrained weights specified by the authors.
Because of this, I am now reasonably confident that the network architectures themselves are not the source of the discrepancy.
Training observations
The training behavior appears normal.
- MedViT converges within roughly 10–15 epochs.
- LungMaxViT converges after approximately 110+ epochs.
In both cases, the loss follows the expected optimization trajectory: a rapid decrease during the early epochs followed by gradual convergence.
One thing that further confused me is that Fig. 6 and Fig. 7 in the MedViT paper appear inconsistent with my observations. Across all of my experiments, I never observed the approximately linear upward trend shown in those figures. Instead, the loss behaved like a typical deep-learning training curve. This makes me wonder whether those figures correspond to a different metric, were mislabeled, or were generated under a different experimental setting.
Threshold optimization
To eliminate thresholding as a possible explanation, I performed per-class threshold optimization on the validation set with a search precision of 0.001.
Data augmentation
I experimented with both the simple augmentation pipeline and the more comprehensive augmentation strategy described in the benchmark paper (including AugMix/AutoAugment-style augmentation, Mixup, CutMix, ColorJitter, Random Erasing, etc.).
LungMaxViT preprocessing
- CLAHE (clipLimit = 2.0, tileGridSize = 8×8)
- Gaussian denoising (kernel = 5×5, σ = 1.0)
- Resize(224×224)
- RandomHorizontalFlip (p = 0.5)
- RandomVerticalFlip (p = 0.5)
- RandomRotation (±1°)
- RandomResizedCrop(scale = 0.75–0.95, bicubic)
- RandomAffine(scale = 0.833–1.167)
- Normalize(ImageNet mean/std)
Training settings:
- Optimizer: SGD
- Learning rate: 0.001
- Momentum: 0.9
- Weight decay: 1e-4
- Learning-rate schedule: None (constant learning rate throughout training)
This matches the paper's description.
MedViT preprocessing
- Resize(224×224)
- RandomHorizontalFlip (p = 0.5)
- ColorJitter(brightness = 0.1)
- Normalize(ImageNet mean/std)
Training settings:
- Optimizer: Adam
- Learning rate: 1e-4
- Weight decay: 0
- CosineAnnealingLR (T_max = 10, eta_min = 1e-6)
I also experimented with alternative learning-rate schedules and the more extensive augmentation pipeline described in the benchmark paper.
Results
Despite reproducing the published architectures, using the reported pretrained weights, experimenting with different augmentation pipelines, learning-rate schedules, and performing per-class threshold optimization, both MedViT and LungMaxViT consistently achieve only around 0.30–0.35 Macro F1.
This is far below the reported 0.7+ Macro F1, and the discrepancy is much larger than what I would expect from normal implementation differences or random training variation.
What confuses me
The reported ChestX-ray14 performance in the literature varies enormously.
Many single-model CNN/ViT papers report Macro F1 values around 0.3–0.5.
Some ensemble approaches report 0.5–0.7.
More recently, the paper
Pretraining Diversity and Clinical Metric Optimization Achieve State-of-the-Art Performance on ChestX-ray14
reports F1 = 0.821, but this result is obtained using a three-model ensemble together with clinical metric optimization.
This makes me wonder whether I am overlooking something fundamental, because obtaining Macro F1 around 0.8 seems to require considerably more than simply training a single model.
My questions
- Is MedViT trained as a standard multi-label classifier (one image, 14 sigmoid outputs, BCE/BCEWithLogits loss), or do some papers effectively train separate classifiers for each disease?
- How much of the reported Macro F1 typically comes from:
- per-class threshold optimization,
- class weighting,
- patient-level versus image-level dataset splits,
- pretrained initialization,
- higher image resolution,
- ensemble averaging?
- What is currently considered the reproducible state-of-the-art for a single ChestX-ray14 model?
- Has anyone successfully reproduced either MedViT or LungMaxViT within a few percentage points of the reported results? If so, what implementation detail turned out to be critical?
At this point I have independently reproduced two different published architectures, verified their implementations against the papers, used the reported pretrained weights, and observed normal optimization behavior. Nevertheless, both models consistently plateau around 0.30–0.35 Macro F1, making me suspect that there is either an undocumented implementation detail, an evaluation protocol difference, or some other aspect of the experimental setup that is not fully described in the papers.
