A 10-year-old CNN beat frontier LLMs on 200 skin lesions — but not on triage safety

A fine-tuned Inception v3 classifier — an architecture first published by Google in 2015 — outscored two frontier multimodal models on a 200-lesion skin-cancer test, hitting 62% top-1 diagnosis accuracy versus 42% for GPT-6 Astra and Claude Fable 5.1.

But the bigger story is not the headline score. It is the operating point. At its preset threshold, the CNN was far more willing to discharge lesions than the LLMs: it discharged 76 of 100 safe lesions, but also 18 of 30 melanomas. The two frontier models were more conservative, discharging far fewer lesions overall — and far fewer melanomas — but also missing many cases that could have been sent home.

That makes the experiment less a simple “old model beats new model” story than a reminder that diagnosis accuracy and triage safety are not the same thing.

What was actually tested

The benchmark comes from a detailed write-up by Lynch Lab, which says it drew 200 biopsy-proven lesions at random from MILK10k, a public dataset released through the International Skin Imaging Collaboration in 2025.

Each lesion had two views:

• a standard clinical close-up
• a dermoscopic image taken with a skin microscope

The sample was split evenly:

• 100 lesions that should not be discharged: 30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas and 20 actinic keratoses
• 100 lesions safe to discharge: 50 benign naevi and 50 seborrhoeic keratoses

That setup matters. Every lesion had been biopsied, including the benign ones. These were not easy, obviously benign pictures pulled from a textbook. They were lesions that had worried a clinician enough to cut them out.

The three systems compared were:

• GPT-6 Astra
• Claude Fable 5.1
• a fine-tuned Inception v3 CNN

The LLMs were shown both images and asked for the most likely diagnosis out of six options, a confidence score, a second choice, and whether the lesion could be safely discharged with no treatment, biopsy or follow-up. Lynch Lab says both models were run at default settings and neither refused a case.

The CNN was not a custom giant. It was Inception v3, pretrained on general photographs and then fine-tuned on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU. The page says the 200 test lesions — and every other photograph of the same lesion or patient — were excluded from training and validation, and the threshold was not chosen using those cases.

The headline number: 62% versus 42%

On first-choice diagnosis, the CNN got 125 of 200 right. The LLMs got 84 and 85.

That is the number most likely to travel. It is also the least complete way to read the result.

The models failed differently. In one example, an invasive melanoma under 0.8 mm thick was labeled a seborrhoeic keratosis by both LLMs, and both said it could be discharged. The CNN called it melanoma. In another, a seborrhoeic keratosis on the leg was misread by both LLMs as actinic keratosis, while the CNN called it squamous cell carcinoma.

So this is not a clean morality tale about one system being right and the others being wrong. It is a narrower result: on this benchmark, the specialist image model got more diagnoses right overall, and it made different mistakes from the frontier models.

The discharge table changes the interpretation

The more important table is the one most people will skim past.

Lynch Lab reports both diagnosis and discharge outcomes:

The CNN’s default threshold made it much more permissive than the LLMs. That helped it discharge 76 safe lesions, but it also meant it discharged 18 melanomas.

The frontier models were much more cautious. They discharged only 32 and 22 safe lesions, and they discharged just 3 and 4 melanomas. In triage terms, that is a very different tradeoff — one that may be safer, but also less useful if the goal is to avoid unnecessary follow-ups and biopsies.

Thresholds matter more than the raw score

The study’s most useful point is that the discharge decision is tunable.

When Lynch Lab says the CNN threshold is matched to the LLMs’ threshold, the safety picture changes fast. The table reports the CNN then keeps 98 or 99 unsafe lesions and discharges only 1 or 2 melanomas.

That does not erase the diagnosis gap. It does show why a single accuracy number is not enough to judge a triage system.

A model can score well on a six-way label task and still be too permissive or too cautious for clinical use. The threshold decides how that model behaves in practice.

What the benchmark does — and does not — prove

This is a controlled comparison, not a clinical validation study.

What it does support:

• a tuned specialist model can outperform frontier multimodal systems on a tightly defined dermatology image task
• threshold choice materially changes discharge behavior
• diagnosis accuracy alone does not capture triage safety

What it does not prove:

• that frontier LLMs are broadly bad at medicine
• that this CNN is ready for deployment
• that a 200-case sample settles real-world triage performance

Lynch Lab is explicit about those limits. The sample is small. The LLMs were each run once. The lesions were intentionally hard. And the CNN is not state of the art; the page notes that newer skin-cancer systems use stronger ensembles and transformer-based approaches trained on more data.

There is also a contamination caveat. MILK10k has been public since 2025, so the LLMs may have seen some of it during training. If anything, that would have helped them. The CNN had not seen these lesions.

The practical takeaway

The practical lesson is not that bigger models always win.

If the job is a narrow visual classification task, a specialist model tuned for that domain can still beat a general-purpose frontier model. But if the job is triage, the real question is not just which model names the lesion correctly. It is where the threshold sits, and what kind of mistake the system is allowed to make.

That is why this comparison is worth reading carefully. It shows that in medicine, benchmark design can change the conclusion. The CNN won on diagnosis. On discharge behavior, the picture depends on the threshold.

The result is still a win for the specialist model. Just not the simple kind suggested by the headline.

Source

• Lynch Lab, A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis: https://lynchlab.net/llm-skin-cancer