I made a molecule model's two inputs disagree. It believed the graph.

A molecule can be written down two ways. As a line of text called a SMILES string, where aspirin is CC(=O)Oc1ccccc1C(=O)O. Or as a graph, atoms as nodes and bonds as edges. MolCA reads both at once and writes an English description of the molecule. The paper treats the two views as partners. Nobody had checked which one the model actually believes when they disagree.

So I made them disagree. I handed the model one molecule's text and a different molecule's graph.

It described the graph.

This was my project for Prof. Dietrich Klakow's graduate seminar on language models at Saarland University. The brief was a critical verification, not a demo: reproduce a published result, then test whether the paper's central claim holds up. Here's what that turned into.

What MolCA is

MolCA (Liu et al., EMNLP 2023) teaches a text-only language model to see molecular structure. A graph neural network encodes the molecule. A Q-Former projector compresses that into eight soft prompts, which sit in the language model's input next to the SMILES string. The language model is Galactica. A LoRA adapter does the task-specific fine-tuning.

The paper's own ablation measures what each view is worth during training: 34.6 BLEU-2 with text alone, 34.5 with the graph alone, 38.7 with both. Read on its own, that table says two roughly interchangeable views combine for a modest gain. It says nothing about what the trained model does when the two views contradict each other. That's the gap I went after.

Step one: reproduce it, on two very different machines

Before you can make a claim about a model, you have to show you're running the model the paper describes. I evaluated the released checkpoint on the full 3,300-molecule ChEBI-20 test split, twice, on hardware that could hardly be further apart.

Two GPU generations, two Python versions, two major PyTorch versions and two transformers versions. The scores still land within 0.8 of each other. The aggregate hides how close they really are: 93.8% of the 3,300 captions are character-for-character identical between the two machines.

I also reproduced the paper's second task, molecule-to-text retrieval. Checking the cluster logs against the paper's tables covers all 32 published retrieval metrics. On the full test set, the largest deviation is 0.12.

Getting there took 14 patches

The released code assumes the authors' rig: two A100s, bfloat16 and a distributed process group. On a single consumer GPU, each of those assumptions breaks somewhere different. A few of the ones worth knowing about:

• The checkpoints carry cuda:4 device tags, so they won't even unpickle on a one-GPU machine.
• Two distributed calls run unguarded. The one in the evaluation hook crashes on a single GPU at the very last step, throwing away a two-hour run that had already finished.
• Validation uses the training batch size instead of the inference one. On an 8 GB card under Windows, that doesn't raise an out-of-memory error. It spills silently into system RAM until throughput drops to 0.03 iterations a second and the machine freezes.
• Every documented retrieval command crashes, because a data module never receives a tokenizer. That one isn't mine: it's issue #13 upstream, open since July 2024. I rediscovered it before reading the tracker. My patched run does settle the question that issue left open: wiring in the model's own tokenizer reproduces all 32 metrics, so a correctly connected tokenizer is all the retrieval code was missing.
• PyTorch 2.6 changed a default in torch.load, which now rejects the pickled graph objects the retrieval data ships as. It was invisible on the authors' PyTorch 2.0.

I searched all fourteen upstream issues and their comments for each finding. Several of these have never been reported. Every patch is idempotent and lives in one script, so anyone can apply them to the upstream code and get the same model I ran.

Step two: make the inputs disagree

With a faithful reproduction in hand, the experiment is simple. For each test molecule, keep its own SMILES string but swap in the graph of the next molecule in the list. Then read what the model writes. Here's the first test molecule:

its own SMILES describes : a steroid ester
the graph it was given   : a branched amino tetrasaccharide
the model wrote          : "The molecule is a branched amino tetrasaccharide ..."

A sugar. Not a steroid. The text sitting right there in its prompt said steroid. The model ignored it.

One example is an anecdote, so I scored it across 1,000 molecules. For every caption, I extracted the chemical class it asserts and checked which of the two molecules it matched. In roughly 90% of the decisive cases, the model describes the molecule that supplied the graph. Measured as class-agreement F1, the graph wins by about 9 to 1, against a floor near zero.

Then the cost of breaking each channel, with the same kind of damage applied to each:

Breaking the graph costs about 2.5 times what breaking the text costs. On the full 3,300-molecule split the gap between the two costs is 21.8 BLEU-2. A paired bootstrap puts its 95% interval at 20.9 to 22.5. Nowhere near zero.

A wrong graph is worse than no graph

This is the result I find most telling. If the graph were a weak side channel, feeding it garbage should cost about the same as feeding it nothing. It doesn't:

A real but wrong graph hurts more than an empty one. A scrambled one hurts more still. The model isn't hedging between two sources. It treats those eight graph prompts as authoritative evidence and follows them wherever they point.

The gap between "no graph" and "wrong graph" is small, 1.78 BLEU-2 on the full split. The whole argument leans on it. So I bootstrapped it: 95% interval 1.37 to 2.30. It holds.

Is it the graph, or just the position?

There's a boring explanation I had to rule out. The graph prompts sit closest to where the model starts writing. Maybe the model simply trusts whatever it read last. Then "the graph wins" is really "the nearest input wins."

The clean test would be to swap the order and put the graph first. I tried it and the model collapsed to 0.01 BLEU-2. So that instrument failed: it tells you MolCA is brittle to prompt layout, not which explanation is right.

The test that worked was gentler. I pushed the SMILES text 37 tokens further from the generation point and left everything else alone. Recency is real: the text loses a little influence when it's further away. But it's far too small to be the mechanism. The ratio between the two channels only moves from 2.46 to 2.74, so recency can't produce a 2.5-times dominance.

Two points show that recency exists, not how it behaves. So I added a middle distance and reran all three on my laptop, so no point compares two machines. Corrupting the SMILES costs 25.1% of the clean score when it sits where it shipped, 23.2% at 13 tokens away and 21.3% at 37. Every step down is significant. That's about a tenth of a point per token.

In the shipped prompt the graph sits eight tokens nearer the output than the end of the SMILES. At a tenth of a point per token, those eight tokens are worth under a point. The gap between the two channels is about 35.

Does it survive a second dataset?

I pointed the ChEBI-20 checkpoint at a different dataset, PubChem324kV2, a pairing the authors never ran. The raw score came out at 49.04 BLEU-2. That looks wonderful, because the paper reports 38.7 for a model trained on PubChem itself.

It's wrong. Finding out why was the most useful part of the project. 49.25% of that test set is structurally present in ChEBI-20's training data. The model had seen half the exam. Split it cleanly and the story changes:

On the molecules the model provably never trained on, it scores 28.45, which is 10.25 below the paper's in-domain number rather than above it. And the finding I care about doesn't need either reading: swapping in the wrong graph still roughly halves the score on the clean half.

I also checked the one contamination claim the whole reproduction depends on. The paper says its pretraining data excludes ChEBI-20's validation and test molecules. Compared on canonical structures, zero of 6,601 appear among the 298,010 pretraining molecules. Exact, not approximate. Their filter holds.

Why this matters beyond molecules

A multimodal model that silently lets one input override the other is a model whose failures are hard to see. If MolCA's graph encoder gets a structure wrong, the correct text sitting in the prompt won't rescue it. The model will confidently describe the wrong molecule.

The paper's ablation couldn't show this, because it averages what each view contributes during training. You only see it when you test behaviour under conflict. That's the same lesson I keep running into in evaluation work: a model can look balanced on the average and be completely lopsided in the case you actually care about.

Where this is weak

• One of my controls turned out not to be independent. I designed a mirror condition that swaps the text instead of the graph. It turns out rotating one list forward and the other backward produces the same pairing: 983 of 1,000 predictions match exactly. It's a consistency check between two code paths, not extra evidence.
• The distance curve only runs one way. I can push the SMILES further out but can't pull the graph back, because reordering broke the model. So "eight tokens are worth under a point" assumes the slope holds at close range.
• My chemical-class extractor has a ceiling. It's built on regular expressions over ChEBI's phrasing and only matches 76.3% of correct captions. The absolute rates are underestimates. The ratios between conditions hold.
• One checkpoint, no training variance. The bootstrap measures which molecules happened to be in the test set, not what happens if you retrain. The authors released one fine-tuned captioning model, so that's the one I could test.
• Captioning and retrieval only. The paper also reports property prediction and IUPAC naming, but the weights for those tasks were never released. The release, not my budget, set the boundary.

Everything is in the repo: the 14 patches, every prediction file, the analysis and bootstrap scripts, plus a verify_numbers.py that re-derives the figures no other saved output records. There's also a NOTES.md for the one question I couldn't close: whether the PubChem split in the public release is the one the paper's 38.7 was measured on. The evidence leans against it. The files needed to settle it no longer exist in public.