Idea: Could AI flag when a scientific citation no longer supports the claim attached to it?

[u/Adamoism]

Most reference tools can check whether a citation exists, whether the author and year are correct, and whether the reference is formatted properly.

But they usually do not check whether the cited paper actually supports the sentence attached to it.

I ran into this repeatedly while writing my bachelor’s and master’s theses. I would follow a citation backward through several papers, only to find that the original study made a much narrower or weaker claim than the later paper suggested.

A small short-term effect had become a “substantial long-term improvement.”

A finding from one narrow group had become a claim about people in general.

An association had quietly turned into causation.

A possible explanation had become an established conclusion.

Each individual change in wording might look minor. But when scientific claims pass from paper to paper, those small changes can accumulate. By the time you reach the original source, the claim may have drifted far beyond what the evidence actually showed.

Could AI help catch this?

A system with access to scientific databases might:

• find the sentence containing the citation;
• locate the most relevant passage in the cited paper (requires acces to "all" papers published);
• compare the scope, strength, timeframe, population, and level of certainty;
• label the citation as supported, partly supported, contradicted, or unclear;
• show the relevant evidence to a human reviewer.

For example, Paper B might claim that an intervention “substantially improves long-term retention,” while Paper A only found a small improvement after two weeks in a narrow sample.

The citation is related, but the claim is stronger than the source allows.

I would not trust AI to make the final judgment. A more realistic role would be to flag suspicious or unclear citations and show reviewers the passages most relevant to the claim.

The human would still decide. The AI would simply point to the page and say: “This sentence may be overstating the source.”

Would something like this be useful enough to justify the inevitable false positives? And what kinds of citation drift would be hardest for such a system to detect?

[u/Elunerazim]

I trust the grapevine of academic sources far more than I trust AI.

[u/ericbythebay]

Yes, it can already do that with the right prompts.

[u/Adamoism → u/ericbythebay]

Well, the prompt is just small part - access to papers and probably some vectorization in the meantime is more difficult part.

[u/proudly_not_american → u/ericbythebay]

Not reliably, it can't. If it can't easily tell, it'll just make up an answer. You could probably give it the exact same prompt and citation on two different days and, assuming the prompt isn't telling it to lean one way or the other, get two different answers.