The sky is not a screensaver. But “astronomers confirm” is a phrase worth taking apart, because in physics the word confirm has a much narrower meaning than it does in a headline.
Start with the threshold. In particle physics, the convention for announcing a discovery is five sigma — shorthand for a result so unlikely to have arisen from random fluctuation that the probability is roughly one in 3.5 million, assuming no signal is present. Three sigma, a far weaker bar, is conventionally called evidence rather than discovery. CERN explains the distinction in exactly those terms, and the reason for the enormous gap is that physicists run a very large number of analyses. When you look in enough places, improbable-looking fluctuations are guaranteed to appear somewhere.
That threshold is not a universal law of science. Different fields use different conventions for good reasons. What travels is the underlying idea: a result becomes a claim about the world only after the possibility of it being noise has been made small, and after somebody other than its authors has had a go at breaking it.
The second thing headlines flatten is the publication route. A large fraction of astronomy and physics results appear first as preprints on arXiv, posted before peer review. This is a genuinely good system — it is fast, open and citable — but a preprint is a paper that has not yet been checked by anyone obliged to check it. Coverage that describes a preprint as a study, without saying so, has removed a real caveat.
The system did exactly what it is supposed to do. It just did it in public, after the headlines, which is where the reputational damage lives.
The best illustration is not hypothetical. In March 2014 the BICEP2 collaboration announced the detection of a particular polarization pattern in the cosmic microwave background, widely reported as the first direct evidence of cosmic inflation. It was a spectacular claim, announced at a press conference, and the coverage matched. Over the following year, analysis of Planck satellite data showed that polarized dust in our own galaxy could account for the signal. A joint BICEP2/Keck Array and Planck analysis published in Physical Review Letters in 2015 concluded that the evidence for a primordial signal did not survive.
Nobody committed fraud. The team’s data were real; the foreground was underestimated. The system did exactly what it is supposed to do. It just did it in public, after the headlines, which is where the reputational damage lives.
Three questions cover most cases.
Is this a preprint or a peer-reviewed paper, and does the coverage say which? The answer is usually one click away and rarely in the article.
What is the claimed significance, and does the collaboration itself use the word evidence or the word discovery? Researchers are generally careful with that distinction even when the press release is not.
Has an independent group looked at it? A result confirmed by a different instrument, run by people with no stake in the first team’s being right, is a fundamentally different object from a result announced once.
The screensaver headline fails all three instantly, which is the point. The genuinely hard cases fail only the third, and they fail it quietly, a year later, on a Tuesday.