The universe is a deeply vexing place. Every breakthrough we make in our understanding of it begets more mysteries about how all this (gestures wildly) actually happened. In the new book Space Oddities: The Mysterious Anomalies Challenging Our Understanding of the Universe, experimental physicist Harry Cliff describes a handful of the most confounding phenomena at play in physics. Cliff charts the path that scientists have taken to arrive at our modern understanding of how it all works.

From masses so small they function more like waves to the black holes that hide their inner workings with exceptional success, Cliff covers the most enigmatic phenomenon known to humans. He also introduces the extraordinary people seeking to break down these anomalies. Solving even one of these mysteries could unlock a new era of scientific understanding.

Below is my conversation with Cliff, lightly edited for clarity.

Isaac Schultz, Gizmodo: This book is your second, after How to Make an Apple Pie From Scratch. Why did you decide to embark on this second project? What was missing, either in your body of work or in the published sphere, as far as particle physics is concerned that needed addressing?

Harry Cliff: It really came out of my research. I work on the Large Hadron Collider. I came in right at the beginning of the Large Hadron Collider, at the end of the first decade of the 21st century. And I’ve been there ever since. Basically what happened is we discovered the Higgs boson, which is great and very exciting, and that kind of rounded off our understanding of 20th century physics in some sense. The great hope was there would be new discoveries of things that we didn’t know about before, like dark matter or supersymmetry or whatever, and none of that appeared. All these expectations were sort of not realized. But throughout high-energy physics, we were seeing these anomalies, which were hinting at the potential existence of new particles or new forces that we hadn’t imagined. That was really, really exciting.

My own research from about 2015 onwards really focused on these anomalies. It’s an interesting idea that people may be not so familiar with, because in the history of physics and our understanding of nature, the biggest breakthroughs often do come from these little weird niggling effects that you might dismiss at first, that no one really understands. They turn out to be some clue to some big new shift in how you see the world.

The book is really an attempt to both explore what’s going on in research, in cosmology and our understanding of the universe at the moment, but also set this in some kind of context and say, “the reason these things are so exciting is because in the past, they’ve led to these really big breakthroughs, and look at where this might be taking us in the future.”

Gizmodo: I speak a lot with folks who are looking for signs of dark matter. It seems like so much of the work right now is just narrowing the mass range. It’s got to be out there. Or at least we expect it to be. But the outstanding question is, “when will this happen?” The public and obviously the media would love for it to be a big “newsflash!” experience. But one thing that you touch on in the book is that science, more often than not, does not work that way.

Cliff: Usually these things emerge gradually. You get your first clues, and sometimes it takes decades or more to unravel these things. One of the examples in the book is this weird problem with the orbit of Mercury that was observed in the 19th century, where Mercury’s turning up too early, basically, for transits of the Sun. That took about a century more to figure out what was a cause of it.

It’s quite rare in science that there’s this ‘eureka!’ moment where everything becomes clear. That happens more often when you’re discovering something you expect to see. The Higgs boson was an example of that. It had been predicted 50 years earlier; you build a Large Hadron Collider to experiment, see this new bump in a graph, but they know what it is, because they’re expecting it. You can say: On the 4th of July 2012, the Higgs was discovered. When you’re really discovering something new that is outside your expectation, it takes a lot longer, because you’ve got to convince yourself of what you’re seeing, you’ve got to convince others of what you’re seeing. People are much more willing to accept things they expected and much more resistant to accept things they didn’t see coming.

One of the stories in the book is about Adam Riess, the Nobel Prize-winning cosmologist. He’s been dealing with this problem with the expansion of the universe. He’s been slogging at this now for a decade, and from his point of view, this anomaly is like gold-plated. They’ve checked every possible effect, and it seems that there really is this anomaly there. But because there isn’t a ready-made theoretical explanation for what’s causing this, the rest of the field is much more skeptical. He’s got a real job on his hands of persuading his colleagues that this is the real deal.

Gizmodo: You open and close the book with the Hubble tension. Why? What makes that the pivot point?

Cliff: It’s partly because space is just sexier than particle physics. I think it’s easier for people to engage with something that’s going on out in space, and stuff that’s going on at the subnuclear level is a little bit more abstract and hard to get your head around. It’s quite romantic to be thinking about galaxies and the expansion the universe. I deal with five big anomalies in the book. There’s five substantial chapters on stuff that’s going on at the moment.

I think of all of them, the Hubble tension is the one that I personally find the most compelling, just because it’s the one where theory is very clear about what should happen, and the experimental evidence seems very strong. It’s not just Adam Riess’ group. There are lots of groups. Every measurement, basically, that has been made of the expansion of space using stuff in the local universe—and by local we’re talking, you know, huge distances still, but galaxies and stuff that you can see—they all basically line up, more or less. There’s a few that sort of wobble about, but it seems very unlikely at this stage, after a decade of scrutiny, that there is some really big mistake that has been missed. There’s something to be understood, for sure. Now, whether that is something that’s truly revolutionary, like a rewriting of the laws of gravity or a new form of energy in the universe that we haven’t understood before, maybe telling us something about dark energy. It may be something to do with the assumptions that we have in cosmology about the idea that the universe looks the same in every direction, and that the place we are in the universe isn’t particularly special. It’s the sort of assumption that we make in order to be able to do cosmology. I think that it is the anomaly that is probably telling us something quite profound. The other four, I think, are much more difficult to say what’s going on.

If you take 100 anomalies—and anomalies come and go in physics all the time—most of them will go away. It might only be one of them that actually turns out to be the real clue. The reason I picked those particular five is because they are ones that have been around for quite a long time. We will learn something important in the process of unraveling these ones, but I think they’re less likely to turn into some big new physics discovery. Whereas I think the Hubble tension, of any of them, is going to do it. That is the one I’d put my money on.

Gizmodo: How did you choose the experiments that you would highlight and the interviews that you would do with physicists, to bring life to each of these mysteries?

Cliff: The very first bit of the prologue is a description of an experiment called ANITA, which is an incredible experiment. It’s basically a giant radio antenna launched into the Antarctic skies on this massive helium balloon. Part of the reason for choosing that story, along with the anomaly being very interesting, is just the experiment is really cool. At the beginning of writing, I was thinking, how could I get a way of wrangling a trip to Antarctica out of this? But I just realized that was not going practical or affordable. So I had to kind of go secondhand. But some of the leading people involved are in London, which is where I’m based. So that was a kind of easy first win.

But I did do a lot of traveling to the States and other places to see people for the other anomalies. I was really led more by the anomalies themselves and less by the experiments. But one of them is about my own research and about the LHCb experiment at CERN. That’s an environment I know very well. So I could describe that firsthand, whereas the others, say, Fermilab, I went there. One of the privileges, I suppose, of working on these sorts of books is you send emails off to people and say, “can I come to your under-mountain lair where you do your dark matter experiment?” And people are very open. “Oh yeah, sure. Come along and we’ll show you around.”

A lot of the environments that particle physics and astronomy experiments are done are really quite extraordinary places. An important part of getting across the science is not just the concepts and the phenomena that being studied, but these extraordinary environments where the scientific research is carried out.

Gizmodo: I sometimes think about physics in two ways, “looking up” and “looking down” science. Particle research deep underground, that would be a “looking down” experiment. Looking at the Hubble constant, studying the Cepheid stars, would be looking up. In the book, you say we live in a universe of fields more than a universe of particles, but we focus on particles because they have mass. How did you strike a balance of the “looking up” science and the “looking down” science, so to speak?

Cliff: We basically have two ways of studying the universe. One is by, as you say, looking up, and the other is by looking in. I say, maybe not looking down so much, but looking inwards. You can glean a certain amount of information from looking at the heavens, but the limiting factor is most of the universe is inconveniently far away and you can’t go. We’ve only been as far as the Moon in terms of human exploration. In terms of machines, out to the edges of the solar system now, with Voyager. But that’s a tiny, tiny fraction of the size of the universe.

It’s really through the combination of these two techniques that we’ve managed to make so much progress. One of the most revolutionary discoveries, and maybe not appreciated in these times outside of astrophysics, was the discovery of spectroscopy. The discovery that atoms of particular elements emit these characteristic wavelengths of light and absorb them. That was the absolute key to unlocking so much about the universe. That discovery was made by using elements that we have on Earth, and then allows us to say what the Sun is made from for the first time, or what the most distant star is made from. So by bringing these two things together, ultimately that is how physics makes progress. They are really just two different ways of looking at the same phenomena. And by bringing these two ideas together, that’s how you get a full picture.

Gizmodo: The high-luminosity Large Hadron Collider is on the horizon. Are you particularly excited for this next generation LHC? What do you think might come of this?

Cliff: It’s going to be really interesting. We’ve only analyzed a tiny fraction of the data that is ultimately going to be recorded by the high-luminosity LHC. In a way, this experiment has become even more crucial, because what we have learned in the last decade or so is that if there is new physics at the energy scales that we’re probing at the LHC, it’s hiding quite effectively. A high-precision machine where you get, you know, orders of magnitude more data will allow us to eke out if there are these very rare events, rare processes that are hiding in the data. That’s going to be our best chance of seeing them.

But the other thing I think a lot of colleagues are now emphasizing is what the legacy of the LHC is going to be. Even if we don’t discover any new physics at the LHC, it’s going to leave this extraordinary legacy of the understanding the basic ingredients of our universe and the laws that govern their behavior. The basic goal by the end of the 2030s, when this thing powers down for the last time, is that we will have really beautiful, precise measurements of the Standard Model. That is going to be really crucial, because when we go to the next experiment, whatever that may be, it’s that kind of groundwork that we’ve done that will allow us to see when eventually the new thing crops up. But of course, we may be lucky, and we may get the new thing in the coming year.

Gizmodo: You have a couple of anecdotes in the book about Fall of Icarus-esque errors, where entire experiments have collapsed due to misunderstanding of the numbers or taking the numbers from the wrong places. It connects with what you wrote about Fermilab’s muon G-2 experiment, where it pays to double-blind yourself from your own experiments. Otherwise the numbers are tantalizing in a way.

Cliff: Yeah, absolutely. One of the quotes that I love that I put in the book is from Feynman, which is that “the first rule is you must not fool yourself, and you are the easiest person to fool.” People are in science because they want to make discoveries. The temptation to believe when you see some effect in your experiment is huge, because everyone wants that excitement, that moment of seeing something that no one has ever seen before. I think the most important quality for experimental physicists is skepticism, and real caution. Sometimes even very, very cautious and skeptical people make mistakes. That may not be because they’ve, you know, massaged the data or done anything wrong. It’s just that there is some very subtle effect that nobody thought of.

And that does happen. In my own area of research, we had a series of anomalies that in the end turned out to be some very subtle backgrounds that we thought we had under control. But when we by chance stumbled upon some evidence that these things were actually not under control, we eventually untangled this. In other cases, it’s theory that can go wrong. Incorrect assumptions can creep in. Or even sometimes really basic, like high school errors where you accidentally put a -1 instead of a +1 or something. That actually did happen in the muon experiment you were referring to. There literally was a sign error in a calculation that made people think they were seeing evidence of new physics.

But then there are examples where people take shortcuts. That comes sometimes from this fierce desire to be first. And if you’re in competition with another experiment, you want to be the one that makes the big discovery. And that’s where the temptation to not do something completely carefully can come in, and that can be pretty disastrous if you then make some big claim that turns out to not be correct. But that is the great thing about science. It’s self-correcting. And even if something gets published that turns out to be wrong, it will get found out almost always, eventually.

Gizmodo: An example of that kind of scientific hubris is the Mercury-Vulcan issue where, as you describe in the book, this prestigious astrophysicist barges into an amateur astronomer’s home, and all of a sudden launches this erroneous discovery. As you say, it takes a century of undoing, but it gets done.

Cliff: That was a crazy one, because the discoverer of this non-existent planet got, like, France’s highest honor, for discovering something that didn’t exist.

Gizmodo: There’s that instance and another moment you describe, where a young Richard Feynman is very nervous about giving a speech in front of Paul Dirac.

Cliff: One of the reasons for bringing in the history is to set the modern experiments in context. They’re part of a long process that stretches back decades often, of experimentation, theorization. You’re kind of building all of this accumulated knowledge and then taking the next step that maybe leads to something exciting.

Gizmodo: You were doing so much traveling, speaking to folks in different fields of physics than your own for the book. What did you learn that was new to you?

Cliff: I suppose the thing I really came away appreciating is just the effort that goes into, particularly, the experiments. You have people dedicating decades of their life to measuring one number. Take the muon G-2 experiment in Fermilab as an example. Chris Polly, who is the spokesperson of the experiment, who showed me around Fermilab, he’s been working on this one number his entire career. He did his PhD on the first version of the experiment. His colleagues led the development of this new version, which involved this massive logistics project of moving this magnetic ring from New York to Chicago via the Atlantic and the Mississippi River, and then years and years and years of painstaking work, understanding every little bit of the experiment, measuring the magnetic fields to crazy precision, controlling the environment within the warehouse. And it’s only after all of this incredible care that finally, at the end of that process, you get a number. And that’s the thing you’re aiming for. I’ve got huge admiration for people like that who are willing to go through decades of slog to actually add a little bit of new knowledge to the bank of our understanding about nature.

Gizmodo: Can you tell me a bit about your work on the LHCb experiment?

Cliff: LHCb is one of the four big experiments on the Large Hadron Collider, this 27 kilometer ring where we collide particles. The B stands for beauty, which is the name of one of the six quarks in nature, also more usually known as a bottom quark. But we’d rather be known as beauty physicists than bottom physicists. Basically, when it was discovered, there was this kind of toss-up about what it was going to be called. Most people call it bottom; we call it beauty.

The reason these things are interesting is that the way they behave, the way they decay, is very sensitive to the existence of new forces or new particles that we’ve not seen before. So these are a great laboratory for searching for indirect evidence of something that we’ve not seen before. It’s a compliment to the other experiments at the LHC, where you bash stuff together and you try and create new particles. So you might look for a Higgs boson or dark matter or whatever. At LHCb it’s a different game, of precision, of measurement, and essentially trying to eke out another decimal place where you might start to see a deviation. That’s the kind of physics that we do. I’ve been on LHCb since the start of my physics career now. So, since 2008, and we’re still going strong. We’ve just had a big upgrade, and the experiment is taking data at an increasing rate. So we’re hopefully going to get more information about these anomalies in the next year or two. It’s an exciting time.

Gizmodo: What was it like writing the book alongside the work you were doing at the LHC?

Cliff: When I started writing the book, the anomalies that we were seeing at the Large Hadron Collider were looking really, really compelling and exciting, and there were quite a few results that came out that got a lot of media attention. There was this real sense that we were on the brink of something very exciting. And then, as I was writing the book at the same time, we were realizing that there was something that we’d missed. So it was kind of a salutary experience as a scientist, going through that process of thinking you’re on the brink of something and then realizing—to your horror—that there is a bug, essentially, in analysis. I didn’t want to shy away from that in the book.

I wanted to give a sense of what science is actually like. And when you’re working at the limits of understanding, you’re really taking risks. You are in real danger of making mistakes because you don’t know what you’re doing. You’re doing the best you can, but you’re on unexplored terrain, and there’s a very high risk of making mistakes. My skepticism, probably my youthful enthusiasm, may have given way to a slightly more middle-age skepticism as a result of this whole experience, which I hope will make me a better scientist in the long run.