Individual vs. collective research and the crisis in theoretical physics

The last progress made in fundamental and theoretical physics, understanding progress as successfully verified and confirmed theories or models, can be traced back to Peter Higg’s prediction in the 60s and the development of the foundations of quantum chromodynamics in the 70s. The year 1973 is claimed to be the date when progress in theoretical physics stalled (the year in which the Standard Model was standardized by adding asymptotic freedom). This long-lasting lack of progress in theoretical physics has not happened in the history of physics since at least the 18^th century.

Other areas of physics, such as experimental or applied physics, has continued to make progress since then. This is what Angela Collier refers to when trying to refute the claims that physics has done nothing in the past 50 years: she lists ten major achievements in experimental physics (such as detecting gravitational waves, taking a picture of a black hole and detecting the Higgs boson) and applied physics (such as MRIs, lasers, achieving Bose-Einstein condensates and superconductors), but all these breakthroughs rely on physics theories and models developed before 1973.

The fact that we currently do not have a theory of everything or a theory of quantum gravity is not the reason why people claim that theoretical physics has done nothing since 1973 (such as Eric Weinstein states), it’s the fact that all the fundamental questions in physics of the 70s still remain unsolved. Thus, the same problems the theoretical physics community is trying to solve today (such as dark matter, the horizon problem, grand unification, the collapse of the wave function, the Born rule, quantum interpretations, the value of particle masses in the Standard Model, the UV Landau pole, the baryon asymmetry, hierarchy problems, etc.), were the ones trying to be solved half a century ago, together with new problems that have appeared thanks to the progress of experimental physics (such as dark energy, neutrino masses, supermassive black hole formation, the Hubble tension, etc.).

The theoretical physics community has not stopped coming up with new hypotheses since the 70s (such as cosmic inflation, Hawking radiation and black hole entropy, string theory, loop quantum gravity, dark matter particles, supersymmetry, proton decay, WIMPS, dark photons, chamaleons, axions, sterile neutrinos, the cosmological constant as an energy of the vacuum, primordial black holes, MOND, interpretations of quantum mechanics, etc.) but none of them have been experimentally verified. Some of them have been falsified (at least at the energy scales they were supposed to appear, which proves that experimental physics is doing its duty), which could be though as progress being made, but no new ideas have come into reality or as established physics. This is clearly portraited in the fact that no Nobel prize in physics has been awarded for theoretical developments done after the 70s, even though there have never been so many theoretical physicists working at the same time and so many ways to accelerate research among them, such as global communication.

One could argue that we currently cannot test these new hypotheses because of a lack of precision in measurements or not been able to reach particular high energies,

but this is not true, because there are already observations performed which contradict our models without the need for more precision or higher energies and have no explanation (such as dark matter and dark energy, neutrino masses, the horizon problem, the Hubble tension, etc.). Some predictions might take more than 50 years to be verified, but this does not justify that none of the predictions in the last 50 years have been verified. Another argument which is often brought up (by the ones who at least recognize there is a crisis), is that the remaining problems in theoretical physics today are just too hard to solve: “the low hanging fruit has been picked”. But haven’t we got the greatest amount of theoretical physicists ever working together with the latest technology and global communication?

It seems like the way in which hypotheses are chosen to be researched is wrong, and indeed several of them fall into the same scheme in which a new particle or model that could solve a problem is first proposed, then it is fixed so that it hasn’t been ruled out yet by observations and it could be proved wrong (this claimed to be a prediction), and when it is not proved right, the model is modified so that a more precise experiment is required to prove it. But hasn’t history showed that the most succesful way of making predictions is by solving an unanswered question within the existing theories, for instance, resolving internal consistencies or inconsistencies with other theories?

But the question is, why is this the case? Why are there no breakthroughs in theoretical physics anymore?

Newton developed his law of gravitation in self-quarantine from the Great Plague. Einstein his theory of special relativity while working in a patent office, outside university. Schwarzschild his solution to Einstein’s field equations in the Russian front during the First World War. Schrödinger his wave equation in a remote mountain cabin on a ski holiday. Peter Higgs the prediction of the Higg’s boson after returning to his Edinburgh apartment from a failed weekend camping trip to the Highlands. John Bell’s theorem during an academical sabbatical. Many more breakthroughs in theoretical physics, such as Maxwell’s equations, Noether’s theorems, de Broglie’s wave electrons, Friedmann equations, Schrödinger’s equation, Max Planck’s constant, Heisenberg’s uncertainty principle, Pauli’s prediction of neutrinos, Dirac’s prediction of positrons, Bell’s theorems, and Weinberg’s unification of the weak and electromagnetic interactions, were single author publications with no co-authors. These cited papers might look old, but due to the ongoing crisis since 1973, they are the last ones to have been verified, and no newer ideas have been confirmed. For the few who at least admit that “that’s not how physics is done anymore”, we should ask ourselfs if whatever changed was to the better. Other research that, even though have not been verified, profoundly influenced theoretical physics, such as Hawking’s black hole radiation, Bekenstein’s entropy for black holes and the AdS/CFT correspondence, have also been single-author papers.

Why are almost all breakthroughs in fundamental and theoretical physics single author papers (or two authors maximum) when there were also publications of several authors? Is it just a coincidence?

All these authors maintained contact with other fellow physicists during their research breakthroughs, but it is a fact that their achievements can be granted to their own individual ideas, personal curiosity and desires to go beyond the established physics in the community.

Today, research in theoretical physics is mostly done in teams or groups at universities, in which they follow research lines that are chosen to be funded by others. Short term grants are given to mainstream short-result projects that reviewers and not the researchers find interesting, and often not to the true desires to research of the researchers themselves. Research institutions are driven by the necessity to acquiring funding (mostly from taxes) from researcher’s grants and scholarships, and grant holders pressure the young researchers to produce papers in which the holder’s name appear, in order to promote the holder’s career. High-risk high-reward research, which might not produce secure short term results, is avoided due to competition for funding and jobs. Funding allocation biases slows down research in fruitful ideas, and new researchers are forced to join these projects in order to continue at university. Who can decide better what to focus research on if not the researchers themselves?

Universities often set limits to individual desires to pursuit scientific knowledge by increasing pressure to publish “Publish or Perish” and get cited in order to be promoted internally, and collective thinking leads to conformity, which does not result in progress in science. Even Peter Higgs stated that “It’s difficult to imagine how I would ever have enough peace and quiet in the present sort of climate to do what I did in 1964.” and “Today I wouldn’t get an academic job. It’s as simple as that. I don’t think I would be regarded as productive enough.” referring to today’s pressure to publish. Also, other jobs, such as computer science or banking, offer better salary opportunities than research, making many graduates disregard research as a job.

Is this the reason why there has been no progress in theoretical physics since the 1970s?

One could argue that individual research is not the solution to the crisis in theoretical physics since anyone (retired or self taught, for instance) could have done research alone and solved any of the problems of the last 50 years, but this has not happened. This reasoning is incorrent, because almost all of the problems solved in the past (and cited earlier) were actually done by academia members, but working individually and independently. The problem lies in academia and professional research. Thus, the question is, how can the new academia members solve the problems the scientific community of theoretical physics faces nowadays (and have faced over the past 50 years)?

Is giving individual freedom of research in academia while improving communication of research ideas and funding the solution to the crisis in fundamental physics?

Methods, philosophy and history of science is not taught in universities. How can we know how to do physics correctly if we don’t know what has worked and what hasn’t in the past?

How can we change the way of doing physics in order to achieve different results if we don’t know what has changed in the past decades? How do researchers successfully choose which hypotheses are more promising and which research ideas to pursuit if no criteria on which of them are better is known?