Rubbia studied physics at the University of Pisa and Scuola Normale in Pisa. He graduated on cosmic ray experiments in 1957 with Marcello Conversi. Rubbia obtained his Italian doctoral degree (Laurea) in 1958 from the University of Pisa.

In 1960 he moved back to Europe, attracted by the newly founded CERN, where he worked on experiments on the structure of weak interactions. CERN had just commissioned a new type of accelerator, the Intersecting Storage Rings, using counter-rotating beams of protons colliding against each other. Rubbia and his collaborators conducted experiments there, again studying the weak force. The main results in this field were the observation of the structure in the elastic scattering process and the first observation of the charmed baryons. These experiments were crucial in order to perfect the techniques needed later for the discovery of more exotic particles in a different type of particle collider.

In 1970 Rubbia was appointed Higgins Professor of Physics at Harvard University, where he spent one semester per year for 18 years, while continuing his research activities at CERN. In 1989, he was appointed Director-General of the CERN Laboratory. During his mandate, in 1993, "CERN agreed to allow anybody to use the Web protocol and code free of charge … without any royalty or other constraint"

In 1976, he suggested adapting CERN's Super Proton Synchrotron (SPS) to collide protons and antiprotons in the same ring — the Proton-Antiproton Collider. Using Simon van der Meers Technology of stochastic cooling, the Antiproton Accumulator was also built. The collider started running in 1981 and, in early 1983, an international team of more than 100 physicists headed by Rubbia and known as the UA1 Collaboration, detected the intermediate vector bosons, the W and Z bosons, which had become a cornerstone of modern theories of elementary particle physics long before this direct observation. They carry the weak force that causes radioactive decay in the atomic nucleus and controls the combustion of the Sun, just as photons, massless particles of light, carry the electromagnetic force which causes most physical and biochemical reactions. The weak force also plays a fundamental role in the nucleosynthesis of the elements, as studied in theories of stars evolution. These particles have a mass almost 100 times greater than the proton. In 1984 Carlo Rubbia and Simon van der Meer were awarded the Nobel Prize "for their decisive contributions to the large project, which led to the discovery of the field particles W and Z, communicators of weak interaction"

Together, these discoveries provide strong evidence that theoretical physicists are on the right track in their efforts to describe Nature at its most basic level through the so-called "Standard Model". The data on the intermediate vector bosons confirm the predictions included in the "electroweak" theory, which gained the 1979 Nobel Prize for Physics to Steven Weinberg, Sheldon Glashow and Abdus Salam. The "electroweak" theory attempts to unite two of the four forces of nature—the weak and the electromagnetic forces—under the same set of equations. It provides the basis for work on the long-standing dream of the theoretical physicists, a unified field theory, encompassing also the strong force which binds together the atomic nucleus, and ultimately, gravity.

In addition to the observation of the intermediate vector mesons, the CERN Proton-Antiproton Collider dominated the scene of high Energy physics from its first operation in 1981 until its close in 1991, when the Tevatron at Fermilab took over this role. An entirely new phenomenology of high Energy collisions has resulted, in which strong interaction phenomena are dominated by the exchange of the quanta of the strong force, the gluons, particles which are similar to the intermediate vector bosons, although, like the photons, they are apparently massless. Instead, the W and Z particles are among the heaviest particles so far produced in a particle accelerator.

In December 1984, Rubbia was nominated Cavaliere di Gran Croce OMRI.

Rubbia's research activities are concentrated on the Problem of Energy supply for the Future, with particular focus on the development of new technologies for renewable Energy sources. During his term as President of ENEA (1999–2005) he has promoted a novel method for concentrating solar power at high temperatures for Energy production, known as the Archimede Project, which is being developed by industry for commercial use.

Carlo Rubbia was principal Scientific Adviser of CIEMAT (Spain), a member of the high-level Advisory Group on Climate Change set up by EU's President Barroso in 2007 and of the Board of Trustees at the IMDEA Energy Institute. In 2009–2010 he was Special Adviser for Energy to the Secretary General of ECLAC, the United Nations Economic Commission for Latin America, based in Santiago (Chile). In June 2010 Carlo Rubbia has been appointed Scientific Director of the Institute for Advanced Sustainability Studies in Potsdam (Germany).

On 30 August 2013, Rubbia was appointed to the Senate of Italy as a Senator for Life by President Giorgio Napolitano.