Rydberg was born 8 November 1854 in Halmstad in southern Sweden, the only child of Sven Rydberg and Maria Anderson Rydberg. When he was 4 years old his father died, and the family was forced to live on a small income. In 1873 he graduated from Halmstads elementärläroverk, where he received high grades in maths and physics. Later that year he enrolled in Lund University, and two years later he was awarded his bachelor's degree. In 1879 he was awarded his Doctor of Philosophy with his dissertation "Konstruktioner af kägelsnitt i 3- och 4-punktskontakt".

Rydberg began his career as an amanuensis in the institution. He became a docent in maths in 1880, and in 1882 became a docent in physics. At this time he began studying the standard atomic weight, because he wondered what was the reason for the seemingly random increase in weight for the atoms in Mendeleev's periodic system. He searched for a formula for several years to no avail.

His next work was about investigating the atomic spectra, explaining why these occurred. Rydberg's research was preceded by Johann Jakob Balmer's, who presented an empirical formula for the visible spectral lines of the hydrogen atom in 1885. However, Rydberg's research led him to publish a formula in 1888 which could be used to describe the spectral lines not only for hydrogen but other elements as well. After his publication in 1890 on the subject, Rydberg returned to his fruitless research on the periodic table.

Rydberg applied for the a professorship in 1897, but despite the recommendations of experts in the subject he was rejected. However, he became an extraordinary professor instead. It wasn't until 1909 that he was upgraded into a full professorship. To earn extra money he worked part-time as a numerical examiner at Sparbanken in Lund from 1891 and as an actuary in Malmö from 1905.

The physical constant known as the Rydberg constant is named after him, as is the Rydberg unit. Excited atoms with very high values of the principal quantum number, represented by n in the Rydberg formula, are called Rydberg atoms. Rydberg's anticipation that spectral studies could assist in a theoretical understanding of the atom and its chemical properties was justified in 1913 by the work of Niels Bohr (see hydrogen spectrum). An important spectroscopic constant based on a hypothetical atom of infinite mass is called the Rydberg (R) in his honour.

In 1913, Rydberg became very ill and was forced to slow down his pace of research, and in 1915 he was granted leave on account of his illness. He died on December 28, 1919 at Lund Hospital and was succeeded by his student Manne Siegbahn. Rydberg is buried at the northern cemetery in Lund and left his wife Lydia Carlsson (1856-1925), son Helge Rydberg (1887-1968) and daughter Gerda Rydberg (1891-1983).