Erlanger was born on January 5, 1874, at San Francisco, California. His family was Jewish and his parents both immigrated from the Kingdom of Württemberg, Germany and met in California during the Gold Rush. Joseph was the sixth of seven children born to the couple. He completed his Bachelor of science in Chemistry from the University of California, Berkeley in 1895. He then completed his M.D. in 1899 from the Johns Hopkins School of Medicine in Baltimore, Maryland, where he finished second in his class.

Upon graduating, Erlanger interned at Johns Hopkins Hospital under william Osler and worked in a physiology laboratory. Erlanger also gave lectures at the school on digestion and metabolism. Erlanger also had an interest in cardiology, specifically the way that excitation transferred from the atrium to the ventricle and researched with Arthur Hirschfelder. Erlanger developed and patented a new type of sphygmomanometer that could measure blood pressure from the brachial artery. While working at Johns Hopkins in 1901, Erlanger published a paper on the digestive systems of canines. This paper caught the attention of william Henry Howell, a physiology professor at Johns Hopkins. Howell recruited Erlanger as an Assistant Professor. Erlanger was promoted to Associate Professor some time before 1906.

In 1906, Erlanger accepted a position as the first chair of physiology at the University of Wisconsin in Madison. In 1910, he left to take a position as professor at Washington University in St. Louis; the St. Louis position offered Erlanger more funding for his projects. Herbert Spencer Gasser, Erlanger's former student at Wisconsin, joined Erlanger's laboratory soon after the move. During World War I, the pair contributed to the research effort examining the effects of shock. As part of this work, Erlanger was able to produce heart block in an animal model by clamping the bundle of His and tightening it. Together, they managed to amplify the action potential of a bullfrog sciatic nerve in 1922 and published the results in the American Journal of Physiology. It is uncertain why the pair had such a sudden shift in interest to neuroscience, as Erlanger was already widely respected in the cardiology field.

He died of heart disease on December 5, 1965 at St. Louis, Missouri. The Joseph Erlanger House in St. Louis was designated a National Historic Landmark on December 8, 1976 as a building of national significance. On January 22, 2009, the International Astronomical Union named a crater on the moon after him.

Erlanger and Gasser were able to modify a Western Electric oscilloscope to run at low voltages. Prior to this modification, the only method available to measure neural activity was the electroencephalograph, which could only show large-scale electrical activity. With this Technology, they were able to observe that action potentials occurred in two phases—a spike (initial surge) followed by an after-spike (a sequence of slow changes in potential). They discovered that neurons were found in many forms, each with their own potential for excitability. With this research, the pair discovered that the velocity of action potentials was directly proportional to the diameter of the nerve fiber. The partnership ended in 1931, when Gasser accepted a position at Cornell University. In 1944, they won the Nobel Prize in Medicine or Physiology for these discoveries.