Karle was born in New York City, on June 18, 1918, the son of Sadie Helen (Kun) and Louis Karfunkle. He was born into a Jewish family with a strong interest in the arts. He had played piano as a youth and had participated in a number of competitions, but he was far more interested in science. He attended Abraham Lincoln High School in Brooklyn, and would later join Arthur Kornberg (awarded the Nobel in Medicine in 1959) and Paul Berg (a winner in Chemistry in 1980), as graduates of the school to win Nobel Prizes. As a youth, Karle enjoyed handball, ice skating, touch football and swimming in the nearby Atlantic Ocean.

He started college at the age of 15 and received his bachelor's degree from the City College of New York in 1937, where he took additional courses in biology, chemistry and math in addition to the required curriculum there. He earned a master's degree from Harvard University in 1938, having majored in biology.

He enrolled at the University of Michigan in 1940 and met his Future wife, Isabella Lugoski, who was sitting at an adjoining desk during his first course in physical chemistry. The two married in 1942. They were both supervised in their Ph.D. studies by physical Chemist Lawrence Brockway. Though Karle completed his studies in 1943, he was awarded his Ph.D. the following year.

Starting in 1943, after completing graduate studies, Karle worked on the Manhattan Project at the University of Chicago with his wife Dr. Isabella Karle, one of the youngest Scientists and few women on the project. In 1944, they returned to the University of Michigan, where Karle worked on a project for the United States Naval Research Laboratory. In 1946, they moved to Washington, D.C. to work for the Naval Research Laboratory.

Jerome Karle was a former President of both the American Crystallographic Association (ACA) (1972) and the IUCr (1981-1984), as well as a co-recipient of the 1985 Nobel Prize in Chemistry for his work on direct methods. Among the many additional honors he received for his work, he was elected to the National Academy of Sciences in 1976 and the American Philosophical Society in 1990.

Karle and Herbert A. Hauptman were awarded the Nobel Prize in Chemistry in 1985 for their work in using X-ray scattering techniques to determine the structure of crystals, a technique that is used to study the biological, chemical, metallurgical and physical characteristics. They were able to employ the Sayre equation in centrosymmetric structure, developing the so-called direct methods. Through isolating the position of the atoms in a crystal, the molecular structure of the material being studied can be determined, allowing processes to be designed to duplicate the molecules being studied. This technique has played a major role in the development of new pharmaceutical products and other synthesized materials.

Karle and his wife retired from the U.S. Naval Research Laboratory on July 31, 2009, after a combined 127 years of Service to the United States Government, with Karle joining the NRL in 1944 and his wife two years later. At the time of his departure from government Service, Karle held the Chair of Science as Chief Scientist of the Laboratory for the Structure of Matter. Retirement ceremonies for the Karles were attended by United States Secretary of the Navy Ray Mabus, who presented the couple with the Department of the Navy Distinguished Civilian Service Award, the Navy's highest form of recognition to civilian employees.