Ross was born on March 15, 1914 to Jewish immigrant parents in New York, New York. He dropped out of Seward Park High School at age 16 to become a singing waiter at the Van Cortlandt Inn in the Bronx. When the cafe added a female Dancer and singer, Ross was promoted to announcer. He added some jokes and became a Comedian.

In 1938, he appeared at the Queens Terrace, near Jackson Heights, New York. Jackie Gleason had already been playing there for 16 weeks, and the manager was about to ask Gleason to stay a while longer. Ross heard of the opening, auditioned for it, got the contract, and also stayed 16 weeks. Ross then turned burlesque comic on the Schuster circuit out of Chicago.

In 1955, Ross worked at a nightclub in Miami Beach called Ciro's. He was spotted by Nat Hiken and Phil Silvers, who were planning You'll Never Get Rich (later known as The Phil Silvers Show and sometimes Sgt. Bilko) and loved Ross's comedy skills. Ross was hired on the spot and cast as the mess sergeant, Rupert Ritzik.

After The Phil Silvers Show ended in 1959, Nat Hiken went on to produce Car 54, Where Are You? and cast Joe E. Ross as Patrolman Gunther Toody of New York's 53rd Precinct. Fred Gwynne, another Bilko alumnus, played Toody's partner, Francis Muldoon. Toody could usually be counted on at some point to say "Ooh! Ooh!", or "Do you mind? Do—you—mind?". Ross became so identified with his policeman role that he recorded an album of songs entitled "Love Songs from a Cop". Roulette Records released the LP in 1964. Ross did the voice for Toody for the episode "Car 54" of Hanna-Barbera's Wait Till Your Father Gets Home, in which Toody and Muldoon moonlight running a day care center and one of the children turns up missing.

Ross also starred as Gronk in Sherwood Schwartz's ill-fated 1966 sitcom It's About Time, which featured two 1960s American astronauts who were thrown back in time to the prehistoric era.

Following the breakup of Allen & Rossi in 1968, Steve Rossi teamed for less than three months with Ross in an act called "Rossi & Ross". Rossi & Ross played once on Ed Sullivan and disbanded in January 1969.

Ross also was a prominent cartoon voice into the 1970s, playing the stereotypical bumbling sergeant in many cartoons such as Hong Kong Phooey (as Sgt. Flint) and Help!... It's the Hair Bear Bunch! (as Botch). He also voiced Roll on CB Bears segment Shake, Rattle and Roll. His "Ooh! Ooh!" phrase was emulated by Frank Welker in the animated series Fangface. He was also one of the few white comedians with 1970's label Laff Records, which specialized in African-American comedians and released his album Should Lesbians Be Allowed to Play Pro-Football?.

Ross died of a heart attack on August 13, 1982 while performing in the clubhouse of his apartment building in Los Angeles, The Oakwood Apartments. He was buried in Forest Lawn-Hollywood Hills Cemetery. Jay Leno delivered the eulogy. Ross' gravestone is inscribed with the double entendre "This man had a ball".