Huber was born in the Bronx to Jewish immigrants from Imperial Russia, who had arrived in the United States as infants. His father was the manager of an optical firm. Harold Huberman entered New York University in the Fall of 1925 at age sixteen. He was a member of the university debate team, and by his third year had become Editor of a school magazine called The Medley. His tenure at that post was marked by an incident, reported in the newspapers, when the administration suspended publication of The Medley in May 1928 for printing "low humor...not fit to bear the name of New York University".

After graduating from NYU in 1929, Huberman attended Columbia University for a short time, reportedly in the School of Law, but apparently dropped out after getting his first acting job in 1930.

On September 22, 1930, Harold Huberman became Harold Huber, for a Broadway adaption of A Farewell to Arms. This first acting job lasted a month. He also appeared in the Broadway productions The Assassin (1945), Merry-Go-Round (1932), Two Seconds (1931), and First Night (1930) before landing roles in some Warner Bros. films shot on location in New York. His face was scarred in an amateur fencing match, adding to his signature character image as heavies.

Huber made his film debut in Central Park in late 1932, followed quickly by a bit part in 20,000 Years in Sing Sing. He appeared in nearly 100 films in the 1930s and 1940s. An early noteworthy role was as the stool-pigeon Nunnheim in The Thin Man (1934). He played many roles requiring him to assume different accents, like Ito Nakamura, a Japanese American in the 1942 film Little Tokyo, U.S.A.. Among his many roles were appearances as a police officer in various Charlie Chan films, including an American in Charlie Chan on Broadway (1937), a French officer in Charlie Chan at Monte Carlo (1937) and Charlie Chan in City in Darkness (1939), and a Brazilian in Charlie Chan in Rio (1941). He played a key supporting role as a member of the French Foreign Legion in Beau Geste (1939). He also played roles in films featuring Mr. Moto and Charlie McCarthy.

Huber starred as Hercule Poirot in The Adventures of M. Hercule Poirot in a weekly half-hour program from February to October, 1945 (the program is also cited as being titled simply Hercule Poirot or Agatha Christie's Poirot). Agatha Christie introduced the initial broadcast of the series via shortwave radio. In October 1946, Huber began a year-long run on radio as Poirot in a daily fifteen-minute program on CBS, called Mystery of the Week, with scripts by Alfred Bester. Huber also portrayed Fu Manchu on radio in an eponymous program.

Huber's television debut came in 1950, as the star of a weekly half-hour drama, I Cover Times Square, on ABC. He played Johnny Warren, a nationally known newspaper and radio columnist. Huber also produced the New York-made show, which lasted only one season.

In September 1958, Huber co-starred with Eva Gabor in a short-lived off-Broadway revival of Frank Wedekind's play Lulu, his last professional credit.

Huber died during surgery at Jewish Memorial Hospital on September 29, 1959, leaving behind his wife Ethel and daughter Margaret. He was buried at Mount Hebron Cemetery in Queens.