Colman was born in Portland, Oregon. As a child actor in local productions, he became active in local radio. He attended the University of Washington and University of Michigan. In 1944, after serving in the U.S. Army during World War II, he moved to New York, where he began a long career in Broadway theater, first appearing in a production of Irwin Shaw's war drama The Assassin. He was soon invited to join Maurice Evans' acting company, where he continued to act on stage, and later in films.

He appeared dozens of times on prime-time television dramas and comedies. In 1964, he appeared in two episodes of Perry Mason, one as a Doctor in police investigations and the other as a prosecuting attorney. Other televisions appearances include My Three Sons (1962), Frasier, Gilligan's Island, McCloud and The Monkees. Films include Them! (1954), The Silver Chalice (1954), Moonfleet (1955), My Gun Is Quick (1957), Wild on the Beach (1965), Maryjane (1968), The Lawyer (1970), Scandalous John (1971), Time Travelers (1976), Norma Rae (1979) and Intolerable Cruelty (2003). In 1983, he portrayed the kindly scientist, Professor Hector Jerrold, in the ABC daytime melodrama General Hospital.

In 1974, Colman played the role of Dr. Zaius in the popular short-lived TV series, Planet of the Apes; the role made famous on the big screen by his former Teacher, Maurice Evans. In the six episodes in which he appeared, he wore the original costume Evans used in Planet of the Apes and Beneath the Planet of the Apes.

Colman played Scrooge over 600 times on stage in A Christmas Carol at the Meadow Brook Theatre in the Detroit area. He died in his sleep in Los Angeles at the age of 91 on December 15, 2014.