In the early 1950s, he moved from his native Mississippi to Los Angeles, where he found work in local radio productions. His big break came when he was selected to join the Bishop's Company, a repertory theatre company which toured the United States. As the only African-American member of the group, he was given the opportunity to play a wide range of roles, which was rare for black actors at the time. One of his best known roles with the company was as "Scratch" in "The Devil and Daniel Webster." Later, he would become a leading actor with the Ebony Showcase Theatre in Los Angeles. He would also direct several productions for the theatre. Regarding his staging of The Odd Couple, Los Angeles Times theatre critic Margaret Harford wrote "James Wheaton's slick direction makes the overall production all one could ask." His role as "J.B. Mongoose" in the Ebony Showcase production of "Carnival Island" was so popular, that he was soon appearing on television.

During the late 1960s and 1970s, Wheaton appeared on some of the most popular television programs of the day including Room 222, Kojak, Good Times, and the final episode of Ironside. He had a recurring role as Nelson B. Davis, "The Friendly Undertaker", on Sanford and Son. Other feature film credits included Black Belt Jones and lastly, Gun Crazy with Drew Barrymore. He also reteamed with Bill Cosby in A Piece of the Action, which also starred and was directed by Sidney Poitier.

In the early 1970s he taught at Figueroa Street Elementary School and Parmelee Avenue Elementary School in Los Angeles.