Coleman was born in Boston, Massachusetts, in 1947. His father, Joe Coleman (Joseph Patrick Coleman), was a pitcher in Major League Baseball between 1942 and 1955. Coleman attended Natick High School in Natick, Massachusetts. In high school, Coleman was considered "an overpowering pitcher" who was "watched enviously by major league scouts." He also attended Ted Williams' baseball camp in Lakeville, Massachusetts, while attending high school. Williams reportedly taught Coleman how to throw a curve ball at his camp, and why it curved.

After a 12-game stint with the Burlington Senators in the Carolina League, Coleman made his Major League Baseball debut at age 18, less than four months after graduating from high school, on September 28, 1965, throwing a four-hitter against the Kansas City Athletics and pitched two complete game victories in the final week of the 1965 season. He compiled a 2–0 record and a 1.00 ERA, and was the youngest player in the American League at the time of his big league debut. After the 1965 season concluded, Coleman and his father opened a store in Natick, Massachusetts known as Joe Coleman & Son Sporting Goods.

Coleman spent most of the 1966 season with the York White Roses in the Eastern League, compiling a 7–19 record in 32 games. In 1967, Coleman won a spot in the Senators' starting rotation, appeared in 28 games, and compiled an 8–9 record with a 4.63 ERA. He remained in the Senators' starting rotation for an additional three years, compiling records of 12–16 with a 3.27 ERA in 1968, 12–13 with a 3.27 ERA in 1969, and 8–12 with a 3.58 ERA in 1970.

On October 9, 1970, Coleman was part of a multi-player deal that sent Coleman, Ed Brinkman, Jim Hannan and Aurelio Rodriguez to the Detroit Tigers in exchange for Denny McLain, Elliott Maddox, Norm McRae and Don Wert. On March 27, 1971, Coleman was knocked unconscious by a line drive hit by Ted Simmons of the St. Louis Cardinals during a spring training game. Coleman sustained a linear skull fracture, was hospitalized for two weeks, and remained on the disabled list for 21 days before joining the Tigers in May 1971. Coleman had a 7–6 record on July 4, 1972, but he finished the season with a 13–3 record from that point forward. During the 1971 season, Coleman compiled a 20–9 record and a 3.15 ERA while recording 16 complete games. His 236 strikeouts in 1971 was a career high and ranked third in the American League behind Mickey Lolich and Vida Blue.

Coleman averaged more than 20 wins per season from 1971 to 1973. He recorded 236 strikeouts in 1971, third best in the American League, was selected to the American League All-Star team in 1972, and recorded 23 wins in 1973, second highest in the American League.

Coleman had another strong season in 1972, compiling a 19–14 record with a 2.80 ERA and 222 strikeouts in 280 innings pitched. He was selected to play for the American League in the 1972 All-Star Game, though he did not appear in the game due to injury. Coleman's father appeared in the 1948 All-Star Game, making the Colemans the first father and son to be chosen to pitch in an All-Star Game. Coleman also helped the Tigers win the American League Eastern Division. In Game 3 of the 1972 American League Championship Series, with the Tigers facing elimination after losing Games 1 and 2, Coleman threw a complete game shutout against the Oakland Athletics and set an ALCS record with 14 strikeouts in a single game. (Coleman's ALCS record broke a record of 12 set by Jim Palmer in 1972, was later tied by Mike Boddicker in 1983 and was broken by Mike Mussina in 1997.)

Prior to the start of the 1973 season, Coleman went on a diet and lost 25 pounds, dropping from 226 to 201 pounds. The trimmer Coleman won the first four starts of the 1973 season. Despite a down season for the Tigers team, Coleman posted a career-high 23 wins, second in the American League behind Wilbur Wood's 24 wins.

On April 11, 1974, Coleman lost a bid for a no-hitter in the eighth inning when Gene Michael of the New York Yankees hit a one-out single. By May 15, 1974, Coleman had a 6–1 record and appeared to be on track for the best season of his career. However, he did not win another game for two months, and his performance dropped off markedly. He finished the 1974 season with a 14–12 record and a 4.32 ERA in 41 starts. He walked an astounding 158 batters in 1974 (a Tigers single-season record) for a career high rate of 5.0/9 IP. At the end of the 1974 season, Coleman said: "It was really strange. My arm felt better than it ever had before in my life. But I couldn't do it. It was just one of those crazy things. At times I didn't have any idea what I was doing out there."

In 1975, Coleman's performance declined further, as he compiled a 10–18 record with a 5.55 ERA. In 1976, Coleman started 12 games for Detroit in the first two months of the season, posting a 2–5 record with a 4.86 ERA.

On June 8, 1976, the Chicago Cubs acquired Coleman by waiver purchase from the Tigers. The Cubs used Coleman principally as a relief pitcher. He appeared in 39 games for the Cubs, all but four in relief, and compiled a 2–8 record with a 4.10 ERA.

Coleman appeared in 43 games for the Athletics in 1977, including 12 games as a starting pitcher. He compiled a 4–4 record with a much-improved 2.96 ERA. He was one of only 10 American League pitchers in 1977 to compile over 100 innings pitched and maintain an ERA under 3.00.

Prior to the 1978 season, Oakland owner Finley again sought to discard Coleman, asking him to report to the Athletics' minor league club in Vancouver. Coleman refused and compiled a 3–0 record and a 0.00 ERA in his first eight appearances of the 1978 season. Coleman had a 1.37 ERA when he was sold by Oakland to the Toronto Blue Jays for the waiver price of $10,000. Finley reportedly "blew his top" after Coleman gave up a home run to Chicago's Lamar Johnson. Bobby Winkles resigned as Oakland's manager after Finley's impulsive decision to sell the club's most effective relief pitcher. Days later, Finley offered to buy Coleman back from Toronto for double the price Toronto had paid. Blue Jays' President, Peter Bavasi, declined Finley's proposal, suspecting that Finley had found another team prepared to pay more for Coleman. Coleman noted at the time that he was happy to escape Finley. After his bid to reacquire Coleman was rejected, Finley claimed he had only wanted to reacquire Coleman to trade him to Pittsburgh and lashed out at Coleman: "He's a very fine old man, but all he does is throw that slop up there. Maybe he can do all right. He doesn't throw hard enough now to break a pane of glass. He was spending his spare time trying to teach our young guys to pitch. How can he do that when he can't pitch himself?"

Coleman's major league career ended with the Pirates in 1979. However, he continued to play in the Pacific Coast League for the Spokane Indians from 1980 to 1982. Pitching principally in relief, Coleman compiled a 4–2 record in 40 games for Spokane. He also served as Spokane's catcher for one game in 1980, allowing all three base-runners to steal second base. After Coleman's catching debut, club manager Rene Lachemann joked, "He (Joe) did an adequate job of catching, but not of throwing." During his time with Spokane, Coleman also served as the club's pitching coach and helped coach the University of Washington baseball team.

Coleman served as a coach in various organizations, beginning with his Service as pitching coach for the Spokane Indians. The Spokane club was an affiliate of the Seattle Mariners in 1980 and 1981, but became an affiliate of the California Angels in 1982. In 1983, the Spokane club disbanded, and the Angels gave Coleman a choice of taking over as pitching coach in Edmonton or becoming the manager of the Peoria Suns, the Angels' newly organized Single A affiliate. Coleman chose the managerial position in Peoria. Coleman joked about the foibles of his young players after arriving in Peoria:

In 1982, Coleman met his Future wife, Donna, after moving to Fort Myers, Florida. Their son, Casey Coleman, was born in Fort Myers in 1987 and pitched in Major League Baseball for the Chicago Cubs from 2010 to 2012 and for the Kansas City Royals in 2014.

Coleman continued to serve in the Angels' organization for several years after his stint in Peoria. In 1987, he was the organization's roving minor-league pitching instructor, and in 1988 he joined the Angels' major league staff as bullpen coach.

In 1991, Coleman left the Angels to become the pitching coach for the St. Louis Cardinals under manager Joe Torre. He served in that position through the 1994 season. In October 1994, the Cardinals opted not to renew Coleman's contract in a front office shakeup that also saw the arrival of a new general manager and the departure of Bucky Dent as the club's third-base coach.

In 1996, Coleman returned to the Angels' organization. He served initially as a scout and then took over as the club's pitching coach for the last two months of the 1996 season. He was named bullpen coach for the 1997 season. He continued to serve as the Angels' bullpen coach through the 2000 season. In 2001, Coleman was hired as the pitching coach for the Durham Bulls, the Triple A affiliate of the Tampa Bay Rays. In 2007, Coleman left Durham and became the pitching coach of the Lakeland Flying Tigers, the Class A affiliate of the Detroit Tigers. He remained in that position until September 2011. Since 2012, Coleman has been the pitching coach for the Jupiter Hammerheads, the Class A affiliate of the Florida Marlins. The 2014 season marked Coleman's 50th year working in professional baseball as a player, scout, coach, instructor or manager.

Shortly after arriving in Toronto, Coleman met with club President, Peter Bavasi. Despite having rejected Finley's bid to reacquire Coleman, Bavasi informed Coleman that he did not fit into the Blue Jays' plans. Coleman recalled, "It was a shock to me. I didn't know why I was here. He didn't want to get into a contract situation with me that would take up a spot of one of the younger kids." Despite not fitting into the Blue Jays' plans, Coleman compiled a 2–0 record in 31 relief appearances for the club. He finished the full 1978 season with an unbeaten 5–0 record.