He attended the United States Military Academy at West Point but, contrary to some sources, did not graduate. Instead he dropped out and returned to Portland and drove a taxi. Then as now, summer stock companies flocked to Maine each year, and in 1929 Flavin was asked to fill in for an actor. He did well with the part and the company manager offered him $150 per week to accompany the troupe back to New York. Flavin accepted and by the spring of 1930 was living in a rooming house at 108 W. 87th Street in Manhattan. Flavin didn't manage to crack Broadway at this time; his Broadway debut would not occur for another 39 years, in the 1971 revival of The Front Page, in which he played Murphy and briefly took over the lead role of Walter Burns from Robert Ryan.

Flavin appeared in nearly four hundred films between 1932 and 1971. He appeared in almost one hundred television episodes, including the NBC sitcom, The People's Choice, starring Jackie Cooper, several episodes as police Detective Sawyer who was being driven nuts by Gracie Allen on the Burns and Allen show,and three times as a sheriff on the western aviation adventure series, Sky King, before his final appearance, as U.S. President Dwight D. Eisenhower in Francis Gary Powers: The True Story of the U-2 Spy Incident (1976), a dramatization of the shooting down in 1960 by the former Soviet Union of the U-2 pilot, Francis Gary Powers.

Flavin portrayed Sam Cooper in the 1958 episode, "The Ed Church Case", of the CBS crime drama series, Richard Diamond, Private Detective, starring David Janssen. In 1959, he guest starred as Big Dan Girod in the episode, "Invitation to a Murder", on the ABC/WB detective series, Bourbon Street Beat, starring Andrew Duggan. In 1960, Flavin appeared in The Twilight Zone episode "A Passage for Trumpet".

From 1960 to 1962, Flavin appeared three times on the CBS sitcom, Pete and Gladys, with Harry Morgan and Cara Williams. Flavin portrayed Fire Chief Hawkins in the 1964 episode, "Beyond a Reasonable Doubt", on the NBC education drama series, Mr. Novak, starring James Franciscus.

Flavin died of a heart ailment at Cedars-Sinai Medical Center in Los Angeles, California on April 23, 1976. His widow Lucile died seventeen days later. They were survived by their son, william James Flavin, subsequently a professor at the United States Army War College. James and Lucile Browne Flavin are interred at Holy Cross Cemetery in Culver City, California.

Flavin was the son of a hotel waiter of Canadian-English descent and a mother, Katherine, whose father was an Anglo-Irish immigrant. Thus, Flavin, well known in Hollywood as an "Irish" type, was—ironically—only one-quarter Irish. He was born and reared in Portland, Maine, a fact that may have enriched his later working relationship with Director John Ford, also a Portland native.