Castellano gained worldwide fame for his role in Lovers and Other Strangers (1970), for which he was nominated for an Academy Award. He achieved further stardom in 1972 for playing the part of Peter Clemenza, in The Godfather. The Godfather became the highest-grossing film up to that time. Castellano, along with several other cast members, became widely known from the popular film. He spoke one of the film's most famous lines, "Leave the gun; take the cannoli," which he partially ad-libbed.

Castellano also appeared on television, playing the lead roles of Joe Girelli in the television situation comedy The Super (10 episodes in 1972). His real-life daughter Margaret Castellano portrayed his character's daughter Joanne. He also portrayed the lead Joe Vitale in Joe and Sons (1975-1976).

Castellano did not reprise his role as Clemenza in The Godfather Part II (1974). He was reportedly excluded because Castellano and his agent insisted on having control over the character's dialogue. Director Francis Ford Coppola said that this was untenable, and wrote Castellano out of the movie. This account was disputed by Castellano's widow in a 1991 letter to People magazine. Castellano said he did not have a part in the sequel because he did not believe that the character of Clemenza would become a traitor. He had other disagreements with Coppola, including confusion over how much weight he was expected to lose for the role. Bruno Kirby portrayed Clemenza as a young man in The Godfather Part II. He had played the son of Castellano's character in The Super.

Castellano died from a heart attack in 1988 at age 55.