Graziano heard from a couple of his friends about a tournament going on with a gold medal for the winner. He entered under the name of Joe Giuliani and was trained by Tobias (Toby) Zaccaria of Kings County (Brooklyn), NY. He fought four matches and ended up winning the New York Metropolitan Amateur Athletic Union Boxing Competition (1939). He sold the gold medal for $15 and decided that boxing was a good way to make cash.

Graziano married Norma Unger of German-Jewish descent, on August 10, 1943. They remained together until his death from cardiopulmonary failure on May 22, 1990 in New York City at age 71. They had two children. Graziano's funeral was held at St. Patrick's Cathedral. He is interred at the Locust Valley Cemetery.

In March 1945, at Madison Square Garden, Graziano scored a major upset over Billy Arnold, whose style was similar to that of Sugar Ray Robinson; he was a slick boxer with lightning-fast combinations and a knockout punch. The Ring magazine and various newspapers across the United States touted Arnold as the next Joe Louis or Sugar Ray Robinson. Arnold was a heavy favorite to defeat Graziano and then to go on to fight for the world title, but Graziano absorbed a beating in the early going, before going on to batter and knock Arnold out in the third round of the scheduled eight-round bout. Following his defeat to Graziano, Arnold was never the same.

In 1946, Graziano was suspended by the New York State Athletic Commission (NYSAC) for failure to report a bribe attempt. In 1948, Abe Green, then-National Boxing Association's President, announced that they were indefinitely suspending him in all parts of the world under NBA supervision, following similar action by the California State Athletic Commission. This was due to his "running out" on a scheduled December 1 bout with Fred Apostoli. The suspension covered all of the American States, Great Britain, the European Boxing Federation, Cuba, Mexico, and Canada. Boxing promoter Ralph Tribuani got him a license to box in Delaware, which led to his reinstatement by both the NBA and NYSAC and Rocky's return to prosperity.

In the 1960s, Graziano opened a pizza restaurant, Rocky Graziano's Pizza Ring, on Second Avenue in Kips Bay, Manhattan, creating a modest franchise for the restaurant in the New York City area. He became the Celebrity spokesman for Lee Myles Transmissions in the New York City area, appearing on dozens of television commercials from the mid-1970s to the mid-1980s.

After his retirement from boxing, Graziano cohosted a short-lived series, The Henny and Rocky Show with famous Comedian Henny Youngman. He was a semi-regular on The Martha Raye Show, as Raye's boyfriend. He appeared as a regular on the United Artists TV series Miami Undercover for its entire run, and appeared in several series and shows, including The Pat Boone Chevy Showroom, Car 54, Where Are You?, and Naked City. He portrayed Packy, an ex-boxer, in the 1967 film Tony Rome.

Graziano was the son of Ida Scinto and Nicola Barbella. Barbella, nicknamed Fighting Nick Bob, was a boxer with a brief fighting record. Born in Brooklyn, Graziano later moved to an Italian enclave centered on East 10th Street, between First Avenue and Avenue A in Manhattan's East Village. He grew up as a street fighter and learned to look after himself before he could read or write. He spent years in reform school, jail, and Catholic protectories. Barbella, who got occasional work as a longshoreman, kept boxing gloves around the house and encouraged Graziano and his brothers to fight one another. When he was three years old, Barbella would make him and his brother, Joe (three years his senior), fight almost every night in boxing gloves. At age 18 he won the Metropolitan A.A.U. welterweight championship. Despite the fame and money that professional fighting seemed to offer, he didn't want to become a serious prize fighter. He didn't like the discipline of training any more than he liked the discipline of school or the Army.