He competed in his first meet in 1954 as a freshman at the University of California, Los Angeles (UCLA). His progress in the event was impressive; he broke the world record in his fourth competition. He pledged Pi Lambda Phi fraternity, America's first nondiscriminatory fraternity, and was class President at UCLA. In 1955, in Mexico City, he won the title at the Pan American Games.

Johnson qualified for both the decathlon and the long jump events for the 1956 Summer Olympics in Melbourne. However, he was hampered by an injury and forfeited his place in the long jump. Despite this handicap, he managed to take second place in the decathlon behind compatriot Milt Campbell. It would turn out to be his last defeat in the event.

Due to injury, Johnson missed the 1957 and 1959 seasons (the latter due to a car accident), but he broke the world record in 1958 and 1960. The crown to his career came at the 1960 Summer Olympics in Rome. His most serious rival was Yang Chuan-Kwang (C. K. Yang) of Taiwan. Yang also studied at UCLA; the two trained together under UCLA track coach Elvin C. "Ducky" Drake and had become friends. In the decathlon, the lead swung back and forth between them. Finally, after nine events, Johnson led Yang by a small margin, but Yang was known to be better in the final event, the 1500 m. Johnson however managed to stay close enough to Yang to win the gold. With this victory, Johnson ended his athletic career.

Johnson was selected by the Los Angeles Rams in the 28th round (333rd overall) of the 1959 NFL Draft as a running back.

While training for the 1960 Olympics, his friend Kirk Douglas told him about a part in Spartacus that Douglas thought might make him a star: the Ethiopian gladiator Draba, who refuses to kill Spartacus (played by Douglas) after defeating him in a duel. Johnson read for and got the role, but was forced to turn it down because the Amateur Athletic Union told him it would make him a professional and therefore ineligible for the Olympics. The role eventually went to another UCLA great, Woody Strode. In 1960, he began acting in motion pictures and working as a Sportscaster. In the 1963–1964 season, he appeared on an episode of ABCs drama about college life, Channing, starring Jason Evers and Henry Jones. Johnson made several film appearances including the James Bond film Licence to Kill as a DEA agent. After his acting career, he worked full-time as a Sportscaster in the early 1970s. He weekend Sports anchored on the local Los Angeles NBC news, but seemed uncomfortable in that position and eventually moved on to other things.

After the Olympics he turned his Celebrity into acting, sportscasting and public Service, and was instrumental in creating the California Special Olympics. His acting career included appearances in The Sins of Rachel Cade (1961), the Elvis Presley film Wild in the Country (1961), Pirates of Tortuga (1961), None but the Brave (1965), two Tarzan films with Mike Henry, The Last Grenade (1970),The Six Million Dollar Man, Roots: The Next Generations (1979), and the 1989 James Bond film Licence to Kill, opposite Timothy Dalton.

Rafer Johnson is the spokesperson for Hershey's Track & Field Games and is very involved in Special Olympics Southern California (www.sosc.org). After attending the first Special Olympics competition in Chicago in 1968, conducted by Special Olympics founder, Eunice Kennedy Shriver, he was inspired to become involved. Johnson, along with a small group of volunteers, then founded California Special Olympics in 1969 by conducting a competition at the Los Angeles Memorial Coliseum for 900 individuals with intellectual disabilities. Following the first California Games in 1969, Johnson became one of the original members of the Board of Directors. The board worked together to raise funds and offer a modest program of swimming and track and field. In 1983, Rafer ran for President of the Board to increase Board participation, reorganize the staff to most effectively use each person's talents and expand fundraising efforts. He was elected President and served in the capacity until 1992, when he was named Chairman of the Board of Governors.

Johnson's brother Jimmy is a member of the Pro Football Hall of Fame and his daughter Jennifer competed in beach volleyball at the 2000 Olympic Games in Sydney following her collegiate career at UCLA. His son Joshua Johnson followed his father into track and field and had a podium finish in the javelin throw at the USA Outdoor Track and Field Championships.

Johnson was named Sports Illustrated's Sportsman of the Year in 1958 and won the James E. Sullivan Award as the top amateur athlete in the United States in 1960, breaking that award's color barrier. He was chosen to ignite the Olympic Flame during the opening ceremonies of the 1984 Summer Olympics in Los Angeles. In 1994, he was elected into the first class of the World Sports Humanitarian Hall of Fame. In 1998, he was named one of ESPN's 100 Greatest North American Athletes of the 20th Century. In 2006, the NCAA named him one of the 100 Most Influential Student Athletes of the past 100 years. On August 25, 2009, Governor Schwarzenegger and Maria Shriver announced that Johnson would be one of 13 California Hall of Fame inductees in The California Museum's yearlong exhibit. The induction ceremony was on December 1, 2009, in Sacramento, California. Johnson is a member of The Pigskin Club of Washington, D.C. National Intercollegiate All-American Football Players Honor Roll. Rafer Johnson Junior High School in Kingsburg, California is named after Johnson, as are Rafer Johnson Community Day School and Rafer Johnson Children's Center, both in Bakersfield, California. This last school, which has classes for special education students from the ages of birth-5, also puts on an annual Rafer Johnson Day. Every year He speaks at the event and cheers on hundreds of students with special needs as they participate in a variety of track and field events. In 2010, Johnson received the Fernando Award for Civic Accomplishment from the Fernando Foundation and in 2011, he was inducted into the Bakersfield City School District Hall of Fame. Additionally, Rafer now acts as the athletic advisor to Dan Guerrero, Director of Athletics at UCLA. He was Inducted into the Texas Track and Field Coaches Hall of Fame, Class of 2016.

In November 2014, Johnson received the Athletes in Excellence Award from The Foundation for Global Sports Development, in recognition of his community Service efforts and work with youth.

On January 15, 2015, Johnson sat for a 30-minute interview where he discussed details regarding his tackling of Robert F. Kennedy's Assassin Sirhan Sirhan at the Ambassador Hotel in June 1968.