Sheffield was born John Matthew Sheffield Cassan in Pasadena, California, the second child of actor Reginald Sheffield and Louise Van Loon (January 21, 1905 – April 14, 1987). His older sister was Mary Alice Sheffield Cassan and his younger brother was william Hart Sheffield Cassan (actor Billy Sheffield). His father was himself a former Juvenile performer when he came to the United States from his native England. His mother, a native of New York City, was a Vassar College graduate with a liberal arts education who loved books and lectured widely.

In 1938, Sheffield became a child star after he was cast in the Juvenile lead of a West Coast production of the highly successful Broadway play On Borrowed Time, which starred Dudley Digges and featured Victor Moore as Gramps. Sheffield played the role of Pud, a long role for a child. He later went to New York as a replacement and performed the role on Broadway.

The following year, his father read an article in The Hollywood Reporter that asked, "Have you a Tarzan Jr. in your backyard?" He believed he did and set up an interview. MGM was searching for a suitable youngster to play the adopted son of Tarzan in its next jungle movie with stars Johnny Weissmuller and Maureen O'Sullivan. When he was 5 years old, Sheffield was taken to an audition where Weissmuller chose him over more than 300 Juvenile actors interviewed for the part of "Boy" in Tarzan Finds a Son (1939). In that same year, Sheffield appeared in the Busby Berkeley movie musical Babes in Arms with Mickey Rooney and Judy Garland, classmates of his at the studio school.

After he outgrew the role of Boy, the teenaged Sheffield went on to star in his own jungle movie series for Allied Artists. In 1949, he made Bomba, the Jungle Boy with co-star Peggy Ann Garner. In all, he appeared as Bomba 12 times, more than any other character he portrayed. Sheffield appeared in his last movie, as Bomba, in 1955.

Sheffield decided to leave the industry and enrolled in college to further his education. He lived and worked for a time in Arizona. John and Patricia Sheffield were married in 1959 in Yuma, Arizona. They had three children: Patrick, Stewart and Regina.

Sheffield's wife, Patty, said that he fell from a ladder while pruning a palm tree. Though his injuries seemed minor, he died of a heart attack four hours later on October 15, 2010, in Chula Vista, California, aged 79.