Dickens was born in Bolt, West Virginia. He began his musical career in the late 1930s, performing on radio station WJLS in Beckley, West Virginia, while attending West Virginia University. He soon quit school to pursue a full-time music career, traveling the country performing on local radio stations under the name "Jimmy the Kid."

Dickens married Connie Chapman in 1944, the marriage ended in divorce in 1955. Later that year, he married Ernestine Jones; she died in 1968 in an automobile accident while traveling in Texas. He married Mona Evans in 1971.

In 1948, Dickens was heard performing on WKNX, a radio station in Saginaw, Michigan, by Roy Acuff, who introduced him to Art Satherly at Columbia Records and officials from the Grand Ole Opry. Dickens signed with Columbia in September and joined the Opry in August. Around this time he began using the nickname Little Jimmy Dickens, inspired by his short stature.

In 1950, Dickens formed the Country Boys with Musicians Jabbo Arrington, Grady Martin, Bob Moore, and Thumbs Carllile. It was during this time that he discovered Future Country Music Hall of Famer Marty Robbins at a Phoenix, Arizona television station while on tour with the Grand Ole Opry road show. In 1957, Dickens left the Grand Ole Opry to tour with the Philip Morris Country Music Show.

In the late 1960s, Dickens left Columbia for Decca Records before moving again to United Artists in 1971. That same year, he married his wife, Mona, and in 1975 he returned to the Grand Ole Opry. In 1983. Dickens was inducted into the Country Music Hall of Fame.

In 1962, Dickens had his first top-10 country hit since 1954 with "The Violet and the Rose".

In 1964, he became the first country Artist to circle the globe while on tour. He also made numerous appearances on television, including on The Tonight Show Starring Johnny Carson. In 1965, he released his biggest hit, "May the Bird of Paradise Fly Up Your Nose", which reached number 1 on the country chart and number 15 on the pop chart.

With the death of Hank Locklin in March 2009, Dickens became the oldest living member of the Grand Ole Opry, at the age of 90. He made regular appearances as a host at the Opry, often with the self-deprecating joke that he is also known as "Willie Nelson after taxes," playing on his resemblance to Nelson in his later years and his own short stature. At the 2011 CMA Awards, Dickens was dressed as Justin Bieber and made fun of Bieber's current paternity scandal.

Dickens was hospitalized after a stroke on December 25, 2014, days after his last appearance on the Opry to mark his birthday. He died of cardiac arrest on January 2, 2015, at the age of 94. He is survived by his wife, Mona Dickens, whom he married in 1971, and two daughters, Pamela Detert and Lisa King. After his funeral on January 8, 2015 at the Grand Ole Opry House, Dickens was entombed in the Cross Mausoleum at Woodlawn Memorial Park Cemetery in Nashville.

Dickens joined the producers Randall Franks and Alan Autry for the In the Heat of the Night cast CD Christmas Time’s A Comin’, performing "Jingle Bells" with the cast (the CD was released by Sonlite and MGM/UA and was one of the most popular Christmas releases of 1991 and 1992 with Southern retailers).