All four of Barrymore's marriages ended in divorce. His first marriage was to Actress Cara Williams in 1952; they had one child, John Blyth Barrymore (b. 1954), before their divorce in 1959. A year later, in 1960, Barrymore married Gabriella Palazzoli. Their daughter, Blyth Dolores Barrymore, was born that same year. Their marriage lasted ten years before ending in divorce in 1970.

In 1958, he changed his middle name to Drew, although he had previously been credited in past works as Blyth, and appeared in many low budget films such as High School Confidential, Never Love a Stranger (1958), Night of the Quarter Moon (1959), and The Keeler Affair (1963) as Stephen Ward. This was followed by a brief resurgence in Italian movies as he appeared in several leading roles. He also appeared several times in the TV series Gunsmoke. However, Barrymore's social behavior obstructed any professional progress. In the 1960s, he was occasionally incarcerated for drug use, public drunkenness, and spousal abuse.

In 1966, Barrymore was signed to play a guest role as Lazarus in the Star Trek episode "The Alternative Factor". However, he failed to show up (and was ultimately replaced at the last minute by actor Robert Brown), resulting in a SAG suspension of six months. He did appear as Stacey Daggart in the 1966–67 NBC series The Road West, starring Barry Sullivan.

After the SAG suspension was served to Barrymore in 1967, he sporadically worked on-screen, sometimes with a few years between appearances. His TV and film career ended permanently by 1976, although even before this point he became more and more reclusive. Barrymore suffered from the same addiction problems that had destroyed his Father, and he became a derelict. He was estranged from his family, including his children, and his lifestyle continued to worsen as his physical and mental health deteriorated.

In 2003, daughter Drew moved him near her home, despite their estrangement. She paid his medical bills until his death from cancer the following year at age 72. He has a star on the Hollywood Walk of Fame for his contributions to television.

He guest-starred in other memorable episodes of classic TV Westerns Rawhide — "Incident of The Haunted Hills" — playing a half-Native, half-White outcast and Wagon Train — "The Ruttledge Munroe Story" — playing a "too cheerful" character who spreads death wherever he goes and turns out to be a figure from Major Adams's (Ward Bond) military past.