When young, Kaye played rugby league for Moldgreen ARLFC before studying at King James's Grammar School, Almondbury, Huddersfield. He worked in hospital radio in Huddersfield (interviewing the Beatles in 1965), and worked in textile mills, a wine factory, and a tractor factory.

He made his TV debut as a railway guard in the BBC's Yorkshire mill drama Champion House (1968) and played small roles in such films as The Party's Over (1965), starring Oliver Reed. Having been seen by Pat Phoenix in Little Malcolm by David Halliwell at Bolton, he was cast as Bernard Butler, the nephew of Elsie Tanner (Phoenix), in the soap opera Coronation Street in 1969. He later made an impression on producer/writer David Croft following guest roles in It Ain't Half Hot Mum and Come Back Mrs. Noah.

Kaye had a small part in Terry Gilliam's film Brazil as desk clerk M.O.I. Lobby Porter and appeared in Gilliam's 1977 film Jabberwocky as Sister Jessica.

He appeared in the 1978 comedy short The Waterloo Bridge Handicap, starring Leonard Rossiter, and featured as Dimes in the feature film version of Porridge (1979) alongside Ronnie Barker. He also appeared in the drama series All Creatures Great and Small and in the private detective series Shoestring. In 1981, Kaye appeared as Frank Broadhurst in the children's drama serial Codename Icarus.

In 1982, David Croft sent Kaye the script for the pilot episode of 'Allo 'Allo! inviting him to play the central character of René Artois. He accepted and appeared in all 84 episodes (the main series ran from 1984, two years after the pilot, until 1992) and 1,200 performances of the stage version.

He played Dr Grant in a television adaptation of Mansfield Park and Lymoges, Duke of Austria in the 1984 BBC production of King John by Shakespeare. He also toured in the National Theatre production of As You Like It, as Touchstone.Also in 1984 he appeared as hard man Sammy, an enforcer employed by agoraphobic bookmaker Albert Wendle in the Minder episode Get Daley!

Kaye was the subject of This Is Your Life in 1986 when he was surprised by Eamonn Andrews at the curtain call of the West End stage version of 'Allo 'Allo! at the Prince of Wales Theatre.

Kaye's autobiography, René and Me: An Autobiography (co-written with Hilary Bonner), was published in 1989. In the book, he described his experiences as a shy, gay, overweight, typecast youth. The unusual spelling of his name (usually spelt "Gordon") was the result of a British Actors' Equity Association typing error.

Kaye suffered serious head injuries in a car accident during the Burns' Day storm on 25 January 1990. Although he could not remember any details of the incident, he retained a scar on his forehead from a piece of wooden advertising boarding that had smashed through the car windscreen. Writing in his memoirs, 'Allo 'Allo! co-writer Jeremy Lloyd said he visited Kaye in hospital, adding, "I believe part of his recovery was due to his agent getting a video and showing reruns of 'Allo 'Allo! to remind him who he was." While recovering in hospital from emergency brain surgery to treat injuries sustained in the accident, Kaye was photographed and interviewed by Sunday Sport Journalist Roger Ordish. He sued the newspaper, but the Court of Appeal held, in Kaye v Robertson, that there was no remedy in English law for an invasion of privacy.

Kaye returned as Rene Artois in a 2007 one-off television revival of 'Allo 'Allo! and in a stage show in Brisbane, Australia, at the Twelfth Night Theatre in June and July, alongside Sue Hodge as Mimi Labonq and Guy Siner as Lieutenant Gruber. The other characters were portrayed by Australian actors, including Katy Manning, Steven Tandy, Chloe Dallimore, and Jason Gann.

Kaye's funeral Service was held at Huddersfield Parish Church on 17 February 2017. His co-stars from 'Allo 'Allo! Vicki Michelle, Sue Hodge, and Kim Hartman all attended, as did Ken Morley from Coronation Street, who gave a tribute. The unofficial Yorkshire anthem "On Ilkla Moor Baht 'at" was sung at the Service.