Kaye took a year out of college in 1984 and worked on a kibbutz in Israel, where he met his Future wife Orly; they married in 1989, and have two children.

On graduation, Kaye designed theatre posters for the King's Head, the Bush Theatre and the Gate Theatre, Notting Hill. He was a scene Painter at the Old Vic Theatre in Waterloo and illustrated regularly for the NME, i-D, Literary Review, Time Out and International Musician magazines between 1987–89. He had two exhibitions of illustration and poster work between 1989–90, firstly at the Soho House Theatre, W1 and then at the Drill Hall, WC1.

Kaye formed and sang in many bands, notably the dark psychedelic outfit We Are Pleb, who played extensively on the Camden scene of 1988–89 (which spawned Blur and Suede) and had a penchant for setting the stage on fire. Kaye was signed to Go Discs in 1992 with a group called TV Eye (formed with ex-members of the band Eat), which released two singles, "Killer Fly" and "Eradicator".

After brief stints presenting two episodes of Transmission (ITV's indie music magazine programme circa 1990) as Pennis, the character next appeared in 1995 on BBC2's The Sunday Show. The basic premise was that Kaye and a camera crew would visit film premieres, press functions and other assorted Celebrity gatherings to attempt to get an "interview" with stars, in between short skits and sketches featuring the character.

In 1993, Kaye filmed a prototype Dennis Pennis, interviewing his own band on a late-night indie music show on Granada TV called Transmission. After the interview, Kaye then went out with the crew, got very drunk and offended as many people as possible up and down Oxford Street. This tape somehow arrived on the desk of producers at Planet 24 six months later, and they offered Kaye the job of knocking on people's doors at 6.00am on The Big Breakfast. Kaye turned them down, preferring to stay on the dole and stick with We Are Pleb; Mark Lamarr eventually took the job.

In 1994, Kaye convinced his old friend Anthony Hines (a car mechanic and some time roadie for TV Eye) to help him write Dennis Pennis when he was offered the job on The Sunday Show. (Hines was later poached by Sacha Baron Cohen to write for Ali G on The Eleven O'Clock Show and went on to receive an Oscar nomination for co-writing Borat in 2006).

Originally, the celebrities would be mainly British stars harassed at assorted London-based events, such as actor Hugh Grant, TV host Ulrika Jonsson and Sports pundit Des Lynam. A 1995 video release of these early clips, Anyone For Pennis, assured success for the controversial Comedian, who sought to broaden Pennis's scope.

The final video release, Dennis Pennis RIP: Too Rude to Live, released in 1997, saw the character killed off. As Kaye explained, "Dennis Pennis had become too expensive. Taking a film crew out every night with no guarantee of getting even a minute's worth of footage of me harassing celebs was just stupid. And then the programme would take months to compile, and it was just boring, man... hanging out in the rain, twiddling your thumbs, playing with your hip flask... he had to go... at the risk of sounding a bit wanky, Pennis was more of an art project in my mind, living out my fantasy of being a naughty boy. Once I'd packed it in, I had no intention of ever doing anything like it again."

In 1998 Kaye appeared in the video to the Fat Les song "Vindaloo" as a Richard Ashcroft look-alike. That year he also appeared as the character DI Lindsay De Paul in the TV comedy movie You Are Here.

Kaye also appeared as the singer of a fictional punk band called Spunk in a 1999 mock-documentary of the same name, which appeared as the 'wrath' part of a Channel 4 series on the seven deadly sins.

In 2000 Kaye starred in the comedy series Perfect World, a sitcom about a down-on-his-luck marketing manager. He also briefly presented a BBC2 quiz show, Liar, in which six contestants would all have a supposed claim to fame and the studio audience voted on which one they believed was telling the truth. In the same year Kaye took a dramatic role alongside Michelle Collins in Two Thousand Acres of Sky.

In 2004, Kaye played the leading role in the film Blackball. His role as deaf DJ Frankie Wilde in the 2005 mockumentary It's All Gone Pete Tong won him the Film Discovery Jury Award at the 2005 US Comedy Arts Festival. He also played in two episodes of the BBC drama series Waking the Dead, playing Dr. David Carney in "Shadowplay". Television appearances in 2006 and 2007 include episodes of Hustle, EastEnders and Kingdom. Kaye is now the chief interviewer on rockworld.tv, in which he interviews up-and-coming punk/indie bands.

Kaye wrote an article for The Guardian calling for peace in Israel after his mother-in-law was killed by a rocket attack there in 2008.

From November 2010 to January 2011, Kaye played Matilda's father, Mr Wormwood, in the Royal Shakespeare Company's musical Matilda, based on the classic Roald Dahl novel Matilda. Kaye reprised the role when the musical transferred to the Cambridge Theatre in London's West End in October 2011. In April 2012, Kaye was nominated for an Olivier Award for Best Supporting Actor in a Musical.

"Best supporting Actor in a Musical-Whats on Stage Awards 2012 (Matilda the Musical) "Best Supporting Actor in a Musical"-Olivier Awards 2012 (Matilda the Musical

In 2013, Kaye appeared as Thoros of Myr in the third season of the HBO series Game of Thrones, and as Danno in the BBC Radio 4 series Love in Recovery. In 2016, Kaye reprised his role as Thoros of Myr in the sixth season of Game of Thrones, and returned for the seventh season.

In 2014, Kaye played Brother Lucian in the movie Dracula Untold. In 2015 he played a Criminal in the BBC drama The Interceptor.

In 2015 he featured as the drunken, haunted Naval Officer Harry Brewer in the National Theatre's revival of Our Country's Good. He also appeared in Doctor Who as an alien funeral Director.