Fleet began his career in the RSC, appearing in several plays in the early 1980s. He has since appeared in touring productions of, among others, Habeas Corpus and In the Club, as well as in Festen and Mary Stuart and others in the West End. He also played Alderman Fitzwarren in Dick Whittington in 2002.

Earlier in his career, Fleet was seen in a 1983 episode of Grange Hill as a Teacher at the eponymous school's upmarket rival Rodney Bennett. In 1999 he starred in the sitcom Brotherly Love

In 1999–2001 he was the voice of "Dog" in the children's TV show Dog and Duck. Probably his most famous role is that of Hugo in The Vicar of Dibley; he appeared in all 20 episodes, broadcast between 1994 and 2007. In 2004 he appeared in an episode of Monarch of the Glen. In 2005 he played a leading role in an episode of the long-running ITV murder mystery series Midsomer Murders. In 2007 he was a guest star in one episode of the sitcom Legit. He appeared as Frederick Dorrit in the BBC's 2008 production of Little Dorrit. When Fleet appeared on the quiz show School's Out, it was revealed that one of his teachers at Banff Academy had written in his school report that "[James] is the stupidest boy I have ever had to teach, out of all the stupid boys I have ever had to teach", and that he was the only student in his sixth form not to have been made a prefect. Despite his apparent lack of scholastic ability, he still won the show. In 2009, Fleet appeared in a cameo role in the third series of Skins.

Between 2000 and 2006, Fleet played the painfully upright and decent Captain Brimshaw in Revolting People, a BBC Radio 4 comedy set in pre-revolutionary America. He also appeared in the radio legal sitcom Chambers, which later moved onto television. As of 2005, he has starred as Duncan Stonebridge MP in the topical radio sitcom The Party Line. He also appeared as the Captain on the BBC Radio 7 series The Spaceship. He also plays the part of Sir John Woodstock in the BBC Radio 4 sitcom The Castle and Inspector Lestrade in the first, third and fourth series of The Rivals. Fleet played John Aubrey in the 2008 BBC Radio 4 Woman's Hour production by Nick Warburton of "Aubrey's Brief Lives".

In 2003 he played Kulygin in Anton Chekhov's Three Sisters alongside Kristin Scott Thomas who played Masha.

Fleet has starred in numerous films. He played the role of Lefevre in the 2004 film adaptation of Phantom of the Opera, John Dashwood in 1995's Sense and Sensibility and that of Lytton Strachey in the 2003 film Al Sur de Granada (South from Granada). In 2014 he played John Constable in the Mike Leigh film 'Mr Turner'.

In 2009 he portrayed Sir Andrew Aguecheek in the RSC Production of Twelfth Night. In 2011 he was in Richard Bean's The Heretic directed by Jeremy Herrin at the Royal Court Theatre in London. Starting in November 2011 he was in the original line-up of The Ladykillers as Major Courtney at the Gielgud Theatre.

Fleet appeared in Coronation Street in 2010. He played a character called Robbie Sloan, a recently released convict, helping escaped prisoner Tony Gordon plot revenge on his ex-wife Carla Connor. They intend to kidnap and kill her. Sloan lures Connor into her Underworld factory, and holds her at gunpoint. Leaving her tied to a chair with her mouth taped shut, Sloan also lures Hayley Cropper into the hostage situation. Sloan was eventually shot by Gordon during a siege at the factory.

In February 2011, Fleet appeared as George (senior), the father of werewolf George Sands, in Being Human.

In December 2013 the BBC aired one of their major dramas for the Christmas season, Death Comes to Pemberley, a three-part British television drama based on characters created by Jane Austen in her novel Pride and Prejudice. The first episode was broadcast on BBC One at 8.15 pm on Boxing Day 2013. It is based on the best selling novel by P. D. James, in which the characters of Pride and Prejudice are involved in a new story involving a murder. Fleet played the part of Mr Bennett in the series. In September 2014, Fleet appeared in the BBC Three sitcom Bad Education as Richard, an ex-boyfriend of Rosie Gulliver. He has most recently appeared in an episode of ITVs second series of 'Plebs,' as Stylax's racing patron. In 2013, Fleet was engaged to play Scottish Historian, the Reverend Dr. Reginald Wakefield, in seasons 1 and 2 of the Award winning Starz adaptation of Diana Gabaldon's "Outlander", broadcast in 2014–2016.