Wilmer made his theatre stage debut in 1945 in repertory at Rugby. He appeared frequently on the London stage, mainly in classical and Shakespearean roles. He made his first major film appearance in Laurence Olivier's Richard III (1955); thereafter, he appeared in a large number of films, mostly in supporting roles. They include several epic films: as M. Desmoulins in The Battle of the River Plate (1956), as Al-Mu'tamin in El Cid (1961), Cleopatra (1963), The Fall of the Roman Empire (1964), as Khalifa Abdullah in Khartoum (1966), as Maj. Gen. Francis de Guingand in Patton (1970), as Sir Thomas Fairfax in Cromwell (1970), and Antony and Cleopatra (1972). Other appearances include Jason and the Argonauts (1963) as Pelias, the Pink Panther films A Shot in the Dark (1964) and Revenge of the Pink Panther (1978), The Vampire Lovers (1970), The Golden Voyage of Sinbad (1974), and Octopussy (1983).

He is mainly associated with the role of Sherlock Holmes, which he first played in the BBC's 1964 production of "The Speckled Band". Together with co-star Nigel Stock, who played Doctor Watson, Wilmer was brought back for a further twelve episodes of the Sherlock Holmes series. In 1973, Wilmer played author Jacques Futrelle's Holmesian detective Professor Van Dusen in The Rivals of Sherlock Holmes for ITV. In 1975, he once again appeared as Holmes (albeit in a supporting role) in Gene Wilder's The Adventure of Sherlock Holmes' Smarter Brother, with Thorley Walters as Dr. Watson. Wilmer also played Sir Denis Nayland Smith in two of Harry Alan Towers' Fu Manchu films, The Brides of Fu Manchu (1966) and The Vengeance of Fu Manchu (1967).

He recorded a series of the stories on audiocassette for Penguin audio books and has appeared as a guest at several UK and US events, the most recent being the Society's Golden Jubilee Dinner in January 2001. His other television credits include: The Adventures of Robin Hood, The Saint, The Troubleshooters, The Avengers, The Baron, UFO, and Space: 1999. He made a cameo appearance in "The Reichenbach Fall" episode of Sherlock as an irate old man in the Diogenes Club.

Wilmer was an honorary member of the Sherlock Holmes Society of London, which considered Wilmer "the definitive Holmes". On 24 March 2009 Wilmer was guest of honour at a launch party for his book, held at the National Liberal Club in Whitehall Place, London.