Parker was born Cecil Schwabe in Hastings, Sussex. He was educated at St Francis Xavier College, and at Bruges in Belgium. He served with the Royal Sussex Regiment in the First World War, reaching the rank of sergeant He began his theatrical career in London in 1922, adopting the surname "Parker" from his mother's maiden name. He made his first film appearance in 1928 and subsequently became a familiar face in British and occasionally American films until his death. He appeared less often on television, but many of his films have remained popular and are often shown.

He acted in two adaptations of A. J. Cronin's novels, The Citadel (1938) and The Stars Look Down (1940), in addition to appearing in The Lady Vanishes (1938) and Under Capricorn (1949), both of the later films were directed by Alfred Hitchcock. Other roles were in Storm in a Teacup (1937), 23 Paces to Baker Street (1956), Dangerous Moonlight (1941), Swiss Family Robinson (1960), and I Was Monty's Double (1958), as well as the comedies A French Mistress (1960), The Ladykillers (1955), The Man in the White Suit (1951), The Court Jester (1955) (in which he played an evil, usurping king of England), Indiscreet (1958) and I Believe in You (1952). Parker was also the original Charles Condomine in the West End production of Noël Coward's Blithe Spirit.

He played an evil, scheming butler on one episode of The Avengers ("The £50,000 Breakfast"). In 1957 he played Dr. Morelle in BBC radio series, "A Case for Dr. Morelle" (13 episodes).

He often played a touchy senior officer or British upper class character, and his last two films were true to form: The Magnificent Two (1967) with the British comedy double act Morecambe and Wise and Richard Attenborough's version of Oh! What a Lovely War (1969).