McGiver was born in Manhattan, New York City, the son of Irish immigrants. He graduated from the Jesuit-run Regis High School in Manhattan in 1932.

He earned a B.A. in English from Fordham University in 1938 and master's degrees from Columbia University and Catholic University. He became an English Teacher and worked as an actor and Director in New York's Irish Repertory Theater. He interrupted those activities and enlisted in the U.S. Army in 1942 and served as an officer in the U.S. Army's 7th Armored Division in Europe during World War II. Returning to civilian life, he continued to teach English and speech at Christopher Columbus High School in the Bronx and worked occasionally in off-Broadway plays until 1955, when he became a full-time actor.

McGiver was married to Ruth Schmigelsky from 1947 until his death; they had ten children: Boris, Brigit, Maria, Terry, Basil, Clare, Oliver, Ian, Clemens and Cornelia. Son Boris McGiver is an actor and one of the stars of television's House of Cards.

In 1959, McGiver appeared in the episode "The Assassin" of NBC's espionage drama Five Fingers, starring David Hedison. In 1962, he appeared as Gramps in the episode "The Seventh Day of Creation" of the NBC medical drama about psychiatry, The Eleventh Hour, starring Wendell Corey and Jack Ging. He appeared in the Alfred Hitchcock Presents episodes "Six People No Music" and "Fatal Figures", and the Twilight Zone episode "Sounds and Silences". In 1971 he guest-starred in Alias Smith and Jones (season 1, episode 8, 'A Fistful of Diamonds'). In 1966, he appeared in Man's Favorite Sport?, which starred Rock Hudson and Paula Prentiss. Between 1963 and 1964, McGiver appeared in five episodes of The Patty Duke Show as J.R. Castle, who was Martin Lane's boss at the fictional newspaper The Chronicle.

The owl-faced, portly character actor with his mid-Atlantic accent and precise diction, was often cast as pompous Englishmen and other stuffy, aristocratic and bureaucratic types. He was known for his performances in such films as Breakfast at Tiffany's (1961); The Manchurian Candidate (1962); and Who's Minding the Store? (1963). He appeared on many television shows and commercials during the 1960s and early 1970s, including the very first of a long running popular series of commercials for the American Express charge card ("Do you know me?").

In the 1964–1965 television season, McGiver played the widower Walter Burnley, the head of the complaint department of a fictitious Los Angeles department store in the CBS sitcom Many Happy Returns. His costars included Elinor Donahue, Mark Goddard, Mickey Manners, and Elena Verdugo. He was also in an episode of Gilligan's Island in 1966, "The Man With a Net". He also made several guest appearances on ABC's hit fantasy sitcom Bewitched over the course of its eight seasons (1964-72).

McGiver died of a heart attack on September 9, 1975 at his home in West Fulton, New York at age 61. His remains were cremated.