A member of the formerly royal house of Bruce, Elgin was born at the family seat, at Broomhall House, Fife, the second son of Charles Bruce, 5th Earl of Elgin and his wife Martha Whyte. He succeeded his older brother william Robert, the 6th Earl, in 1771 while he was only five. He was educated at Harrow and Westminster, and studied at St Andrews and Paris. He entered the army as an ensign in the 3rd Guards, and eventually rose to the rank of major-general.

He was elected as a Scottish Representative Peer in 1790. Although he formally remained a representative peer until 1840, shortly before his death, he took almost no part in Parliament after 1807.

In 1791, he was sent as a temporary envoy-extraordinary to Austria, while Sir Robert Keith was ill. He was then sent as envoy-extraordinary in Brussels until the conquest of the Austrian Netherlands by France. After spending time in Britain, he was sent as envoy-extraordinary to Prussia in 1795. Elgin was appointed as ambassador to The Porte in December 1798.

Elgin married twice. On 11 March 1799, he married Mary (1778–1855), only child of william Hamilton Nisbet, of Dirleton; They had a son and three daughters:

Acting on the advice of Sir william Hamilton, he procured the services of the Neapolitan Painter, Lusieri, and of several skilful draughtsmen and modellers. These artists were dispatched to Athens in the summer of 1800, and were principally employed in making drawings of the ancient monuments, though very limited facilities were given them by the authorities. About the middle of the summer of 1801, Elgin received (as is said) a firman, from the Porte which allowed his lordship's agents not only to 'fix scaffolding round the ancient Temple of the Idols [the Parthenon], and to mould the ornamental sculpture and visible figures thereon in plaster and gypsum,' but also 'to take away any pieces of stone with old inscriptions or figures thereon.' Due to the loss of the original firman, it isn't sure that the translation is correct.

Part of the Elgin collection was prepared for embarkation for England in 1803, considerable difficulties having to be encountered at every stage of its transit. Elgin's vessel, the Mentor, wrecked near Cerigo with its cargo of marbles, and it was not till after the labours of three years, and the expenditure of a large sum of money, that the marbles were successfully recovered by the divers. On Elgin's departure from Turkey in 1803, he withdrew all his artists from Athens with the exception of Lusieri, who remained to direct the excavations which were still carried on, though on a much reduced scale. Additions continued to be made to the Elgin collections, and as late as 1812, eighty fresh cases of antiquities arrived in England.

After their marriage ended in divorce Mary later married Robert Ferguson of Raith (1777–1846) who had been cited in the divorce. Elgin, on 21 September 1810, married Elizabeth (1790–1860), youngest daughter of James Townsend Oswald of Dunnikier. They had four sons and three daughters, including:

On the recommendation of a parliamentary committee, which also vindicated Elgin's conduct, the "Marbles" were bought by Great Britain in 1816 for £35,000, considerably below their cost to Elgin (estimated at £75,000), and deposited in the British Museum, where they remain.