Rózsa sought to study music in Germany. He enrolled at the University of Leipzig in 1925, ostensibly to study chemistry at the behest of his father. Determined to become a Composer, he transferred to the Leipzig Conservatory the following year. There he studied composition with Hermann Grabner, a former student of Max Reger. He also studied choral music with (and later assisted) Karl Straube at the Thomaskirche, where Johann Sebastian Bach had once been the organist. Rózsa emerged from these years with a deep respect for the German musical tradition, which would always temper the Hungarian nationalism of his musical style.

Rózsa's first two published works, the String Trio, Op. 1, and the Piano Quintet, Op. 2, were issued in Leipzig by Breitkopf & Härtel. In 1929 he received his diplomas cum laude. For a time he remained in Leipzig as Grabner's assistant, but at the suggestion of the French organist and Composer Marcel Dupré, he moved to Paris in 1932.

Rózsa achieved early success in Europe with his orchestral Theme, Variations, and Finale (Op. 13) of 1933 and became prominent in the film industry from such early scores as The Four Feathers (1939) and The Thief of Bagdad (1940). The latter project brought him to America when production was transferred from wartime Britain, and Rózsa remained in the United States, becoming an American citizen in 1946. His notable Hollywood career earned him considerable fame, earning 17 Academy Award nominations including winning for Spellbound (1945), A Double Life (1947), and Ben-Hur (1959), while his concert works were championed by such major artists as Jascha Heifetz, Gregor Piatigorsky, and János Starker.

Rózsa's first major success was the orchestral Theme, Variations, and Finale, Op. 13, introduced in Duisburg, Germany, in 1934 and soon taken up by Charles Munch, Karl Böhm, Bruno Walter, Hans Swarowsky, and other leading conductors. It was first played in the United States by the Chicago Symphony Orchestra under Hans Lange on October 28 and 29, 1937, and achieved wide exposure through a 1943 New York Philharmonic concert broadcast when Leonard Bernstein made his famous conducting debut.

However, it was not until Rózsa moved to London that he was hired to compose his first film score for Knight Without Armour (1937), produced by his fellow Hungarian Alexander Korda. After his next score, for Thunder in the City (1937), he joined the staff of Korda's London Films, and scored the studio's epic The Four Feathers (1939). Korda and the studio's music Director, Muir Mathieson, brought Rózsa onto their Arabian Nights fantasy The Thief of Bagdad (1940) when the operetta-style approach of the original Composer, Oscar Straus, was deemed unsuitable. Production was transferred to Hollywood when the war broke out, and Rozsa completed his score there in 1940. The film earned him his first Academy Award nomination. A further two followed with Lydia (1940) and Sundown (1941). In 1943, he received his fourth nomination for Korda's Jungle Book (1942)

In 1943, Rózsa scored his first of several collaborations with Director Billy Wilder starting with Five Graves to Cairo, the same year that he also scored the similarly themed Humphrey Bogart film Sahara. In 1944, his scores for his second Wilder collaboration, Double Indemnity, and for The Woman of the Town, earned him separate Academy Award nominations in the same year. However, Max Steiner won the Oscar for that year for Since You Went Away.

In 1945, Rózsa was hired to compose the score for Alfred Hitchcock's film Spellbound, after Bernard Herrmann became unavailable due to other commitments. The score, notable for pioneering the use of the theremin, was immensely successful and earned him his first Oscar. However, Hitchcock disliked the score, saying it "got in the way of his direction". Two of his other scores from that year, The Lost Weekend and A Song to Remember, were also nominated, making Rózsa, to date, the only Composer to have won against two of his own scores. Rózsa, who also reportedly hated the interruptions and interference by Producer David O Selznick, never worked for either Hitchcock or Selznick again.

Rózsa earned another Oscar nomination for scoring The Killers (1946) which introduced Burt Lancaster to film audiences. Part of the famed theme for the Dragnet radio and TV show duplicated part of Rozsa's The Killers main theme, and he successfully sued for damages, and subsequently was given co-credit for the Dragnet theme. Rózsa received his second Oscar in 1947 for A Double Life, which also won Ronald Colman an Academy Award for Best Actor. That same year Rózsa and Eugene Zador orchestrated music by Nikolai Rimsky-Korsakov for the film Song of Scheherazade, about a fictional episode in the composer's life. Rózsa also wrote original music for the film. Among the other films scored by Rózsa during the 1940s were the prison drama Brute Force (1947), also with Lancaster, and The Naked City (1948), the latter also including music by Frank Skinner. Both of those films were directed by Jules Dassin.

Madame Bovary (1949) was Rózsa's first important score for Metro–Goldwyn–Mayer, which produced most of the later films that he scored. Other popular scores that he composed for MGM pictures include Quo Vadis (1951), Ivanhoe (1952), Plymouth Adventure (1952), Knights of the Round Table (1953), Green Fire (1954). Moonfleet (1955), Bhowani Junction (1956), Lust for Life (1956), Ben-Hur (1959), King of Kings (1961) and The V.I.P.s (1963). For Ben-Hur, he received his third and final Oscar. His final two nominations (one each for Best Original Score and Best Original Song) were for the 1961 Samuel Bronston film El Cid.

By 1952, his film score work was proving so successful that he was able to negotiate a clause in his contract with MGM that gave him three months each year away from the film studio so that he could focus on concert music.

Rózsa's Violin Concerto, Op. 24, was composed in 1953–54 for the Violinist Jascha Heifetz, who collaborated with the Composer in fine-tuning it. Rózsa later adapted portions of this work for the score of Billy Wilder's The Private Life of Sherlock Holmes (1970), the plot of which, Wilder has said, was inspired by Rózsa's concerto.

His theme, "The Duchess of Brighton" from his 1961 film The V.I.P.s was used as the intro of the Belgian children's TV series Johan en de Alverman (1965-1966).

Rózsa's Cello Concerto, Op. 32 was written much later (1967–68) at the request of the Cellist János Starker, who premiered the work in Berlin in 1969.

In 1968, Rózsa was asked to score The Green Berets, after Elmer Bernstein turned it down due to his political beliefs. Rózsa initially declined the offer, saying, "I don't do westerns." However, he agreed to compose the score after being informed, "It's not a Western, it's an 'Eastern'." He produced a strong and varied score, which included a night club vocal by a Vietnamese singer Bạch Yến. However, one cue which incorporated stanzas of "Onward Christian Soldiers" was deleted from the film's final edit.

His popular film scores during the 1970s included his last two Billy Wilder collaborations The Private Life of Sherlock Holmes (1970) and Fedora (1978), the Ray Harryhausen fantasy sequel The Golden Voyage of Sinbad (1973), the latter-day film noir Last Embrace starring Roy Scheider, and the time-travel fantasy film Time After Time (1979) for which Rózsa won a Science Fiction Film Award, saying in his televised acceptance speech that of all the film scores he had ever composed, it was the one he had worked on the hardest.

After completing work on the music for the spy thriller Eye of the Needle (1981), Rózsa's last film score was for the black-and-white Steve Martin film Dead Men Don't Wear Plaid (1982), a comic homage to the film noir films of the 1940s, a genre to which Rozsa himself had contributed scores. Although Rózsa's career as a Composer for films ended following a stroke he suffered while on holiday in Italy later that year, he continued to compose various concert pieces thereafter; one of his last works being Sonata for Ondes Martenot, op. 45 (1989). He returned to California at the behest of his son, and remained sequestered at his home for the remainder of his life.

Rozsa died on 27 July 1995 and is buried at Forest Lawn in the Hollywood Hills. His wife Margaret died in 1999 aged 89.

The sixth variation (after the theme) of his "Theme, Variations And Finale," Op. 13, was used as part of the Soundtrack in four episodes—most notably "The Clown Who Cried"—of the 1950s television series "The Adventures of Superman".