He made his film debut in the comedy Ants in His Pants in 1938, as an extra, produced by Ken G. Hall. At that time, he was managing a wine cellar in Bond Street, Sydney. Rafferty caught the acting bug and got another unbilled role, as one of several inept firemen in Hall's Dad Rudd, M.P. (1940).

The Oxford Companion to Australian Film refers to Rafferty as "Australia's most prominent and significant actor of the 1940s–60s".

Rafferty married Ellen Kathleen "Quentin" Jameson on 28 May 1941. He enlisted in the Royal Australian Air Force the next day and entertained troops.

During the war, Rafferty was allowed to make films on leave. He appeared in a short featurette, South West Pacific (1943), directed by Hall. He was reunited with Chauvel and Grant Taylor in The Rats of Tobruk (1944), an attempt to repeat the success of Forty Thousand Horsemen.

Rafferty was discharged on 13 February 1945, having reached the rank of Flying Officer.

Ealing Studios were interested in making a feature film in Australia after the war, and assigned Harry Watt to find a subject. He came up with The Overlanders (1946), a story of a cattle drive during war time (based on a true story) and gave the lead role to Rafferty who Watt called an "Australian Gary Cooper." Rafferty's fee was £25 a week. Ealing were so pleased they signed Rafferty to a long term contract even before the film was released. The film was a massive critical and commercial success and Rafferty was established as a film star.

Ealing were associated with Rank Films, who Rafferty in the lead of Bush Christmas (1947), a children's movie where Rafferty played the villain. It was very popular.

Rafferty was meant to follow this with a comedy for Ealing co-starring Tommy Trinder. Instead, Ealing put the two actors in a drama about aboriginal land rights Bitter Springs (1950). The film was not widely popular and Ealing wound up their filmmaking operation in Australia.

Rafferty kept busy as an actor, appearing on radio in a show Chips: Story of an Outback. He was cast by 20th Century Fox in a melodrama they shot in Australia, Kangaroo (1952). The studio liked his performance enough that they flew him (and Charles Tingwell) over to Los Angeles to play Australian Soldiers in The Desert Rats (1953), a war movie.

Their first movie was The Phantom Stockman (1953), directed by Robinson and starring Rafferty, and produced by them both. The film was profitable. It was followed by King of the Coral Sea, which was even more popular, and introduce Rod Taylor to cinema audiences. Rafferty and Robinson attracted the interest of the French, collaborated with them on the New Guinea adventure tale, Walk Into Paradise (1956). This was their most popular movie to date.

Rafferty also appeared as an actor only in a British-financed comedy set in Australia, Smiley (1956). It was successful and led to a sequel, Smiley Gets a Gun (1958),in which Rafferty reprised his role. In England he appeared in The Flaming Sword (1958).

He also participated in cinema advertisements that were part of an Australian Government campaign in 1957 called "Bring out a Briton". The campaign was launched in a bid to increase the number of British migrants settling in Australia.

Rafferty and Robinson raised money for three more movies with Robinson. He elected not to appear in the fourth film he produced with Robinson, Dust in the Sun (1958), their first flop together. Nor was he in The Stowaway (1959) and The Restless and the Damned (1960). All three films lost money and Rafferty found himself in financial difficulty.

Rafferty returned to being an actor only. He had a small role in The Sundowners (1960), with Robert Mitchum and Deborah Kerr and played a coastwatcher in The Wackiest Ship in the Army (1960) with Jack Lemmon and Ricky Nelson. He was in the Australian-shot TV series Whiplash (1961).

In 1962, the 6 foot 5 inch actor was socialising with fellow expatriates in a London club when one, who unbeknownst to Rafferty was a Wrestler, claimed he was being ignored and started an argument. Rafferty was eventually provoked into accepting a challenge to 'step outside', where he was badly beaten. In addition to deep grazes to his face that that may have cost him the chance of roles in two major film productions the incident brought on a heart attack.

Rafferty collapsed and died of a heart attack while walking down a Sydney street at the age of 62 shortly after completing his role in Wake in Fright. His wife Quentin predeceased him in 1964 and they had no children. His remains were cremated. His ashes were scattered into his favourite fishing hole in Lovett Bay.

He travelled to the US and guest starred in episodes of d The Wackiest Ship in the Army (1965) (as a different character to the role that he played in the movie version). He played a Union soldier in The Big Valley (1066_ with a noticeably Australian accent. He was also in episodes of Gunsmoke (1966) and Daktari (1966).

Back in Australia Rafferty had a good part in the Australian-shot comedy They're a Weird Mob (1966) a big local success. He returned to Hollywood to appear in episodes of The Girl from UNCLE (1967), Tarzan (1967) and The Monkees, as well as the Elvis Presley movie Double Trouble (1967) and the adventure tale Kona Coast (1968)

Back in Australia he guest-starred in Skippy the Bush Kangaroo, Adventures of the Seaspray (1967), Rita and Wally (1968), Woobinda, Animal Doctor (1970) and Dead Men Running (1971). He continued to make films such as Skullduggery (1970).

In the 1971 New Years' Honours, Rafferty was made a Member of the Order of British Empire (MBE) by Queen Elizabeth II for his services to the performing arts.

Australia Post issued a stamp in 1989 that depicted Rafferty in recognition of his work in Australian cinema, and in March 2006, Broken Hill City Council announced that the town's Entertainment Centre would be named in honour of Rafferty.

He was then cast as one of the mutineers in MGM's Mutiny on the Bounty, with Marlon Brando. The filming of Bounty dragged on – meant to take six months in Tahiti, it would end up taking 14. However, the money earned by Rafferty – he dubbed the film The Bounteous Mutiny – restored him to financial health after the failure of his production company; it enabled him to buy a block of flats which supported him for the rest of his life.