He made over 55 film and TV appearances beginning in 1959, with his career peaking in the 1960s. He became known for his performances in gangster or Western films in the 1960s where he typically portrayed a bandit or mercenary. His dark features, similar to those of his Italian co-stars meant that he could play Latin characters, such as Mexicans.

His most noted Western performance was his role as Esteban Rojo in Sergio Leone's 1964 production A Fistful of Dollars alongside Clint Eastwood and Gian Maria Volontè. He appeared in many other Westerns in the 1960s such as Blood at Sundown (1966) although he appeared in the romantic adventure Angelique and the Sultan (fr) in 1968.

During the 1970s, his film career diminished. However, he portrayed several TV characters afterwards, notably in Tatort as Investigator Kressin between 1971 and 1973. From 1985 he performed at the Theater in der Josefstadt in Vienna on stage. He retired from film and television acting in 1995. Rupp lived in Vienna towards the end of his life.