Harpo was born on November 23, 1888 in Manhattan. He grew up in a neighborhood now known as Carnegie Hill on the Upper East Side of Manhattan, on East 93rd Street off Lexington Avenue. The turn-of-the-century tenement that Harpo later called (in his memoir Harpo Speaks!) "the first real home I can remember" was populated with European immigrants, mostly artisans—which even included a glass blower. Just across the street were the oldest brownstones in the area, owned by people like David L. Loew and william Orth.

In January 1910, Harpo joined two of his brothers, Julius (later "Groucho") and Milton (later "Gummo"), to form "The Three Nightingales", later changed to simply "The Marx Brothers". Multiple stories—most unsubstantiated—exist to explain Harpo's evolution as the "silent" character in the brothers' act. In his memoir, Groucho wrote that Harpo simply wasn't very good at memorizing dialogue, and thus was ideal for the role of the "dunce who couldn't speak", a Common character in vaudeville acts of the time.

Harpo had changed his name from Adolph to Arthur by 1911. This was due primarily to his dislike for the name Adolph (as a child, he was routinely called "Ahdie" instead). The name change may have also happened because of the similarity between Harpo's name and Adolph Marks, a prominent show Business attorney in Chicago. Urban legends stating that the name change came about during World War I due to anti-German sentiment in the US, or during World War II because of the stigma that Adolf Hitler imposed on the name, are groundless.

His first screen appearance was in the film Humor Risk (1921), with his brothers, although according to Groucho, it was only screened once and then lost. Four years later, Harpo appeared without his brothers in Too Many Kisses, four years before the brothers' first released film, The Cocoanuts (1929). In Too Many Kisses, Harpo spoke the only line he would ever speak on-camera in a movie: "You sure you can't move?" (said to the film's tied-up hero before punching him). Fittingly, it was a silent movie, and the audience saw only his lips move and the line on a title card.

Harpo became known for prop-laden sight gags, in particular the seemingly infinite number of odd things stored in his topcoat's oversized pockets. In the film Horse Feathers (1932), Groucho, referring to an impossible situation, tells Harpo that he cannot "burn the candle at both ends." Harpo immediately produces from within his coat pocket a lit candle burning at both ends. In the same film, a homeless man on the street asks Harpo for money for a cup of coffee, and he subsequently produces a steaming cup, complete with saucer, from inside his coat. In Duck Soup, he produces a lit blowtorch to light a cigar. As author Joe Adamson put in his book, Groucho, Harpo, Chico and Sometimes Zeppo, "The President of the college has been shouted down by a mute."

In 1933, following U.S. diplomatic recognition of the Soviet Union, he spent six weeks in Moscow as a performer and goodwill ambassador. His tour was a huge success. Harpo's name was transliterated into Russian, using the Cyrillic alphabet, as ХАРПО МАРКС, and was billed as such during his Soviet Union appearances. Harpo, having no knowledge of Russian, pronounced it as 'Exapno Mapcase'. At that time Harpo and the Soviet Foreign Minister Maxim Litvinov became friends and even performed a routine on stage together. During this time he served as a secret courier; delivering communiques to and from the US embassy in Moscow at the request of Ambassador william Christian Bullitt, Jr., smuggling the messages in and out of Russia by taping a sealed envelope to his leg beneath his trousers, an event described in David Fromkin's 1995 book In the Time of the Americans. In Harpo Speaks, Marx describes his relief at making it out of the Soviet Union, recalling how "I pulled up my pants, ripped off the tape, unwound the straps, handed over the dispatches from Ambassador Bullitt, and gave my leg its first scratch in ten days."

Harpo was also caricatured in Sock-A-Bye Baby (1934), an early episode of the Popeye cartoon series created by Fleischer Studios. Harpo is playing the harp, and wakes up Popeye's baby, and then Popeye kills him. (After Popeye hits him, a halo appears over his head and he floats to the sky.)

Harpo married Actress Susan Fleming on September 28, 1936. The wedding became public knowledge after President Franklin D. Roosevelt sent the couple a telegram of congratulations the following month. Harpo's marriage, like Gummo's, was lifelong. (Groucho was divorced three times, Zeppo twice, Chico once.) The couple adopted four children: Bill, Alex, Jimmy, and Minnie. When he was asked by George Burns in 1948 how many children he planned to adopt, he answered, "I’d like to adopt as many children as I have windows in my house. So when I leave for work, I want a kid in every window, waving goodbye."

Harpo recorded an album of harp music for RCA Victor (Harp by Harpo, 1952) and two for Mercury Records (Harpo in Hi-Fi, 1957; Harpo at Work, 1958).

In 1955, Harpo made an appearance on the sitcom I Love Lucy, in which he and Lucille Ball re-enacted the famous mirror scene from the Marx Brothers movie Duck Soup (1933). In this scene, they are both supposed to be Harpo, not Groucho; he stays the same and she is dressed as him. About this time, he also appeared on NBC's The Martha Raye Show.

Harpo and Chico played a television anthology episode of General Electric Theater entitled "The Incredible Jewelry Robbery" entirely in pantomime in 1959, with a brief surprise appearance by Groucho at the end.

Harpo made television appearances in the 1960s. In 1960, he appeared with Ernest Truex in an episode of The DuPont Show with June Allyson entitled "A Silent Panic". He played a deaf-mute who, as a "mechanical man" in a department store window, witnessed a gangland murder. In 1961, he made guest appearances on The Today Show, Play Your Hunch, Candid Camera, I've Got a Secret, Here's Hollywood, Art Linkletter's House Party, Groucho's quiz show You Bet Your Life, The Ed Sullivan Show, and Your Surprise Package.

In 1961 Harpo published his autobiography, Harpo Speaks. Because he never spoke a word in character, many believed he actually was mute. In fact, radio and TV news recordings of his voice can be found on the Internet, in documentaries, and on bonus materials of Marx Brothers DVDs. A reporter who interviewed him in the early 1930s wrote that "he [Harpo] ... had a deep and distinguished voice, like a professional announcer", and like his brothers, spoke with a New York accent his entire life. According to those who personally knew him, Harpo's voice was much deeper than Groucho's, but it also sounded very similar to Chico's. His son, Bill, recalled that in private Harpo had a very deep and mature soft-spoken voice, but that he was "not verbose" like the other Marx brothers; Harpo preferred listening and learning from others.

Harpo's two final television appearances came less than a month apart in late 1962. He portrayed a guardian angel on CBS's The Red Skelton Show on September 25. He guest starred as himself on October 20 in the episode "Musicale" of ABC's Mr. Smith Goes to Washington, a sitcom starring Fess Parker, based on the 1939 Frank Capra film.

Harpo's final public appearance came on January 19, 1963 with singer/comedian Allan Sherman. Sherman burst into tears when Harpo announced his retirement from the entertainment Business. Comedian Steve Allen, who was in the audience, remembered that Harpo spoke for several minutes about his career, and how he would miss it all, and repeatedly interrupted Sherman when he tried to speak. The audience found it charmingly ironic, Allen said, that Harpo, who had never before spoken on stage or screen, "wouldn't shut up!" Harpo, an avid croquet player, was inducted into the Croquet Hall of Fame in 1979.

Harpo Marx died on September 28, 1964 (his 28th wedding anniversary), at age 75 in a West Los Angeles hospital, one day after undergoing heart surgery. Harpo's death was said to have hit the surviving Marx brothers very hard. Groucho's son Arthur Marx, who attended the funeral with most of the Marx family, later said that Harpo's funeral was the only time in his life that he ever saw his Father cry. In his will, Harpo Marx donated his trademark harp to the State of Israel. His remains were cremated, and his ashes were scattered at a golf course in Rancho Mirage, California.

Marx was portrayed by actor Daniel Fortus in the Broadway production of Minnie's Boys, a Broadway musical that ran for 64 performances at the Imperial Theatre from March to May 1970. The show focused on the early days of the Marx Brothers' act and the importance of their mother Minnie's strong hand in guiding and molding them into a successful vaudeville and film comedy team.

Phoebe Snow's Harpo's Blues (1975) was written about "a guy in a band who became Harpo Marx ... he wouldn't talk, and his eyes would roll around ... "

Actress Patricia Lopez played Harpo in a send-up of Hollywood movie making, the 1980 Broadway production, "A Day in Hollywood/A Night in the Ukraine", for which she won a Tony Award for Best Performance by a Featured Actress in a Musical.

He was also portrayed by Les Marsden in Groucho: A Life in Revue, which appeared off-Broadway at the Lucille Lortel Theatre. The show ran for 264 performances from October 1986 through May 1987.

In 2002, a Golden Palm Star on the Palm Springs, California, Walk of Stars was dedicated to him.