Arkin guest-starred in the award-winning television show Happy Days in episode 35, season 2 in 1975. Also in 1975, he made a guest appearance on Barney Miller (episode: Grand Hotel). His first starring role in television was as Lenny Markowitz, the central character in the 1977 situation comedy Busting Loose. He since has appeared in various television series such as A Year In The Life (1988), The Twilight Zone (1986), Northern Exposure (CBS, 1990–95), where he played the mercurial barefooted chef Adam, and Chicago Hope (CBS, 1994–2000). He appeared in two Law & Order episodes, "Self Defense" (season three, 1992) as jewelry store owner George Costas and "Red Ball" (season 16, 2005) as a district attorney named Charles Graham. He also appeared in Picket Fences (season two, episode 13). Other television appearances include The West Wing (1999) (as trauma specialist and Psychiatrist Dr. Stanley Keyworth), Frasier, as an obsessive fan of Frasier's, for which he was nominated for a Primetime Emmy Award for Outstanding Guest Actor in a Comedy Series, Boston Legal, Baby Bob, Monk and 8 Simple Rules for Dating My Teenage Daughter (2002). In 2007, he starred in the NBC drama Life in the role of Ted Earley. He played a white separatist leader named Ethan Zobelle during the second season of the FX series Sons of Anarchy reuniting with former 8 Simple Rules castmate Katey Sagal. In 2011, he appeared in The Closer episode "To Serve With Love" (season seven) as a suspect named Steven Hirschbaum. He also portrayed mob boss Theo Tonin on Justified in 2011, and an FBI agent in The Chicago Code, also in 2011.

Arkin also played a minor role in the NPR radio drama of Star Wars, A New Hope as the voice of Fixer. For PBS, he voiced Meriwether Lewis in Ken Burns' The Voyage of the Corps of Discovery (1997). He also provided character voice work for the Emmy-winning series The National Parks.

He has a daughter, Molly, with his first wife, Linda. He was married to Phyllis Anne Arkin (née Lyons) from 1999 to 2013, and they had one son together. Phyllis filed for divorce in August 2013.

Arkin's film appearances include Hitch (2005) and Halloween H20: 20 Years Later (1998). He has performed in Broadway, off-Broadway, and regional theatre productions, including both the South Coast Repertory world premiere and Broadway production of Brooklyn Boy by Playwright Donald Margulies. In addition, he is known for his directing work, having done episodes of Grey's Anatomy, Boston Legal, The Riches, Dirt, Ally McBeal, Sons of Anarchy, The Blacklist, Justified, and Masters of Sex. He won an Emmy for directing the Showtime television film My Louisiana Sky. In April 2008, Arkin guest-starred in the Web series Gorgeous Tiny Chicken Machine Show. He played the part of a divorce Lawyer in the film A Serious Man (2009), directed by Ethan and Joel Coen.

He has directed three episodes of the 2013 Cold War television drama, The Americans, and the final episode of the second season (2014) of Masters of Sex.