On Broadway, Mandvi appeared as Ali Hakim in the 2002 production of Oklahoma! directed by Trevor Nunn. He also appeared in the play Homebody/Kabul by Tony Kushner. He portrayed Fritz Haber in the off-Broadway play Einstein's Gift.

Mandvi made his television debut as a doorman at the Miami Biltmore Hotel in the episode "Line of Fire" of the series Miami Vice. He has since appeared in numerous television shows including ER, The Sopranos, Sex and the City, CSI, Oz, Ed, The Bedford Diaries, Jericho, Sleeper Cell and various editions of Law & Order, including Criminal Intent, Special Victims Unit and Trial by Jury. He was the book reader for audio editions of Salman Rushdie's Shalimar the Clown (2005) and V. S. Naipaul's Magic Seeds (2004). In 2011 he appeared in Taco Bell commercials.

In 2006, Mandvi auditioned for The Daily Show. He was hired immediately and appeared on the show the same day. Mandvi became a regular correspondent in 2007. He often appears in segments satirizing and commenting on Islamic, Middle-Eastern, and South-Asian-related issues with such titles as "Senior Asian Correspondent," "Senior Middle East Correspondent," "Senior Foreign Looking Correspondent," and "Senior Muslim Correspondent."

Mandvi played a dentist alongside Ricky Gervais in the 2008 romantic comedy Ghost Town, as well as office employee Bob Spaulding in The Proposal, starring Sandra Bullock. Today's Special, which Mandvi co-wrote with Jonathan Bines, premiered at the London Film Festival in October 2009 and New York's Mahindra Indo-American Arts Council Film Festival on November 11, 2009. He appeared in It's Kind of a Funny Story, a coming-of-age film written and directed by Anna Boden with Ryan Fleck, adapted from the 2006 novel by Ned Vizzini. He also co-starred as a Guantanamo captive in the film The Response, a script based on the transcripts of Combatant Status Review Tribunals convened in Guantanamo in 2004. In M. Night Shyamalan's The Last Airbender (released in 2010) he played a major role as Commander Zhao. Mandvi played the role of Mr. Chetty in the 2013 comedy The Internship, which starred Vince Vaughn and Owen Wilson and portrayed Ash Vasudevan in the 2014 film Million Dollar Arm.

Mandvi is involved with disaster-relief organizations such as the charity initiative Relief 4 Pakistan, which assists in flood relief in Pakistan. In 2010, he hosted the "Stand Up for Religious Freedom" comedy event to raise money for the organization. After the 2010 Haiti earthquake, he performed with fellow Daily Show correspondent Wyatt Cenac at Conan O'Brien's "I'm with Coco" Benefit for earthquake victims. He is also a supporter of the Endometriosis Foundation of America.

Mandvi played Melchior in On the Razzle by Tom Stoppard at the Williamstown Theatre Festival and appeared in the docudrama Guantanamo: Honor Bound to Defend Freedom at the Culture Project. In 2012 Mandvi starred in Disgraced at Lincoln Center's Claire Tow Theater. He played the lead role of Amir, a Pakistani American Lawyer struggling with his identity and Islam in the drama by Ayad Akhtar. The play went on to win the 2013 Pulitzer Prize for Drama. Mandvi was nominated for a Lucille Lortel Award for his role.

In October 2013, during a segment on The Daily Show, Mandvi's interview with Don Yelton led to Yelton's resignation from the North Carolina Republican Party office.

In November 2014, he appeared on The Dan Patrick Show. On the show, he revealed the story from his book No Land's Man where his father moved his family to the U.S. because the U.S. has brunch.

Beginning in June 2015, Mandvi portrayed Rafiq Massoud in the HBO comedy series The Brink. Mandvi also serves as a Writer and co-producer on the series. In April 2015, Mandvi appeared on Person of Interest as Sulaiman Khan, the CEO of a software security firm.

In 2016, Mandvi joined the climate change documentary show Years of Living Dangerously as one of its Celebrity correspondents.

On August 26, 2017, he married his girlfriend of 3 years Shaifali Puri, who is an author and humanitarian in Atlanta, GA in front of 220 guests made up of friends and family.