Sermisy was most likely born either in Picardy, Burgundy, or Île-de-France, based on the similarity of his surname to place names there. Sometime in his early life he may have studied with Josquin des Prez, if Pierre Ronsard is to be believed, but many musicologists consider the claim unreliable; at any rate he absorbed some of the older composer's musical ideas either early, or later, as he became acquainted with his music. Josquin was possibly at the French court between 1501 and about 1503, though this has never been definitely established, so a master-pupil relationship would have been possible then; Sermisy's whereabouts before 1508 are not known, but presence at the Royal Chapel was certainly possible.

In 1508 the young Sermisy was appointed as a singer in the Royal Chapel of Louis XII, where he was also a cleric. His birthdate is inferred from the date he joined the royal chapel; 18 was about the right age for such an appointment. In 1515 he went to Italy with Francis I, and in 1520 he was part of the musical festivities arranged by Francis I and Henry VIII of England at the Field of the Cloth of Gold, directed by Jean Mouton, where he was almost certainly a singer. He may have been a Composer of some of the music there as well. In 1532 he also participated in the similar meeting between the kings at Boulogne, for which he wrote a ceremonial motet.

For a while in the early 1520s Sermisy was a canon at Notre-Dame-de-la-Rotonde in Rouen, but he left there in 1524 to take a similar position in Amiens. By 1532 he was music Director of the Royal Chapel, still under Francis I, who reigned until 1547. At this post he was expected to teach and care for the boys of the choir, as well as find talented Singers to recruit. In 1533, in addition to his post at the Royal Chapel, he became a canon of the Sainte-Chapelle, which would have required him to live in Paris. He acquired a large house there, large enough to shelter refugees from the church in St Quentin when the Spanish sacked their city in 1559. In 1554 he also was given a prebend at Ste Catherine in Troyes. Few biographical details are available about his last years, but he seems to have been active as a Composer up to the end of his life based on publication dates of works. He was buried in the Sainte-Chapelle.

Sermisy wrote two of the few polyphonic settings of the Passion found in French music of the period; the musical setting is simple, compared to his masses and motets, and he strove to make the words clearly understandable. The gospels chosen were those of St. Matthew and St. John. Sermisy's settings were published in the 10th volume of Motets published by Pierre Attaignant.