Ella was born at Thirsk 19 December 1802. He was intended by his father, Richard Ella, for the law; but in 1819 he was taught the violin by M. Fémy. On 18 January 1821 he made his first appearance as a professional musician in the orchestra of Drury Lane Theatre. In the following year he was promoted to the band of the King's Theatre.

In 1826, on the completion of his musical education under Thomas Attwood, and subsequently under François-Joseph Fétis in Paris, that he took his place as a member of major orchestras of London, such as the Philharmonic and the Ancient Concerts. The Saltoun Club of Instrumentalists and the Società Lirica were perhaps founded by him as early as this period of his life. They were intended for the practice and performance of unfamiliar operatic music. He played in the orchestra on the occasion of Carl Maria von Weber's funeral, 21 June 1826. Around this time he was appointed to a subordinate post at the Royal Academy of Music, and became musical Editor of the Athenæum and other papers.

In 1830 he seems to have given public concerts under the patronage of Augustus FitzGerald, 3rd Duke of Leinster. During his journeys to the continent he made the acquaintance of musical celebrities. The set of chamber concerts which he inaugurated in 1845, under the name of the "Musical Union", and which originated in a weekly meeting at his own house, had an effect on public taste, particularly through its successor, the Popular Concerts. By the formation of an aristocratic committee, and by making the concerts in some measure social gatherings, for which membership could only be obtained by personal introduction, he secured for his scheme a high prestige. The programme always contained at least two concerted instrumental works and the executants were generally artists of established position, many of whom had not appeared before in England. The annual series consisted of eight afternoon concerts given during the season, at first in Willis's Rooms, and a benefit concert when vocal music was allowed to form part of the programme. The artists played in the middle of the room, with the audience surrounding them, and programmes were printed and sent out to the subscribers a few days before the concert. The undertaking met with such support that a series of evening concerts, at somewhat lower prices, was started in the early part of 1852, under the title of "Musical Winter Evenings".

He wrote a Victoria March in November 1837, almost his only experiment as a Composer. In 1855 he was appointed musical lecturer to the London Institution, and the substance of three lectures on melody, harmony, and counterpoint was given in the Musical Union Record, i.e. the programmes. A set of four lectures on dramatic music was also published (1872). In 1869 he published Musical Sketches Abroad and at Home, a volume of anecdotes, autobiographical and otherwise, bearing on music. The book ran through two editions, and a third, edited by the author's friend John Belcher was published in 1878. A Personal Memoir of Meyerbeer, with Analysis of "Les Huguenots" is Ella's other important contribution to musical literature.

In 1858 both sets of concerts were transferred to Hanover Square Rooms, and in the following year to the newly opened St James's Hall. In the same year, the Monday Popular Concerts having been set on foot, Ella's evening series was given up. His title of professor was derived from a post at the London Institution. He was honorary member of the Philharmonic Academy of Rome, and of the Philharmonic Society of Paris.

The Musical Union ceased to exist in 1880, when Ella gave up active work. For the last twenty years of his life he lived at 9 Victoria Square, London, where he died on 2 October 1888, after repeated attacks of paralysis. For some years before his death he had been totally blind. He was buried in Brompton Cemetery on 5 October.