James was offered a temporary appointment at Winchester College in 1933, where he soon secured a permanent staff position. He taught chemistry, as well as a variety of related subjects. He remained there until 1945, and was High Master of The Manchester Grammar School from 1945 to 1962. He then became the first Vice-Chancellor of the University of York, serving from 1962 to 1973. He is credited with creating the collegiate structure of the University of York.

In 1939 James married Cordelia Wintour, the sole daughter of Major-General Fitzgerald Wintour, and sister of Charles Wintour, an Editor of the British daily newspaper 'Evening Standard'. Cordelia is a paternal aunt of the fashion magazine Editor Anna Wintour and of the British Journalist Patrick Wintour.

James set forth his educational ideas in two books, An Essay on the Content of Education (1949) and Education and Leadership (1951). His Elements of Physical Chemistry, written in collaboration with another master during his Winchester period, became a standard school textbook.

James strongly believed in equality of opportunity. He strove to distinguish between equality of opportunity and uniformity of treatment. He held that able children are a nation's most precious asset, and that the academically gifted will best develop their talents in company with their peers. In the 1950s and 1960s it required considerable moral courage to stand out against the dominant educational opinion, which had settled dogmatically upon scrapping grammar schools in favour of the supposedly more egalitarian comprehensive system. He advocated a pure meritocracy. Selection at Manchester Grammar School was by competitive examination, with no marks added for wealth or family connections. It was the essence of his philosophy that grammar schools should serve as ladders, giving all levels of society access to the highest places in the land. His approach bore fruit; in the mid-1950s Manchester Grammar School was attaining up to 45 scholarships every year at Oxford and Cambridge. However, in the short term his efforts were doomed to failure; Manchester was forced to become a fee-charging school.

James was knighted in 1956, and was created a life peer as Baron James of Rusholme, of Fallowfield in the County Palatine of Lancashire, in 1959.

As a member of the Central Advisory Council on Education he was part of the 1959 Crowther report on secondary education, which contained two of his special concerns - the raising of the school leaving age to 16; and for specialised studies in sixth forms.

He proved more successful than many of his peers in dealing with the student troubles of the 1960s. He made himself available for discussion with anybody. Upon his retirement in 1973, the President of the student union lauded the respect and admiration which the Vice-Chancellor had earned.

In 1962 James moved to the University of York, named as Vice Chancellor. He was involved with every detail of the university's development. He described his work with the Architects as one of the most exciting times of his life. The institution which he helped to shape exhibited many characteristics of Oxford; James stressed that at least half the students should live on the campus; he housed them in separate colleges which were conceived not simply as halls of residence, but as the centres of the university's social and academic life. To create the closest possible relationship between Teacher and taught, tutors as well as students were given rooms in the colleges; and the tutorial became the basic form of instruction. Half of the undergraduates were women.

In 1970, British Secretary of State for Education and Science Margaret Thatcher, asked James to chair a government inquiry into Teacher training. His slant on education of Future teachers included a conviction that teachers should be educated rather than trained. He wanted teachers who could inspire children; beside this great principle, what or how they should teach were questions of lesser import. He held that a single subject, well taught, might form the basis of a true education; equally that an apparently alluring spread of studies, badly taught, would simply become a drab routine.

The James Report on Teacher Training, which appeared in 1972, envisaged the colleges of education as mini-universities where students would follow a two-year, purely academic, course leading to a Diploma of Higher Education, before engaging specifically with teaching. Only after the DipHE should Future teachers undertake a two-year course addressed to the theory and practice of work in the classroom. It was also proposed that teachers should attend an in-service course for not less than one term every seven years.

After his 1973 retirement from, James served on the Social Science Research Council. He also chaired of the Personal Social Services Council, and the Royal Fine Art Commission. He broadcast in both the radio and television versions of The Brains Trust. However, he campaigned vigorously against popular television, holding that it produced a "diseducative" effect on young viewers.