Black met Hilary Joan Vaughan (1924–1986) at a ball at university in 1944 and the couple married in 1946 upon his graduation. He described her as the "mainspring" of his life until she died aged 61 in Surrey. The couple had a daughter, Stephanie, born in 1951. Black remarried in 1994, to Professor Rona MacKie. Black was a very private man who was averse to publicity and was horrified to discover he had won the Nobel Prize.

Black had large debts upon his graduation from university, so he took a teaching job in Singapore for three years, before moving to London in 1950 and then on to join the University of Glasgow (Veterinary School) where he established the Veterinary Physiology Department and developed an interest in the way adrenaline affects the human heart, particularly in those suffering from angina. Having formulated a theory of an approach by which the effects of adrenaline might be annulled, he joined ICI Pharmaceuticals in 1958, remaining with the company until 1964, during which time he invented propranolol, which later became the world's best-selling drug. During this time Black pioneered a method of research whereby drug molecules were purposefully built instead of being synthesised first and then investigated for their potential medical uses. The discovery of propranolol was hailed as the greatest breakthrough in the treatment of heart disease since the discovery of digitalis.

At the same time, Black was developing a similar method of inventing drugs for treatment of stomach ulcers, but ICI did not wish to pursue the idea so Black resigned in 1964 and joined Smith, Kline and French where he worked for nine years until 1973. While there, Black developed his second major drug, cimetidine, which was launched under the brand name Tagamet in 1975 and soon outsold propranolol to become the world's largest-selling prescription drug.

Until 1967, University College, Dundee was the site for all clinical medical activity for the University of St Andrews. He matriculated at University College (which eventually became the University of Dundee) in 1943 and graduated from University of St Andrews School of Medicine with an MB ChB in 1946. During his time at St Andrews, Black lived in St Salvator's Hall.

Black was appointed professor, and head of department, of pharmacology at University College London in 1973 where he established a new undergraduate course in medicinal chemistry but he became frustrated by the lack of funding for research and accepted the post of Director of therapeutic research at the Wellcome Research Laboratories in 1978. However he did not agree with his immediate boss there, Sir John Vane, and resigned in 1984. Black then became Professor of Analytical Pharmacology at the Rayne Institute of King's College London medical school, where he remained until 1992. He established the James Black Foundation in 1988 with funding from Johnson and Johnson and led a team of 25 Scientists in drugs research, including work on gastrin inhibitors which can prevent some stomach cancers.

Black was elected a Fellow of the Royal Society (FRS) in 1976 and the same year he was awarded the Lasker award. His certificate of election to the Royal Society reads:

In 1979, he was awarded the Artois-Baillet Latour Health Prize. In 1982 Black was awarded the Wolf Prize in Medicine, and the year after the Scheele Award. He was awarded the 1988 Nobel Prize in Medicine along with Gertrude B. Elion and George H. Hitchings for their work on drug development. In 1994 he received the Ellison-Cliffe Medal from the Royal Society of Medicine.

Black was made a Knight Bachelor on 10 February 1981 for services to medical research, receiving the honour from the Queen at Buckingham Palace. On 26 May 2000 he was appointed to the Order of Merit, of which there are only 24 members at any one time, by Queen Elizabeth II.

In 1980, Black's association with the University of Dundee was renewed when the institution recognised his many achievements by conferring him with the Honorary Degree of Doctor of Laws. In 1992 he accepted an offer to succeed the 16th Earl of Dalhousie as Chancellor of the University and was installed as Chancellor at the award ceremony held in Dundee Repertory Theatre on 29 April 1992. Appropriately the first degree he conferred was to Professor Robert Campbell Garry, who had been responsible for his original appointment at University College Dundee. Sir James remarked at this ceremony that by returning to Dundee he was "in a real sense, coming home".

As Chancellor, Sir James Black did much to promote the University of Dundee and was a popular figure within the University. He was awarded a second honorary degree, that of Doctor of Science, in 2005. He retired from his post the following year, and his association with the University of Dundee was marked with launching of the £20 million Sir James Black Centre. The centre, intended to promote interdisciplinary research in the life sciences, was opened by Sydney Brenner in 2006. Sir James Black himself visited the centre in October 2006 and was reportedly excited and pleased by what he saw.

Shortly after his death the Bute Medical School of the University of St Andrews, where Black had studied for his initial degree in Medicine, announced that an honorary 'Sir James Black Chair of Medicine' would be created. In September 2010 the first Chair of Medicine at the University was given to Professor Stephen H Gillespie MD, DSc, FRCP (Edin), FRC Path, who left his post as Professor of Medical Microbiology at UCL.