Thomson was educated at Worsley's School (1866–74), and at Eton College (1874–9). He then went up to New College, but suffering from depression, he left Oxford University after only two terms and in 1882 emigrated to the United States to train as a farmer in Iowa.
According to his biographer, Noel Rutherford: "In 1883 he learned that Grace Webber was contemplating marriage to another, which led to a relapse of his nervous condition and a precipitate return to England. He was able to reach an understanding with the Webbers that if he could establish himself financially a marriage proposal might be entertained, and with that end in mind, and through the good offices of his father, he obtained a place as a cadet in the colonial service attached to Sir William Des Voeux, governor of Fiji."
In 1884 Thomson was appointed stipendiary magistrate at Nadroga. Richard Deacon argues that "he had a natural gift for learning languages and was made a magistrate at the end of three months, instead of having to wait two years like his fellow cadets." After three years in Fiji he was transferred to British New Guinea. However, he contracted malaria and was invalided home. After make a full recovery he married Grace Webber in October 1889. The following year he became an adviser to the high commissioner for the Western Pacific. Over the next eleven months Thomson reformed taxation and introduced penal reforms. In 1891 he became assistant commissioner for native affairs in Suva, but in 1893, because of the health of his wife, Thomson returned to England.
Thomson entered the Inner Temple and read for the bar examinations. He also embarked on a career as a writer. This included the publication of South Sea Yarns (1894), The Diversions of a Prime Minister (1894) and The Indiscretions of Lady Asenath (1898). Unable to make a living from writing Thomson became successively governor of Cardiff, Dartmoor and Wormwood Scrubs prisons and from 1908 until 1913 he served as secretary to the Prison Commission. According to Noel Rutherford: "As a prison governor Thomson had to attend all executions carried out in his prison. This seems to have affected him little and he remained a firm advocate of capital punishment. As secretary of the Prison Commission he had to deal with those opposed to it and gave them short shrift. He was equally dismissive of suffragettes, especially when they responded to imprisonment by engaging in hunger strikes."
In 1913 Thomson was appointed assistant commissioner of the Metropolitan Police and head of the Criminal Investigation Department (CID) at New Scotland Yard. When the First World War broke out in 1914 the CID became the enforcement arm of the War Office and Admiralty in intelligence matters. Thomson now became head of the 114-man Special Branch, a unit set up to conduct investigations to protect the State from perceived threats of subversion.
Thomson joined forces with Vernon Kell and Eric Holt-Wilson of the Home Section of the Secret Service Bureau, that had responsibility for investigating espionage, sabotage and subversion in Britain, to draft The Defence of the Realm Act (DORA). This was an attempt "to prevent persons communicating with the enemy or obtaining information for that purpose or any purpose calculated to jeopardise the success of the operations of His Majesty's Forces or to assist the enemy." This legislation gave the government executive powers to suppress published criticism, imprison without trial and to commandeer economic resources for the war effort. During the war publishing information that was calculated to be indirectly or directly of use to the enemy became an offence and accordingly punishable in a court of law. This included any description of war and any news that was likely to cause any conflict between the public and military authorities.
Thomson later recalled that a major problem in 1914 was spy mania as reports flooded in of German agents working in Britain: "It assumed a virulent epidemic form accompanied by delusions which defied treatment... It attacked all classes indiscriminately and seemed even to find its most fruitful soil in sober, stolid, and otherwise truthful people." Of the twenty-one German suspects arrested only one was brought to trial. As Christopher Andrew, the author of The Defence of the Realm: The Authorized History of MI5 (2009) has pointed out: "After the outbreak of war.... German military intelligence began to target Britain, though its main priorities remained France and Russia... The most active period for German espionage in Britain was the first winter of the war."
On 20th October 1914, one of Thomson's agents, Jeremiah Lynch (1888-1955), arrested German spy, Carl Hans Lody. He received a public trial with his case was reported widely in the press. This provided Thomson with the image of a successful spy catcher. Lody was convicted of war treason and was executed in the Tower of London on 6th November.
Thomson had the responsibility of arresting and interrogating German spies. Twelve of these were executed during the First World War. According to Richard Deacon: "Thomson was contemptuous of the calibre of German spies, claiming that they were untrained for gathering information of any practical value. He was himself one of the most formidable interrogators of his day."
Basil Thomson recruited Arthur Maundy Gregory as an agent. According to Brian Marriner: "Gregory, a man of diverse talents, had various other sidelines. One of them was compiling dossiers on the sexual habits of people in high positions, even Cabinet members, especially those who were homosexual. Gregory himself was probably a latent homosexual, and hung around homosexual haunts in the West End, picking up information.... There is a strong suggestion that he may well have used this sort of material for purposes of blackmail."
Thompson later admitted that it was Gregory who told him about the homosexual activities of Sir Roger Casement. "Gregory was the first person... to warn that Casement was particularly vulnerable to blackmail and that if we could obtain possession of his diaries they could prove an invaluable weapon with which to fight his influence as a leader of the Irish rebels and an ally of the Germans."
On 21st April 1916, Casement was arrested in Rathoneen and subsequently arrested on charges of treason, sabotage and espionage. As Noel Rutherford points out: "Casement's diaries were retrieved from his luggage, and they revealed in graphic detail his secret homosexual life. Thomson had the most incriminating pages photographed and gave them to the American ambassador, who circulated them widely. They were a significant, if unmentioned, ingredient in the trial and subsequent execution of Casement." Later, Victor Grayson claimed that Arthur Maundy Gregory had planting the diaries in Casement's lodgings.
In January 1917, the German military attaché in Madrid transmitted radio messages to Berlin describing the helpful activities of a German spy, code-named H-21. French intelligence agents intercepted the messages and, from the information they contained, identified H-21 as Margareta Zelle (Mata Hari). On 13th February 1917, she was arrested in Paris. Thomson went to France to interrogate her, and concluded that there was no evidence that she was a spy. However, she was executed on 15th October, 1917.
Thomson worked very closely with Vernon Kell of the Home Section of the Secret Service Bureau (MI5). Thomson and Kell decided to create a card-index system on all potential subversives. It is claimed that acquired details of over 16,000 people. It has been claimed that most of these people were just members of left-wing organisations and were not guilty of subversion.
Noel Rutherford has argued: Thomson's most controversial activities concerned his surveillance of labour organizations. In 1916 the Ministry of Munitions asked him to organize an intelligence operation to report to it on industrial unrest. Thomson culled some of the best men from the CID for this service, and on the basis of their assessments issued regular reports to the ministry and later to the Home Office. In May 1917 a major strike occurred among engineering and munitions workers in response to a ‘comb-out’ to draft unskilled workers from these protected industries into the army. The war cabinet sought Thomson's advice on the matter. He advised prosecuting the ringleaders. Seven were arrested and the strike was called off in return for a pledge that no further arrests would be made."
In early 1918 Thomson asked Arthur Maundy Gregory to spy on Victor Grayson, the former MP for Colne Valley, who was described as a "dangerous communist revolutionary". Gregory was told: "We believe this man may have friends among the Irish rebels. Whatever it is, Grayson always spells trouble. He can't keep out of it... he will either link up with the Sinn Feiners or the Reds." Gregory became friendly with Grayson. David Howell writes that "Grayson subsequently lived in apparent affluence - a contrast with his recent poverty - in a West End flat. His associates included Maundy Gregory... The significance of this relationship and the source of Grayson's income remain unknown."
In 1919 Thomson was appointed as head of the Directorate of Intelligence. This placed him in overall control of naval, military, foreign, and domestic intelligence. Influenced by the events of the Russian Revolution, Thomson developed a strong fear of a revolution. He later wrote that: "February 1919 was the high-water mark of revolutionary danger in Great Britain. Many of the soldiers were impatient at the delay in demobilization. Russia had shown how apparently easy it was for a determined minority to seize the reins of power."
Thompson's promotion created a great deal of jealousy in the intelligence services. Eric Holt-Wilson of MI5 wrote: "Despite statements to the contrary in the press and elsewhere, Sir Basil Thomson's organization has never actually detected a case of espionage, but has merely arrested and questioned spies at the request of MI5, when the latter organization, which had detected them, considered that the time for arrest had arrived. The Army Council are in favour of entrusting the work to an experienced, tried and successful organization rather than to one which has yet to win its spurs. Sir Basil Thomson's existing higher staff consists mainly of ex-officers of MI5 not considered sufficiently able for retention by that Department. The Army Council are not satisfied with their ability to perform the necessary duties under Sir Basil Thomson's direction, and they are satisfied that detective officers alone, without direction from above, are unfitted for the work."
In 1921 a Secret Service Committee of senior officials was instructed to make recommendations "for reducing expenditure and avoiding over-lapping". In its report published in July, Thomson's Directorate of Intelligence, was criticized for overspending, duplicating the work of other agencies and producing misleading reports. Sir William Horwood, the Commissioner of the Metropolitan Police, joined in the attack and sent David Lloyd George a memorandum denouncing "the independence of the Special Branch" under Thomson as a "standing menace to the good discipline of the force" and that the Directorate of Intelligence was both wasteful and inefficient. As a result of these complaints Thomson was asked to resign.
Thomson's great friend, William Reginald Hall, took up his case in the House of Commons. On 3rd November 1921, Hall declared: "There is no man who has been a better friend of England than Sir Basil Thomson". He went on to argue that his downfall was due not merely to his "open enemies", the Bolsheviks, the Russians, the extremists" but to a secret plot that involved the Labour Party.
In December 1925, Thomson and a young woman named Thelma de Lava, were arrested in Hyde Park and charged with committing an act in violation of public decency. Thomson pleaded not guilty and said he was carrying out investigations for an article on prostitution. He was found guilty and fined £5. Christopher Andrew, the author of The Defence of the Realm: The Authorized History of MI5 (2009) has argued: "Thomson's supporters hinted darkly that he had been framed either by his enemies in the Met or by subversives."