In 1919, the British government had its first major encounter with terrorism, when the Irish Republican Army was established to drive the British out of Ireland. The government responded to the IRA's acts of terror – which included the assassination of civilians as well as soldiers – with indiscriminate reprisals; these were met in turn by further escalation from the IRA. The prime minister, David Lloyd George, declared that the British government would never talk to the "murder gang", as he described the IRA. But by 1920, it became clear to both sides that a military victory was impossible. Lloyd George secretly began to initiate contact with Michael Collins and other IRA leaders, using a relatively junior former customs official, Alfred Cope – who managed to open up a channel to the rebels and negotiate a ceasefire. This led to full-blown talks in Downing Street in 1921, and eventually to an agreement, albeit a flawed one that later unravelled.

Seventy-six years later, in December 1997, Tony Blair and I sat down in the same cabinet room in Downing Street with Gerry Adams and Martin McGuinness; the negotiating teams, from Sinn Féin and the British government, even sat on the same sides of the table as they had in 1921. On both occasions, the meeting was a big event. There were more TV cameras outside Downing Street than there had been on election day seven months earlier, and we were all nervous. Alastair Campbell had ordered the Christmas tree be removed from in front of the door of Number 10, so that there could be no pictures of terrorists in front of festive decorations.

In 1921, Frances Stevenson, Lloyd George's secretary and mistress, said she had never seen the prime minister "so excited as he was before De Valera arrived. He kept walking in and out of my room and I could see he was working out the best way of dealing with Dev … He had a big map of the British empire hung on the wall in the cabinet room, with great blotches of red all over it. This was to impress on Dev the greatness of the British empire and the King." In 1997, before we sat down Martin McGuinness tried to break the ice, and said: "So, this is where all the damage was done, then." We thought this was a reference to the IRA attack on Downing Street in 1991, and I responded by saying "Yes, the IRA mortars landed in the garden behind you, and blew the windows in. My brother dragged John Major under the table and four overweight policemen came running in waving their revolvers." McGuinness was horrified. "No, I didn't mean that," he said. "I meant this was where Irish Republicans gave everything away all those years ago." As is so often the case, the terrorists had a better memory for what had gone before than the government. (I use the word "terrorist" here for the sake of simplicity, but it isn't a particularly useful term to define a group – terror is a tactic employed by governments, groups, and individuals. I mean it to refer to non-state armed groups that use terror and enjoy significant political support.)

When it comes to terrorism, governments seem to suffer from a collective amnesia. All of our historical experience tells us that there can be no purely military solution to a political problem, and yet every time we confront a new terrorist group, we begin by insisting we will never talk to them. As Dick Cheney put it, "we don't negotiate with evil; we defeat it". In fact, history suggests we don't usually defeat them and we nearly always end up talking to them. Hugh Gaitskell, the former Labour leader, captured it best when he said: "All terrorists, at the invitation of the government, end up with drinks in the Dorchester."

The IRA in Derry, 1972
The IRA in Derry, 1972. Photograph: Keystone-France/Keystone-France/100% Keystone

Certainly that was true throughout the history of the British empire. Menachem Begin's Irgun was responsible for blowing up the King David Hotel in Jerusalem in 1946, killing 91 and injuring 46. The British authorities called him a terrorist and tried to hunt him down. But when he became prime minister of Israel and made peace with Egypt, we lauded him as a statesman. We accused Jomo Kenyatta of being a terrorist and locked him up, but later negotiated Kenyan independence with him. We exiled Archbishop Makarios to the Seychelles for supporting terrorism but made peace with him and he became the first leader of an independent Cyprus.

It is hard not to respond emotionally to a terrorist act in the heat of the moment. When we see videos of western journalists being beheaded or TV footage of small children being blown up by IRA bombs it seems obvious that the only answer is force. It is easy to regard any suggestion that we should ever talk to people capable of such savagery as immoral. George W Bush, in a speech to the Knesset in 2008, even suggested that talking to terrorists in this way amounted to appeasement. That is, however, to misunderstand appeasement. Chamberlain's mistake in 1938 was not talking to Hitler – it was entirely sensible to take every possible step to avert another catastrophic world war – but to think Hitler could be bought off with a slice of Czechoslovakia. Talking to terrorists and agreeing with them are not the same thing. The British government negotiated with Sinn Féin, but we did not concede to their demand of a united Ireland at the barrel of a gun.

* * *

I have not always been convinced of the need to talk to terrorists. The first time I met a terrorist, I did so reluctantly. My father had been hit by an IRA bullet in an ambush in 1940, and my eldest brother was on an IRA death list for eight years while he worked for Margaret Thatcher. When I first met Gerry Adams and Martin McGuinness, in Belfast in October 1997, I declined to shake their hands – a petty gesture I now regret, but one that recurs again and again at encounters between governments and terrorists.

A few days after that meeting, I got a call from McGuinness, who asked me to come to meet him alone in Derry, without involving what he called "the securocrats" in the British government. I flew to Aldergrove, took a taxi to Derry, and stood on a street corner feeling mildly foolish. After 10 minutes or so, two men with shaved heads approached me, said "Martin sent us," and pushed me into the back of a taxi. They drove me round in circles for an hour until I was completely lost and then ejected me when we arrived in front of a neat modern house on a small estate. I knocked on the door and McGuinness opened it on crutches, making a not very funny joke about kneecapping, the IRA's favoured method of punishment. I spent three hours with him in front of an open fire with tea and sandwiches left by the considerate owner of the house. We didn't make any breakthroughs, but it set a pattern and I spent a good part of the next 10 years flying back and forth across the Irish Sea to meet Adams and McGuinness in safe houses in west Belfast, Derry and Dublin, going on to their turf rather than demanding they come to grand government buildings. The shared risks we took helped establish a relationship of trust, in which ideas could be explored informally and progress made. That encounter, and my 17 years of experience since in talking to terrorists in different parts of the world have convinced me of the wisdom of this approach if we want to resolve armed conflict in a lasting way.

Terrorists are nearly always keen to talk. Gerry Adams could not persuade David Trimble or his Ulster Unionist party to talk to him even when they entered negotiations; they insisted on directing all of their comments to the chair, US Senator George Mitchell, rather than to Sinn Féin. Adams resorted to desperate tactics: he would loiter in the corridor until one of the unionists entered the gents, and then follow them in, stand at the next urinal, and try to engage them in conversation. They still refused to talk.

I am an unlikely peacenik. I grew up in a military family, and I was involved in the decisions on all of Tony Blair's wars. I do not think that war is always wrong: sometimes it is necessary to stop a dictator, prevent massive human-rights abuses, or expel an invader. But I have also seen that in the modern world, civil wars are the greatest threat to humanitarian security. If you want to fight starvation, the spread of disease, and mass rape – or to help suffering children, whether child soldiers or the victims of war – then the most important thing you can do is to help end armed conflicts, which is why I have decided to dedicate the rest of my life to that goal.

* * *

We usually delay talking to armed groups too long, and as a result, a large number of people die unnecessarily. General David Petraeus admitted that, in Iraq, the US left it far too late to talk to those "with American blood on their hands". We delay because it is argued that talking is too risky – but experience suggests the real risk lies in not talking.

When governments have been wrestling for years with a terrorist group, and it becomes clear that the military option is not working, the tendency is to try it again – what we might call the "one last heave" approach. In Afghanistan, this was the argument for another "surge" of troops, under Petraeus, to put the Taliban on the back foot. Unfortunately, this tactic rarely works. Terrorist groups may not accept that they are being outfought, and generally believe they can wait out the conventional forces arrayed against them.

It has been argued, by some revisionist historians, that talks delayed, rather than hastened, the end of the conflicts in Northern Ireland and the Basque country. Had politicians allowed the army and police to go after the terrorists without "their hands tied behind their backs" by worries about human rights, the argument goes, they would have finished off the terrorists without any need for talking. As someone intimately involved in both negotiations, I can categorically declare this to be false. Although the IRA had been penetrated – and was suffering from the advances made by security forces, which frustrated an increasing number of their attacks – they were not about to be defeated.

As Eliza Manningham-Buller, the former head of MI5, put it in her Reith lectures in 2011, "The divisions in Northern Ireland society, manifested in terrorism, could not be solved militarily. Nor could intelligence and police work, however successful in preventing attacks and informing government, resolve those divisions, although that work could buy time for a political process." The same was true of the Basque conflict. Both were brought to an end by talking. The idea that "one last heave" can deliver a military victory invariably proves to have been a delusion.

A Hezbi-i-Islami insurgent east of Kabul.
A Hezbi-i-Islami insurgent east of Kabul. Photograph: Ghaith Abdul-Ahad

When every other option has been eliminated, and governments are prepared to contemplate talking to terrorists, they face a series of bad arguments against doing so.

First, that talking will give the terrorists legitimacy. It is true that armed groups crave legitimacy, and will go to great lengths to secure it. But it is equally true that such legitimacy is usually temporary, and disappears if they end the talks and return to violence. The Farc in Colombia attained some respectability in the Caguan negotiations in 1998-2002 but lost it as soon as they went back to war, being dismissed as narco-terrorists who had rejected a reasonable offer from the government.

Second, it is argued that agreeing to a ceasefire so that talks can begin allows insurgents to rest, regroup, and re-arm. In fact, experience suggests that it is the armed groups that suffer most from ceasefires, and find it harder to motivate their forces to return to killing if the ceasefire ends.

Third, critics suggest that talking can undermine moderate politicians by favouring the extremists. The Blair government was accused of having sidelined the moderate SDLP to talk to Sinn Féin, but this was another false charge: in fact, we tried at the beginning to build an agreement from the centre outward, but the SDLP made it clear they would not move forward without Sinn Féin.

These specious arguments only cause governments to leave opening up conversations with the terrorists far too late. We forget how such discussions were conducted on previous occasions, and try to reinvent the wheel each time. Opening a channel with a covert group is difficult and dangerous; building trust and turning the channel into a negotiation takes much more time than people think.

Terrorists are much better at learning the lessons of history, partly because leaders of armed groups tend to stay in place much longer than democratically elected prime ministers and presidents. Gerry Adams has seen eight British prime ministers come and go while he has been the leader of Irish Republicans. The terrorists deliberately study what has happened in the past. The Fenians looked at the Russian nihilists, and the original IRA were fascinated by the Boers. Menachem Begin studied both the campaigns of the IRA in 1919-21 and the Russian anarchists. His own memoir was in turn found in an al-Qaida training camp in Afghanistan.

To say that we need to learn the lessons from the past is not simply to suggest that we can adopt a model from Northern Ireland – or anywhere else – that can be plugged into any conflict and used as a template to solve it. Every conflict is different; its causes are different and its solution will be different. But I have now studied most of the negotiations between armed groups and governments in the last 30 years – including those that have succeeded in Mozambique, South Africa, El Salvador, the Philippines and Indonesia, as well as those that have failed in Sri Lanka, Colombia and the Middle East – and there is clearly a pattern to what works and what does not.

Above all, what these experiences demonstrate is that there isn't really an alternative to talking to the terrorists if you want the conflict to end. Hugh Orde, the former chief constable in Northern Ireland, rightly says, "There is no example that I know of, of terrorism being policed out" – or fully defeated by physical force – anywhere in the world. Petraeus said that it was clear in Iraq that "we would not be able to kill or capture our way out of the industrial-strength insurgency that was tearing apart the very fabric of Iraqi society". If you can't kill them all, then sooner or later you come back to the same point, and it is a question of when, not whether, you talk. If there is a political cause then there has to be a political solution.

Should we pay hostage ransoms to terrorists?

Some academics suggest there are ways of ending a terrorist campaign other than negotiation, but a careful look at their examples shows the argument doesn't stand up. It is said that decapitating a movement's leaders can bring about its end. But the assassination of Sheikh Yassin did not lead to the end of Hamas, nor did the capture of Abdullah Ocalan lead to the demise of the PKK. It is also argued that the fact that the Baader-Meinhof gang and the Symbionese Liberation Army both eventually petered out shows that police work can succeed. But it is ludicrous to compare tiny groups of troubled middle-class children with movements that enjoy real political support like the PLO or the FMLN. And lastly, it is claimed that Sri Lanka shows a military solution can work. But Sri Lanka doesn't demonstrate anything of the sort. President Rajapaksa managed to defeat the Tamil Tigers only because its leader, Velupillai Prabhakaran – who had been considered a military genius for most of his life – turned out to be a military fool, in the words of one of the Norwegian negotiators who worked on the peace process. If he had stuck to a guerrilla campaign rather than trying to beat the Sri Lankan army in a conventional war, he would probably still be in the jungle fighting now. And the measures used by the Sri Lankan army to wipe out the Tigers are not methods that could be used by any western government that respects human rights and the rule of law. Finally, although the war is over and there is, thank goodness, no sign of the resurgence of a terrorist campaign, the political problem of Tamil rights still remains unresolved, and trouble will continue until it is.

It is true that an autocratic regime can, by using unrestrained force and blacking out all publicity, finish off a terrorist movement in Chechnya or anywhere else. But as we found out after the Soviet Union collapsed, all this does is to put the problems in the deep freeze; sooner or later they will reappear. Hafez al-Assad thought he had resolved the problem of Sunni rebellion in 1982, when his forces killed tens of thousands of civilians in the city of Hama. But as his son Bashar found out 30 years later, he had not really resolved the problem at all.

Our past experience therefore suggests there is little alternative to talking to an armed group if we want them to stop fighting.

* * *

In February 2008, after I had left government, I argued in the Guardian that we should also talk to Hamas, to the Taliban and even to al-Qaida. My former colleagues in the British government predictably rubbished me. They insisted that while it was acceptable to talk to the IRA and the PLO it was out of the question with these new terrorists.

But are the so-called "fourth wave" terrorists (following on from the anarchists, the nationalists and the Marxists) really so different that there is nothing to be learned from the way we dealt with their predecessors? Or are the issues in the end political – however gruesome the methods they use – and therefore resolvable only by political means? It is true that the ambition of their violence is on a greater scale. Rather than the handful of people the IRA or ETA would kill in a car bombing or an assassination, they want to kill thousands of people. But the 19th-century anarchists had the same ambitions – and in any case, the number of people killed doesn't make any difference to the moral nature of terrorism. It is just as wrong to kill one innocent person as it is to kill a thousand, and it can often be more shocking. It is true that the new terrorists use suicide bombers to maximise the terror they inflict, but so did the Russian nihilists – and the art of suicide bombing, including the use of female suicide bombers, was perfected by the Tamil Tigers in the 1980s.

It is argued that their violence is based on religion, and so it is impossible to negotiate with them – in the words of one former Israeli cabinet member: "When you get God into discussions, God never compromises." But there is no evidence that religious armed groups are harder to engage than secular ones: peace has been made with Islamic armed groups, including the Free Aceh Movement in Indonesia and the Moro Islamic Liberation Front in the Philippines. It is also said that because the aims of these groups are apocalyptic and their demands unnegotiable, it is impossible to treat with them.

While it is true that it is unlikely that any government is going to agree to the creation of a global caliphate, the terrorist groups we encountered in the past also put forward demands that would never be acceptable. No British government was ever going to concede a united Ireland against the wishes of a majority of the people in Northern Ireland. Once discussions were begun with the Irish Republicans, we discovered that they were prepared to settle for something else.

Government officials may have criticised me six years ago, but in the intervening period they seem to have overcome the sense that it is absolutely impossible to talk to these groups. The Israeli and US governments have now both negotiated with Hamas on the release of Gilad Shalit, the kidnapped Israeli Defence Force corporal, and over the recent ceasefire in Gaza, albeit indirectly. In Afghanistan, after a long mating ritual, the US government finally negotiated with the Taliban for the release of Sergeant Bowe Bergdahl a few months ago in return for freeing five prisoners held in Guantánamo Bay.

Velupillai Prabhakaran, leader of the Tamil Tigers
Velupillai Prabhakaran, the late leader of the defeated Tamil Tigers. Photograph: AFP/Getty Images

Even al-Qaida no longer seem to be completely beyond the pale. Eliza Manningham-Buller believes that "to say that you're never going to speak to them or never going to try to, I think that's foolish … we're obviously a great deal away from anything that you could call a negotiation, even if that were possible, but to think about these questions and to make efforts to try and have those conversations must be a starter." So it turns out that we are ready to talk to these fourth generation groups despite the assertion only a few years ago that we could never deal with them.

Now we face the group that calls itself the Islamic State (Isis), the latest terrorists to confront us. And yet again we have met them with an emotional response based on the horror deliberately generated by their acts. We agree to bomb them and insist we will never speak to them because they are quite unlike any terrorist group we have ever met before.

Of course there are differences. Their violence is more grisly than al-Qaida, and unlike many previous terrorist groups they, and Boko Haram in Nigeria, are taking and holding territory. This time, however, we should stop and consider what we have done in similar circumstances before. We need to work out a longer term strategy for dealing with whatever threat they pose, rather than opting once again for a kneejerk response to satisfy opinion polls. That strategy will certainly include security measures – if the terrorists feel they have the prospect of winning, they will just carry on fighting.

But we will also need to address the grievances of the Sunni community in Iraq and to separate out the ex-Baathists and former members of Saddam's army, who give the movement its real power, from the jihadis. And while Isis may not want to talk to us at the moment, we need to start building a channel to them, as we did with the IRA in 1972, so we can communicate. At some stage, we will need to negotiate with violent Islamic extremism, whether in this form or another one, if their ideas continue to have political support and we want to find a lasting solution to conflict in the region. They are unlikely to simply fade away. We need to bear in mind that such negotiations do not usually succeed at first; they have to go through many iterations, and an agreement is usually reached only when a mutually hurting stalemate exists, in which both sides realise that they cannot prevail militarily.

* * *

Nor should we expect that we are going to defeat terrorism for good even if we succeed in dealing with al-Qaida and Isis; there will be new groups and terrorism will always be with us. It is the ugly twin of democracy and has grown up with it. If minorities cannot get their way by the ballot box they will sometimes resort to extreme violence to attract attention and find satisfaction for their demands.

Yet we continue to approach terrorism as if it is something that can be solved or cured. Some experts look for a solution in technology. Drones, jungle-penetrating radar and electronic counter-measures can help combat terrorists for a while but the terrorists soon work out a way round the counter-measures and technology helps them develop more deadly means of attack. The use of mobile phones and social media can make it easier to track terrorists but it also makes it easier for them to organise without being detected. There is not going to be a miracle cure for terrorism and we should stop hoping one will turn up. The solution lies in the tools we already have in our hands – fighting and talking. Success depends on combining military force with offering a political way out.

Israeli soldier Gilad Shalit
The Israeli soldier Gilad Shalit with his father on the day of his release from five years' captivity in Gaza. Photograph: AP

The one thing I have learned, above all else, from the last 17 years is that there is no such thing as an insoluble conflict with an armed group – however bloody, difficult or ancient. Even the Middle East peace process, which has stuttered on for decades, will in the end result in a lasting agreement. The fact that it has failed so many times before does not mean that it will always fail, and an eventual settlement will be built on the past failures and the lessons learned from these failures, as was the peace in Northern Ireland.

It is remarkable how quickly a conflict can shift from being regarded as "insoluble" to one whose solution was "inevitable" as soon as an agreement is signed. Beforehand, and even up to a very late stage in the process, conventional wisdom states that the conflict can never be resolved; but before the ink is dry on the agreement, people are ready to conclude that it was inevitable. They put it down to outside events like the end of the cold war, to the effect of 9/11 or to changing economic circumstances. But this conventional wisdom is wrong.

Just as no conflict is insoluble, nor is it inevitable that it will be resolved at any particular moment in history. Believing that a solution is inevitable is nearly as dangerous as believing a conflict cannot be solved. If people sit around waiting for a conflict to be "ripe" for talks to start, or for the forces of history to solve it for them, then it will never be resolved. If the negotiations are handled badly, they will fail, which is why it is worth trying to learn from the experience of others. Dealing effectively with a terrorist threat requires political leadership, patience and a refusal to take no for an answer. What we need are more political leaders who are capable of remembering what happened last time – and prepared to take the necessary risks.

Jonathan Powell is CEO of the charity Inter-Mediate which works on armed conflicts; his new book Talking to Terrorists, How to End Armed Conflicts is published by Bodley Head