The Dutch prime minister, Mark Rutte, has seen off a challenge from the anti-Islam populist Geert Wilders to claim a resounding victory in parliamentary elections widely seen as a test for resurgent nationalism before other key European polls.

With nearly 95% of votes counted and no further significant changes expected, Rutte's centre-right, liberal VVD was assured of 33 MPs, by far the largest party in the 150-seat Dutch parliament, the national news agency ANP said.

Wilders' Freedom party (PVV) looked certain to finish second, but a long way behind on 20 seats, just ahead of the Christian Democrat CDA and liberal-progressive D66, which both ended third with 19 seats.

"Our message to the Netherlands - that we will hold our course, and keep this country safe, stable and prosperous - got through," Rutte told a cheering crowd of supporters at the VVD's election night party.

After Britain's shock Brexit vote and Donald Trump's presidential victory in the US, he added, the eyes of the world had been on the vote: "This was an evening when ... the Netherlands said 'Stop' to the wrong sort of populism."

Wilders, who had led the polls for the better part of two years and was at one stage credited with a 25% vote share before slumping to barely half that figure on polling day, said it was obvious he would have preferred to have been the largest party.

But he noted the VVD had lost eight seats while he had gained five, and promised to offer stiff resistance. "We are not a party that has lost," he said. "We gained seats. That's a result to be proud of ... And Rutte is certainly not rid of me yet."

Although he ended up with fewer seats than his highest previous total, in 2010 - and met his third successive defeat at the hands of Rutte - Wilders will not necessarily be too downhearted.

Outside government he will not have to compromise and can continue to drag the Dutch debate onto his chosen territory of immigration and integration. Rutte adopted parts of Wilders' rhetoric during the campaign, including telling immigrants to respect Dutch norms and values or leave.

"Wilders did not want to enter government," said André Krouwel, a political scientist at Amsterdam's Free University. "What he wanted - and he's pretty much already achieved it - is for the two mainstream rightwing parties … to say and do what he wants. In a sense, he had already won the elections."

Dutch coalition options

A first-place finish for the anti-immigration, anti-EU PVV would have rocked Europe. In France, the far-right leader Marine Le Pen is expected to make the second-round runoff in the presidential election in May, while Germany's Eurosceptic Alternative für Deutschland (AfD) is on target to win its first federal parliament seats later in the year.

Relieved European politicians were quick to applaud the Dutch results. A spokesman for Jean-Claude Juncker, the European commission president, hailed "a vote against extremists" while the French foreign minister, Jean-Marc Ayrault, tweeted: "Congratulations to the Netherlands for halting the advance of the far right."

The big winners were the pro-European leftwing ecologists of GreenLeft, who leapt from four seats to 14 and could conceivably enter a ruling coalition. But the social democratic Labour party (PvdA), Rutte's outgoing coalition partner, slumped from 38 seats to a historic low of nine.

Mark Rutte says Dutch people have rejected 'wrong sort of populism'

Rutte is now set to begin the often lengthy process of building a new coalition, most likely based around the VVD, CDA and D66 - a combination that falls five MPs short of a 76-seat majority, leaving him seeking a fourth coalition partner.

Wilders, who pledged to "de-Islamise" the Netherlands and take it out of the European Union, was widely seen as unlikely to enter government however he fared, since most other parties - including the VVD - had vowed not to enter a coalition with the PVV.

Turnout was high at 80.2% in an election both Rutte and Wilders cast as a test of whether the Dutch wanted to end decades of openness and centrist politics and opt instead for anti-immigration nationalism.

"The Dutch have woken up in a 'normal' country, as Prime Minister Mark Rutte puts it," the NRC newspaper wrote in its editorial "There was no populist revolt." But the paper pointed out that the big lesson was that governing does not pay.

"The outgoing government presented a dream budget this year," it said. "What should have gone up went up, what should have come down came down. The Netherlands is one of the best-performing countries in the EU. And still the outgoing coalition was punished, severely. For voters, apparently, politics is about more than the economy."

Geert Wilders on Dutch election day: 'Genie will not go back in the bottle' - video

The political commentator Roderick Veelo, however, cautioned against assuming the populist far-right challenge was over. "Rutte is still standing, but so too is social discontent about uncontrolled immigration, failed integration and the power of Brussels," he said.

"That is not going away. The broad coalition that will govern this country soon must show responsibility and courage on these subjects and go to work with real solutions. Only when that happens will the populist revolt die a quiet death."

Rutte was also thought to have benefited from his firm handling of a fierce row with Turkey over the government's refusal to allow Turkish ministers to address rallies of Dutch Turks before a referendum next month on plans to grant Turkey's president, Recep Tayyip Erdoğan, sweeping new powers.

The diplomatic standoff did no harm, either, to Denk, a new party aimed mainly at the Turkish and Moroccan communities, which picked up three seats. Also faring well in the record line-up of 28 parties were the Party for the Animals, with five MPs, and the Eurosceptic Forum for Democracy with two.

ANP said its final forecast was unlikely to be available until later on Thursday or even Friday because several large municipalities including Amsterdam, The Hague and Utrecht would not finish counting all votes until then. But it said it expected no further changes to the outcome. Official results will be published on 21 March.