The 'no' vote had proved successful, broadcaster RUV reported on Sunday, in a result that will disappoint Brussels.
Some officials had cast Iceland as a low-risk case that could help ease scepticism toward EU enlargement and strategically strengthen the bloc when Arctic security is in focus.
The government had framed its membership plan as a "now or never" moment, and Prime Minister Kristrun Frostadottir told voters in August that a "no" would put all EU speculation aside.
In earlier counting, based on the first 152,000 votes, 50.1 per cent were against the government's proposal to restart negotiations, while 49.9 per cent were in favour.
Voting ended at 10pm on Saturday (8am AEST Sunday), kicking off a complex operation to bring ballots from towns and rural polling stations by plane, boat and car.
On Friday, a Gallup poll indicated a narrow majority opposed the government's proposal of reopening negotiations, a shift from earlier surveys that pointed to a narrow lead for supporters of renewed talks.
'Yes' campaigners backing talks have said joining the bloc could ease high interest rates and offer more security in a world unsettled by war in Ukraine and US President Donald Trump's rhetoric about annexing fellow Arctic island Greenland.
'No' campaigners have argued that EU membership would threaten the island's sovereignty and fishing waters.
The referendum campaign has fuelled debate about Iceland's place in the world, Prime Minister Kristrun Frostadottir told reporters after casting her ballot in the capital on Saturday.
"This is a great day, we've been waiting for this for years to be able to have a vote," Frostadottir said, adding that her government would continue its work regardless of the outcome.
Though a 'yes' vote would have opened up discussions with Brussels, under Icelandic law, there would have to be a second referendum, likely several years later, on whether to join the EU.
Iceland first applied to join the EU in 2009 amid a financial crisis that hit the country's small, exposed economy but talks ended in 2013.
Geopolitical turmoil including war in the Middle East and US import tariffs has brought urgency to an issue Icelanders have debated on and off for decades.
Debate inside Iceland has focused more on interest rates, fishing rights and the cost of living than on security.
Iceland has grappled with persistently high inflation rates and interest rates that have made mortgages expensive.
Supporters of a "yes" vote have said EU membership, and possibly adoption of the euro, could bring relief. Iceland's central bank interest rate is 8 per cent, compared with 2.25 per cent at the European Central Bank.
"It would not solve Iceland's structural faults, but it could become a catalyst to a more healthy economy," said chief economist Vilhjalmur Hilmarsson at Viska, one of Iceland's largest labour unions.
Opposition Leader Gudrun Hafsteinsdottir, who has led the 'no' campaign, argued the economic case was overstated.
"These are not problems that Brussels will solve for us. These are problems that Icelandic politicians must solve," she said.
