Thousands of people turned out across the country at the weekend as protesters were met with counter-demonstrations in Sydney and Melbourne.
The rallies have been widely condemned because of the presence of neo-Nazis, but politicians including One Nation senator Pauline Hanson and Queensland MP Bob Katter joined marchers.
While only a small contingent of the people who showed up identified as white nationalists, terrorism expert Levi West said there was an overlap that extremists could exploit.
"A rally that has a presence from a group like the neo-Nazi organisation presents a prime opportunity for recruitment," said Dr West, a research fellow at the Australian National University.
"People whose grievances are potentially based in conspiracy theory or misinformation can quite easily be drawn towards the simplicity of the ideas that are embedded in neo-Nazi ideology as providing some sort of solution or solace for the thing that they feel a grievance about."
Mr Albanese said while there would have been "good people" who attended the rallies to express their views, the protests were about sowing division.
"What we have here is neo-Nazis being given a platform," he told ABC News on Monday.
"The tone, of course, of much of the rallies was unfortunate - is the best way that you could put it - but hateful in some of the extreme examples."
Asked if resentment towards migrants was rising, Mr Albanese said those elements had always been present.
"What we need to do is to be vigilant about it," he said.
"It's much easier to organise through social media ... for people to have their views reinforced, often ... based upon things which aren't just fact."
The federal government is close to releasing its permanent migration target for 2025/26, which is expected to remain around the 185,000 mark.
"I think it is very soon," minister Amanda Rishworth told Nine's Today show on Tuesday.
"We are working with states and territories on that - that target will be released shortly."
Opposition Leader Sussan Ley said the weekend rallies had been "hijacked by violent neo-Nazis spouting hate and racism".
"Extremism has no place in Australia," she said.
Ms Ley said the coalition was ready to work with the government to repair social cohesion in the community.
Opposition multicultural affairs spokesman Paul Scarr said the coalition supported sustainable levels of migration, but the debate about the issue needed to be "terribly careful".
"If we don't have these debates in a considered and measured way, then unfortunately, the extremists will use the lack of debate and discussion to obtain a foothold in the debate, and that's something we really need to guard against," he told ABC Radio on Monday.
