Four Australian women, who have been in the al-Roj camp in Syria since the fall of the terrorist group, and 13 children arrived in Sydney last month following a secret operation.
Opposition Leader Peter Dutton said there had not been "an ounce" of consultation with the communities where the families would be resettled.
He is concerned the government is acting against national security advice.
"I don't believe it's consistent with the security advice because (it) must have done a 180-degree flip from when we were in government not that long ago," he told Nine's Today Show on Friday.
"There's a political decision that's been made here."
Education Minister Jason Clare said the Liberals repatriated families while they were in government, a decision overseen by Mr Dutton while he was home affairs minister.
"I live in Western Sydney. I want our community to be safe," Mr Clare told Seven's Sunrise on Friday.
"I trust our national security agencies and we do this on their advice.
"For the (Liberal-National) coalition to now say that they are against it makes them hypocrites. They did exactly the same thing."
But deputy opposition leader Sussan Ley said Mr Clare was being misleading, noting the coalition government only brought eight orphaned children home on a humanitarian basis, not women.
She said the difference was "the women that are now coming back deliberately left Australia to join a terrorist state ... that is different to repatriating on a humanitarian level."
Further repatriations did not occur under the coalition following advice from security agencies.
Ms Ley said Western Sydney communities had raised concerns about the lack of consultation before the resettlement.
"There has been so little communication with the communities into which they are now moving. It is just not good enough," she said.
"These are terrorist regimes (that) have threatened Australia's way of life. We have to be extremely cautious."
Ms Ley said the government was withholding national security advice from responsible agencies.