Australian Prime Minister Anthony Albanese stared down calls to postpone a national vote on an Indigenous advisory body to Parliament, as flagging polls and growing opposition spark fears among activists that the proposal might be heading for defeat.

Albanese was widely expected to announce the date for the referendum at the four-day Garma Festival on Saturday, the country’s largest annual festival of Indigenous Australian culture. However, he instead used the event to rally support for the “Yes” campaign.

“I can promise all of you – and all Australians – there will be no delaying or deferring this referendum,” Albanese said in a speech to Australia’s largest Indigenous festival on Saturday. “We will not deny the urgency of this moment. We will not kick the can down the road.”

Australia will vote in a national referendum before the end of 2023 on whether or not to embed an Aboriginal and Torres Strait Islander Voice to Parliament in the country’s constitution. If it is successful, it will create an advisory body which will give the government feedback on policies affecting the Indigenous population.

Read More: Why Australia Plans to Vote on an Indigenous ‘Voice’: QuickTake

However, opponents for the Voice have called for the compulsory vote to be delayed until at least next year, as polls show the “No” campaign is gaining traction. Albanese’s main political rivals in Parliament have said the plan lacks details, is divisive and won’t do enough to improve the lives of Indigenous Australians.

Albanese says the Voice will help facilitate improvements in the quality of life for the nation’s Aboriginal and Torres Strait Islander populations, which are among the oldest continuing cultures in the world. After colonization by British settlers in the 18th century, they’ve been far more likely to die earlier, drop out of school or be incarcerated than the rest of the population.

After coming to power in May 2022, Albanese’s center-left Labor party promised to hold a national referendum on the proposal for a Voice to Parliament. The advisory body was initially suggested by a gathering of Indigenous elders in 2017.

However, over the past six months, voter support for the Voice has declined. In February, a survey by Newspoll said 56% of voters were in favor of the advisory body but by July that number had declined to 41%. Australia’s constitution can only be amended through a national referendum which requires a majority of votes and a majority of support in the six states to succeed.

Albanese was welcomed warmly by a crowd of hundreds as he entered the Garrtjambal auditorium to speak at the Garma Festival on Saturday.

“We can get this done, together. And we can get this done, now. Because if not us, how? And if not now, when?” Albanese said.