Two family groups who knew each other were at the Royal Daylesford Hotel sitting in the roadside beer garden when a BMW SUV came down the street, mounted the kerb and hit patrons.
A young boy, a teenage girl, two men in their 30s and a woman in her 40s were killed.
Two children were also injured - an 11-month-old boy is in Ballarat Base Hospital in a stable condition.
Another boy, believed to be about six, is in the Royal Children's Hospital also in a stable condition, after initially being listed as critical with broken legs and internal injuries.
Three adults remain in the Royal Melbourne Hospital, a 35-year-old woman is in intensive care in a serious condition, while a 43-year-old woman and a 38-year-old man are both stable.
Chief Commissioner Shane Patten said the crash was an "absolute tragedy" and commended first responders at the scene.
"But it comes at a cost as well," he told reporters on Monday.
"They will take that scene with them for many, many years - someone may never lose that scene, it may haunt them forever."
Five people are dead and others injured in the crash at the Royal Daylesford Hotel.
Police are waiting to speak to the 66-year-old driver, who suffered non-life-threatening injuries.
Mr Patten said he was breath tested at the scene and did not show any alcohol in his system.
Crash reconstruction specialists are trying to piece together what happened.
Victoria Ambulance regional director Trevor Weston said emergency crews were confronted with a chaotic scene.
He commended the efforts of people who came to the aid of the injured until ambulance paramedics arrived.
Police are still trying to piece together what caused the fatal crash in Daylesford.
"(Injuries were) quite traumatic for a number of those patients, particularly the four that were airlifted to hospital," he told reporters on Monday.
Daylesford was a close-knit community and some of the initial crews were from the area, he said.
"You never want to respond to any incident like this but certainly not in your home town," Mr Weston said.
Paramedics will be offered support in coming days and weeks.
"Any incident involving children is that next level of distress for our paramedics but the crews that responded last night did so professionally and provided the best care they could for all of those patients," Mr Weston said.
Hepburn Shire Mayor Brian Hood said Daylesford would have been crowded on Sunday, with many people enjoying the warm evening over the unofficial Melbourne Cup long weekend.
"This will send shock waves through the community for some time," he told ABC TV.
Police have asked the public to avoid Daylesford so emergency services can access the scene.
First responders to the crash were confronted by chaotic scenes.
Premier Jacinta Allan thanked those who went to the victims' aid.
"The reports of the Daylesford car crash are horrific," she wrote on X, formerly known as Twitter.
"My thoughts are with all those injured, and with the friends and family of those who have tragically died.
"To everyone who stepped in to help, and to all the emergency service workers involved - thank you so much."
Opposition Leader John Pesutto also offered his condolences on the platform, as did federal Infrastructure Minister Catherine King and state Health Minister Mary-Anne Thomas, whose electorates both take in Daylesford.
Anyone who witnessed the crash or has dash cam or CCTV footage is urged to contact Crime Stoppers.
