Two families who knew each other were at the Royal Daylesford Hotel sitting in the roadside beer garden when a BMW SUV came down the street, mounted the kerb and hit patrons about 6pm on Sunday.
A 38-year-old Tarneit man and his 11-year-old son were killed at the scene while his 36-year-old wife and six-year-old son were hospitalised.
A 30-year-old Point Cook man and his partner, a 44-year-old woman, were killed along with her nine-year-old daughter.
An 11-month-old boy was injured but is in a stable condition at the Royal Children's Hospital.
A 43-year-old Kyneton woman and a 38-year-old Cockatoo man were injured and both flown to the Royal Melbourne Hospital.
The Hepburn Shire Council will hold a vigil on Monday night for mourners.
Police are waiting on DNA results to confirm the identities of the victims and the coroner has been notified.
A steady stream of devastated mourners, including families and community members, arrived throughout the day to lay flowers, teddy bears and leave messages of condolences at the site of the crash.
"(We) are broken. Rest in peace," one message read.
Chief Commissioner Shane Patton said the crash was an "absolute tragedy" and commended first responders at the scene, including officers who heard the crash from the local police station and ran over.
"But it comes at a cost as well," he told reporters on Monday.
"They will take that scene with them for many, many years - some may never lose that scene, it may haunt them forever."
The 66-year-old driver is expected to remain in hospital for shock and non-life threatening injuries overnight.
Police expect to speak to him on Tuesday.
Mr Patton said the man returned a negative breath test at the scene and he is not known to police.
Police are still trying to piece together what caused the fatal crash in Daylesford.
Crash reconstruction specialists are trying to piece together what happened.
Victoria Ambulance regional director Trevor Weston said emergency crews were confronted with a chaotic scene.
He commended the efforts of people who came to the aid of the injured until paramedics arrived.Â
"(Injuries were) quite traumatic for a number of those patients, particularly the four that were airlifted to hospital," he told reporters on Monday.
Some of the initial crews were from the area, he said.
Paramedics will be offered support in coming days and weeks.
"Any incident involving children is that next level of distress for our paramedics but the crews that responded last night did so professionally and provided the best care they could for all of those patients," Mr Weston said.
Hepburn Mayor Brian Hood said Daylesford would have been crowded on Sunday, with many people enjoying the warm evening over the unofficial Melbourne Cup long weekend.
"This will send shock waves through the community for some time," he told ABC TV.
First responders to the Daylesford crash were confronted by chaotic scenes.
The council acknowledged many witnesses will be affected and will work with local groups to offer support.
"This is a terrible event for our close-knit community," it said in a statement.
Police have asked the public to avoid Daylesford so emergency services can access the scene.
Politicians including Premier Jacinta Allan, Health Minister and local state MP Mary-Anne Thomas, Opposition Leader John Pesutto, and local federal MP Catherine King thanked those who went to the victims' aid and offered their condolences to those affected.
Anyone who witnessed the crash or has dash cam or CCTV footage is urged to contact Crime Stoppers.
