The family had booked for a week's holiday in a Bulcock Beach apartment with their three children, aged between four and nine.
But the couple were forced to call Workplace Health and Safety when they became concerned about the "white dust" that covered their car and was floating through the air.
Workers were busy removing and renovating gutters and roofing on an older building next door that appeared to to contain asbestos.
The family had arrived at the unit on the Saturday and the work was ongoing until the Thursday when they called for an investigation.
An investigator arrived the same day and shut the operation down.
Under Queensland law, a contractor removing more than 10 sqm of asbestos or asbestos containing material must have an asbestos removal license issued by WHSQ.
"The inspector has taken enforcement action by issuing a Statutory Notice to stop unlicensed asbestos removal work," the spokesman said.
"The inspector has also provided advice with regard to engaging a competent person to check the area is free of asbestos waste."
But the knowledge the operation was stopped had not eased the family's concerns
"We were worried about all the dust flying around," they said.
"Our car, parked underneath the building was also covered in white dust."
And there were "buckets" of asbestos-looking material left lying around the premises.
"The young guy on the job breaking up the asbestos did it without a mask too," she said.
The workplace inspector had reassured the family the white dust floating around "wouldn't all be asbestos" however, even small quantities can be harmful.