Will the swallow birds leave and if so how soon?

The baby's hatched about 2 weeks ago and the nest is on a vent to my kitchen well I now have mites comming in my kitchen and my husband wants to knock it down because we have a 1 year old daughter. I don't want to hurt the babies but I can't have my child getting bitten up either. And I will not have an infestation.