Worries include broken windows and impacts to the local environment, tourism and fishing industries.
One of the confounding things about astronomy is that simple dust is an obstacle that astronomers must work hard to overcome. In a Universe that contains beguiling things like supermassive black holes, amino acids on the surface of comets, and puzzling, powerful bursts of extragalactic radio waves, it's somewhat humbling that simple dust particles require so much effort to deal with. One of the reasons the powerful JWST was built is to deal with this dust.