Asteroid apocalypse: How big must a space rock be to end human civilization?

How big a space rock must be to no longer be just a mesmerizing sight? What size is big enough to wreck a whole city? And a continent? And what else does the damage depend on?