several (well, actually five with the sixth on the way: UN Perp GMT +/-, IWC Portugiese, JLC Master Perp, Andersen Secular Calendar and Parmigiani Toric Perp.) I see no problem wearing any of the watches. With the exception of the UN, yes, the only risk is that they can not be set back if the date is passed (actually I believe that the Toric can but it is a pain). I think that any of them are robust and are no more "delicate" than any other watch. I would not be afraid to handle or wear any perpetual at any time or under most conditions (within reason).
I do not understand, however, the perpetuals that are hand wind with only the standard 40 hr power reserve. To me that is just an inconvenience and takes away from the practicality and functionality of the watch. The same, I might add, could be said about moon phase watches with the same 40 hr power reserve. The watches with longer power reserves are certainly more convenient and those that are automatic can, and I do, be kept on winders so that they are ready to go at a moment's notice. Of the hand wind ones, the JLC is easily adjusted as the single pusher handles all functions and the day/night indicator is a safety feature that prevents inadvertent advancing beyond the day. The IWC corrects all the functions with the crown.
Mitch