I don't thing there is an ideal number. Greater than one is a start.
I think you really have so many reasons for collecting watches for lack of a better term. Without getting all technical you could look at it from a category perspective with a goal of wearing your watches. I am not addressing the non watch wearing safe queen category here.
You need a dress watch or two, a tool watch or two, a swim/dive watch or two, a chrono or two, a complicated watch or two, a "I don't know why I bought this watch" or two, a vintage watch or two, a pocket watch or two. That's about eight to sixteen watches. This a good start for a horizontal approach. Next you go vertical in the category or two you favor most and add another eight to ten watches.
You could take a brand specific approach and remain faithful or play the field and remain non committal and free wheeling.
I think from here the sky is the limit.
Try and change watches at least once a day i.e. when you get home swap your watch for another and keep the stable happy.
Soooo much fun.
Bill