On one hand, as you've said, the case metal determines if I buy the watch. I wear white metals (mostly SS, sometimes WG) at my workplace as few wear RG or YG. So white metals allow me to blend in a little better, and SS is more practical.
On the other hand, sometimes you buy a watch simply because the case and dial go well together as a combination.
An example would be my FPJ Chronometer Bleu where the grey tantalum case goes so well with the blue dial. I wouldn't buy a watch just because of a tantalum case, but it was the combination that won me over.

Ditto for the warm brown of my Nomos 'mocca' and the RG case. It wouldn't work as well with a YG or SS case I think.
