It’s what i have noticed when i use my 21mm which i usually set at f/5.6. With my 50mm lux which i tend to use wide open, it can be a challenge. Also, if you’re shooting from afar, like mountains for example, you can just set the focus to infinity (assuming that the lens is well calibrated - which they usually are) and the scene would be more or less sharp and in focus.
So based on your M preference but with a certain degree of doubt with regard vision, I think it’s worth the gamble to get into an M. The minute your eyes experience that pop when those focus patches align, it can be one of the nicest feelings in the world.. like a well executed gear shift if you will 