One of the reasons that we don’t get "glasses that you wear" confused with "glasses that you drink out of" is context. That would also be how the stairs and ladders thing escapes confusion.
"Are you wearing your glasses?" can only have one meaning, because one doesn’t "wear" a drinking glass.
"Can I refill your glasses?" when said by a host or hostess with bottle, carafe or jug in hand is also fairly unambiguous, when he or she is addressing a group of guests.
In most circumstances, one would refer to a (drinking) glass in the singular, which also helps distinguish between them. "Where is your glass?" needs no disambiguation from "Where are your glasses?".
After a few glasses one might well enquire "Where is the bathroom?" and be told by the host "Up the stairs and to the left". If we used the same word for "stairs" as for "ladder", it would be relatively clear that "stairs" were meant in this context, as people rarely have ladders standing around indoors, and one even more rarely finds bathrooms at the top of them. 
As we learned at Bible college, context is everything
(Although back then it was in the context of translating a given word using a concordance). The acronym we were taught was COMB - context, other, meaning, background. Most of the time we unconsciously do this as we hear someone speak. (The "other" referred to in this context is "other verses of scripture which support this interpretation".)
With reference to the glasses one wears, "specs", it amuses me that we continue to call them such even though in most cases they are no longer made of glass.
However, if we referred to them as "plastics", that would open up an entirer ‘nother can of worms, which I’m not going to even speculate about now.
Azh
(remembering the lens grinder who fell into his machine and made a spectacle of himself)