In modern psychoanalytic theory, castration anxiety is not about a man losing his penis. It is about masculine identity as a source of power. The reason the female object is threatening is because, in more modern representations, sexuality equals power. In this context, men derive pleasure from their dominant sexual role. If the female is stealing the power, that is threatening and therein lies the castration anxiety. It has nothing to do with the organ itself.
In my personal theory, men subconsciously think during a moment of anxiety they've lost their penis already, and need to be reminded that it is still there.
There is another 'gaze' that is ignored, which is one of my personal favourite cafe pastimes: watching men watch women. There really isn't any better way to see frustration and impotence manifest as facial features than in that setting. It's fun watching some scruffy tool gaze at a woman because there's always a moment when he realises he can never so much as talk to her, let alone have her. It flashes across his face as a pouty lower lip, a clenched jaw and a slight bowing of the head. (On second thought, maybe the castration anxiety really is about men feeling they've lost their penis. More study is needed. Off to Starbucks!)
Women don't realise the power they have to remind men of their utter uselessness. I would like to see them abuse this power more. I'm only joking a little bit.