Why does it feel condescending when a man calls you sweetheart, but not when an older woman does?

I get the 'ick' whenever a man calls me 'sweetheart' (other than my dad, older male relatives, and my husband). It makes me feel like he considers me a child or someone not worthy of respect, but when a woman in her 50s+ calls me that I get the warm and fuzzys. Same when an older Latina woman calls me 'mama.' I don't feel condescended, I feel cared for. Why is that?