All Manhattan was suffering urban decay during the 70s/80s?

It’s widely known the situation of places like Midtown, SoHo, Lower East Side, Harlem etc before revitalization/gentrification.

But I was wondering what was going on with spots like Upper East/West Side. Never heard any story of them during these decades - if it was also struggling or so. Some rich people didn’t move to suburbs and preferred to stay in the messy Manhattan at that time?