It's an odd lyric, from the POV of a person who's been caught in an extramarital (or extrarelationshippal) affair & is basically saying "c'mon, don't make a big deal out of this" to the partner.
The version I know best is sung by Ella, who does change the pronouns, making it feel a bit different from the feel of the cheating man who's saying "she means nothing to me" to the (omigod) woman who fooled around & expects forgiveness to come easily, with the lilting melody & lite feel of the whole business:
I must confess what you say is true
I had a rendezvous with somebody new
That's the only one I ever had
Baby, baby, don'cha go away mad
Cheatin' shows and it never goes
You got a reason to be mad, I suppose
But he only was just a passin' fad
Baby, baby, don'cha go 'way mad
His kind's a dime a dozen
That's not the kind I want
Who'd ever think your cousin
Would visit there in that restaurant?
I understand that you feel upset
Whaddya you say that we forgive and forget?
Come on and kiss me just to prove you're glad
Baby, baby don'cha go 'way mad
The bridge, with the cheesy cousin and restaurant part, is my favorite part to sing.
But anyway, about Ella & lyrics. One of my favorite songs of hers is her version of "Brown Skin Gal in the Calico Gown." I've dug that song since way back when, discovering it on one of my folks' l.p.s. There are other occasions on which she doesn't bother to change pronouns so she'd be singing about a man instead of a woman, but that's a song you couldn't do that to if you wanted. Did it stop her from singing it? No indeed.