Patricia is loads better. I actually really don't mind it. I'd personally go with a completely separate nickname such as
Trish
Patty not only comes across as sounding too nickname-ish, I can't help but to picture a
Patty on an older woman (50's or above) she's shorter in stature and has a large girth, ugly, frog eyes (bulging type of eyes) really rude, grumpy, too serious about every aspect of life. A teacher or principal hated mutually by all students.
Funny,
Polly is actually my mother's name.