Too often it is pronounced Alivia, which to my Yorkshire ears sounds like I live ere! My husband pointed this out to me, ruining the name for me forever. So I prefer Olive, because it actually gets pronounced with an 'Oh'.
For some reason, the pronunciation of this name sounds so much like "I'll leave ya" to me and it leaves a bad taste in my mouth. Otherwise, it's a beautiful name.