"Esme is supposedly a boys name. But could it be a girls name or is that too weird?"
Although it's true that Esme was originally a boys' name (introduced to Scotland from France by a cousin of James VI), I think it may have gone the same way as Lindsey these days, and is mainly see worn by girls.
While still a rare name in the U.S.A., in the year 2000 the Social Security Dept. reported a total of 38 baby girls having been named Esme. In 1999 there were a total of 45 baby girls named Esme.
In both the years 2000 and 1999, there were fewer than 4 (if any) baby boys named Esme, as reported by Social Security.
Note to Mike C -- Looks like you may have to change your entry on Esme to indicate that it's both a masculine *and* feminine name.