Esmé is the male form (EZ-may), Esmée the female form (EZ-mee) in French, though in the modern era that is observed less and less and Esmé now figures most often as a female name. It's Old French, based on past participle form of the verb esmer 'to love, esteem' (Latin aestimare 'to value, esteem') which was absorbed by the verb amer to became the modern French aimer as in J'aime Paris. Based on nothing more than it's visual resemblance, some try to connect it with Esmeralda but it's baseless folk etymology, nothing more.
The name was brought to Scotland in the 16th C. and has been used occasionally as a male name ever since, often without the accent (which gives pronunciation guidance in French but in English is largely ornamental, the absence of it doesn't change the English pronunciation at all).
In Scotland the female form Edmé arose. The reason for the change from S to D isn't clear but is generally put down to the influence of co-existing given names Esmond and Edmund. Edmé and Edmée are exclusively English/Scottish in use, in spite of the Frenchified accent marks.
Famous bearers - Esmé Stuart, cousin of James VI of Scotland (1542-83). Director Esmé Collings. Authors Esmé Raji Codell and Esmé Ellis. Actually Irish dramatist Lennox Robinson's real name was Esmé Stuart Lennox Robinson (the first Esmé Stuart was 1st Duke of Lennox...Robinson's folks were clearly fans). Esmé features in Saki's Chronicles of Clovis. Esmé dolls.