I was a huge Scooby-Doo fan growing up (still am, if I'm really honest) so all I can think of when I hear this name is a ginger, purple-clad, animated teenager! Not a terrible association but a strong enough one to put me off using this name personally.
To hear or see this name on a young child I think I would be rather pleased. Daffy makes me think of Daffy duck who is, of course, male and also not very nice for the most part but I think Daffy could be cute and would grow on me with the right child.
I quite like the sound of
Daphne, it's soft enough without being too flimsy.
http://www.behindthename.com/pnl/117870?sort=rank&pop=ew^Top 2 =
Alice and
James ~~ Bottom 2 = Stark and
Indiana (m)