I’d actually have this the other way round. Finding them attractive = appearance, fancying someone = deep in the feels.
Not that I think you could fancy someone you didn’t find attractive though, so I don’t even know what I think. I wouldn’t say someone was attractive/desirable if they’re a dog but have wicked bants.