I just finished reading Broken Kingdoms by NK Jemisin and it got me to thinking about ethnicity. The main character is black. I found no problem relating to her. And maybe that is because she is Maro. A culture that only exists in the book.
I was raised to be tolerant of other religions and skin colors. But I also remember the riots in the 60's and hearing gunshots and knowing that my parents were afraid. That's a powerful realization for a child.
A long time ago I had a job as an assistant manager of a quick-print store. One of my supervisors was a 6'+, solid muscle, black man. I'm 5' and Irish pale. After some awkward encounters we sat down to talk and I confessed to a being afraid of black men in general because of those memories. He confessed that he remembered the riots, too, and as a result feared white people. We had a bit of a laugh, and proceeded to work together more easily.
When I went to college my dorm roommate was of Italian descent. She had some strong traditions and expectations. As a result, she was so totally different from me, that I was convinced I couldn't understand any other culture, regardless of the skin color. It was a shock to see that other people did things differently, ate different food, celebrated different things.
And yet, in fantasy books, I have no problem relating to elves and hobbits and demons and people who are blue or green or black. I've always loved learning about other cultures. I guess when it's in your face, it's a different thing all together.