Well, his parents might just be old school, in the sense of causing pain through distance, and expectations, and conditional love. Plenty of damage to be found there - abuse isn't always what we read about in the papers. Parents can be proper and charming and even likable, and still cause their children no end of pain and scarring. It's the lack of positive experience, of unconditional love, that causes the most harm in my experience. Sherlock struggles with empathy, as though he simply never learned, rather than being incapable of it. Part of that might be the isolation that true genius creates, but it makes all kinds of sense that family played a role in it, too. It's your story, and you can change the make of the gun :-)