Actually, that was a part of the last episode that really worked … when Merle saw the scars on Daryl’s back and we realized that they had both been horribly abused children.
Those plot twists are why I love this show.