"Won't drag you by your hair," Daryl says, and then there's a note of bitterness-- "That's what you did."

Okay, it's definitely childish, but Daryl still feels annoyed at how he becomes the bottom rung of importance when other people feel like they know best. Everybody talks big, but when push comes to shove, he gets pushed and shoved around. Wouldn't bother him so much if everybody didn't keep pretending it weren't so.