Honestly, a good part of the hostility is Byerly's honest dislike of children. Part of it is transitioning, though, into a calculated, considered sort of possibility. He comes back to the mouth of the cave, and crouches, and starts arranging the stones in a pattern.