Why is centripetal force directed toward the centre and not outward?

The outward pull you feel is a common misunderstanding. For any object moving in a circle, the direction of motion keeps changing, which means the velocity vector keeps changing, so there must be an acceleration, and that acceleration points toward the centre of the circle. The force causing it, the centripetal force, is therefore also directed toward the centre. In the case of the stone, the string tension pulls the stone inward toward your hand, providing exactly this force. What you feel as an outward pull is the string's reaction on your hand and the tendency of the stone to fly off straight (inertia), not a real outward force on the stone. If you let go, the stone flies off along a straight tangent, never outward along the radius, which proves there was no outward force.