The white paste on the piece you pulled off is adhesive. The brown is just the outer layer on the plasterboard which has peeled off.

But all you need to focus on is that there is nothing to worry about apart from your anxiety