That's because the Shoutbox is nothing more than a "stand-alone" messenger block and not a "separate" module. It is really no different than you going into your messenger app, creating a new talk, and manually adding people to it for a group chat. Only difference is, you can put a ShoutBox block on certain pages. 

As Alexey mentioned you can set the privacy levels to who has access to that specific page.