Hello!

So can i setup a private server and then point that to the website? or does the jotserver and the una platform need to be on the same server? Yes, you can install Jot Server to any server separately from UNA site, you just need to set jot's server's configuration properly and don't install Jot server on servers without https certificate in case if your site has it. 

2nd question is what are other alternatives I can use besides jotserver? If you mean any another servers apps to use for Messenger module, then no. There is no such kind of alternatives yet.