Hello!

Yes, it is not a problem to install Jot Server and Jitsi on the same server. You just need to put correct link for Jot Messenger to connect. 

You may install Jitsi server and then Jot Server, but for Jot Server you need to have another opened port on your server to connect and it should be set in Jot Messenger settings like https://jitsi.example.com:5000. In this case port with number 5000 should be opened on your server for external connect.