Hello!
Looks like you have internet with very weak transit capacity. When there are only two participants in conversation, users communicate directly bypassing main server and connection depends only on their internet connection.

May be you have internet with weak transit capacity. During conversation you may see connection indicators near each user which allows to understand if they have bad signal. Also there are should be tool tips with recommendations about what to do when there are problems with connection during connection.
You may measure your internet bandwidth using any available service in internet.

Also you may try to play with settings during Jitsi confenecase and decrease video quality, just to understand where is the problem.