The problem is that in UNA messenger doesn't work in popups. It works on a dedicated page.

It is possible to modify the code above to force it to open a page in a new popup window - but nowadays such popups are being blocked by modern browsers by default (asking for permission first to show a popup) so that will be a bad user experience.

Another alternative is to have a link instead of a button, which will open a messenger in a new tab. So you can try this code instead:

if (
    !isset($aContentData['profile_id']) ||
    ($sModule != 'bx_persons' && $sModule != 'bx_organizations') ||
    $aContentData['profile_id'] == bx_get_logged_profile_id())
return ' ';
return $oMenuObject->getUnitMetaItemLink(
    'Message',     
    ['href' => BX_DOL_URL_ROOT.'page/messenger?profile_id='.$aContentData['profile_id'], 'target' => '_blank']
);