Additional params can be added to sys_transcoder_filters table, find the transcoder where you want to add it, for example 'bx_videos_video_webm', then edit this row by adding ffmpeg_options param to filter_params field, where serialised array is stored, for example for 'bx_videos_video_webm' it would be:

Before adding custom options:

a:2:{s:1:"h";s:3:"480";s:10:"force_type";s:4:"webm";}

After adding custom options:

a:3:{s:1:"h";s:3:"480";s:10:"force_type";s:4:"webm";s:14:"ffmpeg_options";a:1:{s:7:"threads";i:0;}}