Maximum concurrent conversions

Hi -

I just installed Plex and MCEBuddy on a AMD 16 core processor Ryzen, and it is loafing at 8 concurrent connections. Watching the processor with Process Lasso shows that it is less than 50% utilized, as low as 20% and as high as 55% occasionally. Is there a reason that I can’t do more concurrent conversions or is it just a GUI limitation?

The machine has a M.2 Samsung SSD, and it hits 350 megabytes a second internally when generating the output file, but that only goes on for about a second. The 16 gigs of RAM are only about 40% used, so the machine overall has easily double the bandwidth for processing as far as I can tell.

Thanks much!

Just GUI, though I can tell you right off the bat that something’s not right. 8 concurrent conversions can max out even the biggest systems with the biggest graphics cards we’ve seen.

When it’s converting even a single file using software based conversions the CPU should be 100% (MCEBuddy is multi threaded). However this would be only during the actual conversion itself and not the other activities (like remuxing and analysis etc). So what profile are you using, do you have hardware encoding enabled (nvidia or intel) and are you looking at CPU utilization during the conversion phase?

Hi -

I have a Asus GTX 1070 TI, which is clocked about the same as the typical 1080 cards, but 15% less cores. Hardware encoding is enabled and working according to the file logs.

The profile is convert to MKV Normal Quality. I have done the integration with PLEX, except for I am having it go through my entire video library (about 6500 items) to check whether there are commercials in files and remove any, and to compress the larger ( mp2 for example) files.

I have watched CPU utilization extensively (logged it for 30 minutes at a time) to determine the most intensive times, etc. The most intense item by far is ffmpeg, when it starts it spins off 32-49 threads per instance, Mencoder spins up 9-12 threads per instance, and Comskip seems to always spin up 7 threads per instance.

I see threads on every core (except the last core, which I have reserved in Process Lasso for Plex tasks), and they seem to be pretty well distributed as far as core utilization goes, but there seems to be something limiting the utilization, and I was assuming that it was possibly a lack of threading when “Analyzing Video Information” for example. I assume that is a function that is occurring within ffmpeg, as there are lots of instances of it (7 right now, along with one comskip). That is the lowest utilization of the CPU when there are lots of sessions doing that.

One interesting note to me is that the overall utilization drops when I drop the concurrent sessions from 8 to 6, so that is why I was asking why not 12 or 16 sessions. I see the utilization drop by about 30% (from a little over half utilized to a little over a third).

Perhaps this is a resource locking thing (spinlocking on something).

Overall the processing is occurring on about 800 videos per day, which seems really fast, and the resulting video seems fine (with the exception of some having audio / video sync problems after comskip).

Thanks!

1 Like

For 8 concurrent sessions that’s almost 400 threads and it doesn’t max out the CPU, I’m impressed.Try this, create a customized profile and include the parameter -threads 100 or whatever the number of threads you want ffmpeg to use and see if that makes a difference.

For Comkip I think you set the number of threads in the INI file, not sure but you can confirm on the discussion forum.

See if that cranks up your CPU utilization. As of the number of concurrent conversions, if you want to crank it up some more, just Stop the engine, open the mcebuddy.conf file, set the max concurrent conversions to whatever number you want (12 or 16) and save the file. Then hit the Start button and it’ll reload the settings (just don’t open the Settings tab and click save as it’ll throw an exception). If that fixes your issue, in the next release we’ll increase the max concurrent conversions.

Are you running a broadcast station?

One other thing, and that is when it is “Analyzing Video Information” and there is some crap in the video file (recorded from antenna, so some files are not high quality), it seems to get stuck on that for a very long (perhaps forever, although I have not let it get stuck more than a day) time. That is another reason why I was thinking that it would be good to have more sessions, so that when some of them get stuck, others can proceed along.

When they get stuck, I note the name, stop the conversions, and delete the files, and re-start. Eventually, it get clogged up with more.

Thanks!

Do upload the file that gets “stuck” for us to analyze to see what’s going on. Multiple things happen during the analyzing phase including analyzing track technical properties to setup/optimize the conversion and also run through entire video looking for cropping information. If cropping is causing the issue, then try to check the box in the expert settings which says disabled cropping and it’ll skip that setup. But anyways do upload so we can look at it.

Hi -

I think I figured out one thing, and that is that Bitdefender was examining the things being written to disk, and I think it was overwhelmed. I told it to not check disk I/O from anything having to do with the video processing and the utilization jumped 25%. So the CPU utilization is now hitting about 80%, averaging about 70%, and bottoming out about 50%. Not perfect, but a significant improvement. I am looking at the Windows defender stuff now to see if it is also resource constrained and if I can get it out of the picture.

Thanks for the futher info, I am going to bump the threads for ffmpeg and see what happens. I bet I can max it out by going to 100%.

Thanks so much for the info!

1 Like

You’re on the right track, at 8 conversions in parallel it may the slowest link that’s the bottleneck like your hard disk. Try to spread out the conversion tasks to different destination and source disks, keep the Temp folder on a RAM drive if you extra RAM to space and you’ll see a HUGE boost in your processing speed.

This should give you a real kicking boost to your conversion, keep your TEMP folder here:

BTW in today’s BETA build you can enter number between 1 and 50 now (that’s a reasonable limit for even the biggest systems)

Goose -

Thanks for your thoughts. I have a M.2 Samsung 512GB NVMe SSD (rated at 8 Gb per second) just for the temp files, and one for Win 10 and Plex. The library is on a RAID 5 disk cluster with 8 4 TB spindles, so the I/O queue (I see 1000 I/Os pending at max) is pretty small when it’s going full blast and gets cleared in milliseconds. At present, I see in excess of 350 megabytes a second going on (for two seconds at a time in bursts), which to me is serious I/O.

I tested out doing a ram disk for temp, 8 GB in size, and besides running out of space with a full load, it seemed about the same speed as the M.2 drives. So I went back to giving the OS all 16 gigs that I have.

I think I am on track to be able to process as many as 1200 videos a day, which is plenty good for my personal video PLEX server. And I think I have identified the bottlenecks, and will be bumping up the thread counts for ffmpeg.

Thanks again!

1 Like

Feel free to post us some benchmarks on your conversions. Great data and awesome setup!

Yup. So that’s going to be in the GUI? Is the area that shows status going to be in a scrolling region, like the file processing list that is above it?

Thanks!

Yes, the main screen will have a scroll bar if there are more queues than the screen an accomodate. The default drop down options are still 1 to 8 but you can manually override it up to 50; although you’ll require an enormous amount of free disk space in your temp folder (1.5 x number of conversions x filesize) to use that many concurrent conversions.

Good to know! Thanks much!

Thomweah5555 what settings did you change in Bitdefender? Under tools -advanced I see some things that might be what you are talking about.
Can you point us in the right direction?
Thanks

Hi -

Sorry, the main culprit was ByteFence, which I whitelisted the temp and video drives. But in Bitdefender, it will help to turn auto-pilot off.

Tom

1 Like