Searching a directory with a large number of files is slow

Request Type:
I use buddy daily and would like to be able to limit the amount of work it does.

I have 40k tv episodes so buddy searches them all ( I don’t know how often) for AVI to be converted to MP4 and MKV to be remuxed to MP4.

I would like for buddy to search episodes added in say the last 3 days instead of searching the whole 40k episodes everytime.
I could use a schedule but that would not work with the random times new media is added…

You can adjust the scan/poll period in the Settings -> System settings page. The default is to check for new files every 5 minutes.

Explain a little more what issue you’re facing?

We’ve tested it with over 100,000 files and it’s pretty quick with little overhead.

Hey Rboy

Thanks for the quick reply.

When I drag mkv or avi into buddy (to be remuxed/converted) it take minutes (or longer) to see the files.

And same with rescan - several minutes b4 they start getting remuxed or (avi) converted.

I am thinking buddy is search my 40,000 episodes every time!

Okay we found why it’s taking so long to search and fixed it, it should also speed up manually dragged files. In our testing here for 50K files, it took about 15 seconds for it to start adding new files. There is no way to avoid searching in the current architecture but it should significantly faster now and manual additions should be instant now.

Try today’s 2.4.9 build

1 Like

In the next major version of MCEBuddy we’re working on a new more efficient architecture to monitor new files which will avoid reading entire filesystems.

Hey Rboy,

yep it does appears to faster - thanks

You could try setting up a different destination directory (custom naming helps too) so that only the files being converted are in the directory being searched for new files and not cluttered with files that are already converted.

If I understand your suggestion I think it defeats my purpose/setup .

Buddy searches automatically two folders TV and other tv.

They are my folders set for Plex to find shows and stream.

My setup finds the MKV and remuxes to mp4 (leaving the mp4 there) and moves the old mkv to another folder ( just as a backup)
AVI get re-coded as mp4
If all works as planned I don’t even see mkv or avi in my plex folders!

Ah. I have an incoming folder that I transcode everything to mkv with MCE Buddy, and them move them into my plex libraries for movies, shows, specials, and sports. Plex plays mkv for me just fine. This is on a PC with a splitter and mkvtools installed (Gabest, Haali, lava, or your pick). I also have MCE Buddy include the meta data into the mkv so the plex local agent can find it as a last resort.