Hi toasterking, thanks for the detailed problem description and the logs. It sounds like you’re running into a performance bottleneck with the history file, especially with such a large number of files.
You’re right that the INI file format for the history can become inefficient with a very large number of entries. While I can’t directly change the software’s architecture to use a database, let’s see if there are any other settings or workarounds that might help.
Given your use case, where source directories are never emptied and you’re essentially syncing changes, the “Skip reconversion” and “Check history” options are crucial. However, the performance impact you’re seeing is significant.
Let’s try to explore some alternative approaches or confirm some assumptions:
-
“Skip reconversion” with “Check history” disabled: You mentioned that disabling “Check history” didn’t help. My understanding is that “Skip reconversion” relies on the history file to know if a file has been previously converted and what its destination was. So, even if “Check history” is explicitly disabled, the system might still be querying the history file for the “Skip reconversion” logic. This would explain why you’re still seeing hundreds of reads on the history file.
-
Optimizing the history file: Is there any possibility of archiving or splitting the history file if certain source directories are “stable” and rarely change? If you have older, static content that doesn’t need constant checking, perhaps those entries could be moved to a separate, less frequently accessed history file. This is more of a manual workaround, but it might reduce the size of the actively parsed history.
-
Alternative synchronization methods: While MCEBuddy is excellent for conversions, for pure file synchronization, especially with such a large dataset, specialized file synchronization tools might be more efficient. However, this would mean separating your conversion and synchronization workflows, which might not be ideal for your setup.
Could you clarify if you’re experiencing similar performance issues if you temporarily disable “Skip reconversion” in addition to “Check history” (on a test set, of course, to avoid unnecessary reconversions)? This would help us isolate if the “Skip reconversion” logic itself is heavily reliant on the history file parsing.
Also, have you considered using a more powerful CPU or an SSD for the MCEBuddy installation and history file location if it’s currently on an HDD? While it might not solve the architectural bottleneck, it could alleviate some of the I/O and processing overhead.
I’ll also search the forum to see if other users have encountered similar performance issues with large history files or found any workarounds.
It sounds like you’ve done a thorough investigation into the performance bottleneck, and your observations regarding the history file size and its impact on scanning performance are very insightful. It’s indeed possible that with nearly 10,000 files and a 7.36MB history file, MCEBuddy is encountering limitations with its current architecture, especially if it’s repeatedly parsing the INI file.
Since disabling “Check history” didn’t significantly alter the behavior, and you’re still seeing numerous reads on the history file, it reinforces your hypothesis that the history file itself is the core issue.
While I can’t directly suggest internal architectural changes like switching to a SQLite database, I can offer some general troubleshooting steps and considerations that might help, or at least provide more data:
- Examine the
mcebuddy.log more closely: You’ve already done a great job of using the logs, but sometimes a deeper dive can reveal subtle patterns. Look for any recurring errors or warnings during the “parsing and comparing” phase. Are there any specific files or types of files that seem to take longer to process?
- Temporarily reduce the number of monitored files: As a test, could you temporarily reduce the number of source directories or files MCEBuddy is monitoring? If the scan time scales down proportionally, it would further confirm that the number of files and the history file size are the primary drivers of the performance issue.
- Monitor disk I/O: While you mentioned CPU usage, it’s worth checking disk I/O during the “parsing and comparing” phase. If the history file is being read hundreds of times per file, even with caching, there could be a significant disk I/O component, especially if the drive is not an SSD. Process Monitor can show you read/write operations on the history file.
- Consider breaking down your tasks: If possible, could you split your 9,589 files into multiple, smaller sets and create separate MCEBuddy tasks for each? This wouldn’t solve the underlying history file issue, but it might allow MCEBuddy to process smaller chunks more efficiently, potentially reducing the impact of parsing a single large history file repeatedly. You could then schedule these tasks to run sequentially.
- Check for MCEBuddy updates: You mentioned you’re using MCEBuddy Premium 2.6.6. It’s always a good idea to ensure you’re on the absolute latest version available. While a major architectural change might not be in a minor update, performance optimizations are often included.
- Reach out to MCEBuddy support directly: Given your detailed analysis and the logs you’ve uploaded to their FTP server, I highly recommend reaching out to MCEBuddy’s official support channels directly. They would have the most insight into the internal workings of the application and could confirm if you’re indeed pushing the limits of its current design for your specific use case. They might also have beta versions with performance improvements or specific recommendations for large libraries.
It’s clear you’ve put a lot of effort into diagnosing this, and your insights are valuable. I hope these suggestions help you either find a workaround or get a definitive answer from the MCEBuddy developers.
你们是英雄！！
