[FEATURE] allow multiple MCEBuddy installs to share the Conversion Que

(Roach) #1

I have Multiple MCEBuddy installed systems (currently a main pc, and a dvr server, but, maybe more if you implement). I would like to see the option to share the workload between systems, so systems not currently que’d could take some of the que from another MCEBuddy install.
I realize, this could take some work around, but, if one install copies it, and transfers it to another install, it should get around some permission issues, this SHOULD require, that both installs be authorized to work with each other (perhaps a pairing-to use bluetooth lingo), and the completed file, would need to be transferred back to the original system, and placed as original system was setup.
Obviously, it should not be cleared from the que, until it is completed and returned, and items being worked on by another system, should be paused in the que, and not started, until X time after augmenting system disconnect, or manually activated. Perhaps even a minimum que before offloading unlocks…
However, given the customizeability of the MCEBuddy system, I would think that you could have a secondary install, process the file, to the originator’s settings, using the secondary’s system. I would recommend that hardware settings be specific to that system, as not all systems are built to the same spec, but, having my main pc, have the option of offloading some of the back que from the dvr system would help…
There would need to be buttons on the MCEBuddy Status panel, that would “Allow Que Assist”. This would be Only active, when a system’s own que, is empty.
Basically, I am after a way, to click a button, and allow a secondary system, to work the transcode, as the original system’s instructions stated, helping to augment a heavy dvr load, and compensate for weaker/older system limitations, by allowing additional systems to help…
As it is now, the files have to be moved off the originator, and moved to a second system to take over the load, this is, rather inefficient and provides additional corruptible layers…

0 Likes

#7

I wrote a batch script to offload to another server or servers. It’s not as smooth as an integrated solution would be, but I think it’s closer than some of the other proposed scripts. It functions using one computer with MCEBuddy as the queue manager which hands off to other servers as appropriate.

Essentially, it does the following:

  • Check if server 1 has reached the threshold number of concurrent conversions you set to trigger offloading to another server.
  • Check if server 2 is responding and welcoming conversions.
  • Check if server 2 is at or below the threshold you set for how many conversions it should have queued up (and/or if its instance of MCEbuddy is idle, if that’s the condition you set).
  • If so, move the file to the folder you defined for that server, and drop from server 1’s queue.
  • If not, then proceed to server 3, server 4, etc. until it finds an available server.
  • If it finishes the list and no other server will accept it, then server 1 processes the file as usual.

The following assumptions apply:

  • you installed mcebuddy in the default directory on the main computer
  • you use the default port to interact with the server
  • you copy the code below and save it as a .bat file after customizing the initial few variables
  • you add a “PreMetaCustomCommandPath=” entry to the conversion profile to point at this .bat file. (This runs on a per-profile basis.)

The following limitations apply (and so you have to configure these things yourself):

  • tell server 2 what directory to monitor
  • tell server 2 how to process the file or where to put it
  • tell server 2 what to do with the original file when finished
  • change a “paused”, “scheduled” or “stopped” mcebuddy server to “started”
  • enable access for server 2 to access the file, or for server 1 to access the directory monitored by server 2
  • retrieve unprocessed files if a secondary server is stopped after hand-off
  • get around windows firewall if you haven’t already configured that
  • leave a pretty log for MCEbuddy. MCEBuddy on server 1 sees each transferred file as a failed conversion.

Other notes:

  • You can define as many servers as you want in the initial variables (set the server, path, and minimum queue length for each) and the rest of the code will handle it from there.
  • It’s set to create a few text files in your root monitoring path. These are just because I’m not very good at batch scripting. If anyone can improve my code, please do! I can imagine a few things that could be more efficiently coded, but I’m not familiar enough (yet).

This is what I have in my \config\profiles.conf under the profile I’m using. The batch script relies on at least the first five CustomCommandParameters to be as I’ve entered here. The other three lines you can customize:

PreMetaCustomCommandPath=C:\Shared\MediaInfo\Copy.bat
PreMetaCustomCommandParameters="%sourcefile%" "%relativesourcepath%" "%originalfilename%" "%originalext%" "%originalfilepath%"
PreMetaCustomCommandHangPeriod=600
PreMetaCustomCommandCritical=false

The hang period is more than needed, but it needs to be long enough to allow copying or moving of files if moving to a different drive, share, computer, etc.

And here’s the actual script:

@echo off
@setlocal enableextensions enabledelayedexpansion

rem *******************************
rem **** Customizable variables ***
rem *******************************

rem ****** Must fill in order; don't skip a number ******
rem ****** Each defined server must also have a defined monitorpath#= and minQueue#= below
set server1=10.1.10.141
set server2=10.1.10.108
set server3=
set server4=

rem ****** Fileserver can be same as MCEBuddy machine. It's only used by default to define monitorpath below. 
rem ****** If you insert absolute paths in monitorpath, fileserver can be ignored or left undefined. ******

set fileserver=10.1.10.108

rem ****** Set to whatever you'd like. These should be defined as accessed from server 1, since that is where the script is running.
rem ****** Configure the other server(s) to each monitor one of these locations corresponding to the server#.
rem ****** Don't use slashes at the end.
rem ****** I put mine inside the original folder and set the primary server to ignore "distributed\server*" and its subfolders, but you can define them each independently, as long as each server can access its folder.

set "monitorpath1=\\%fileserver%\F\Multimedia\Video"
set "monitorpath2=%monitorpath1%\distributed\server2"
set "monitorpath3=%monitorpath1%\distributed\server3"
set "monitorpath4=%monitorpath1%\distributed\server4"

rem ****** Set to 0 to skip that server; set to 1 to use only when idle; or set any other number.
rem ****** Server 1 will process all until this number of concurrent conversions is met, then start passing to next server until it too reaches this number, and so on. 
rem ****** Overflow stays with server 1 until another server falls below its threshold.

set minQueue1=10
set minQueue2=2
set minQueue3=2
set minQueue4=2

rem *******************************
rem ******* Fixed variables; generally don't need adjusting *******
rem *******************************

set relativesourcepath=%2
set relativesourcepath=%relativesourcepath:"=%
set filename=%3.%4
set filename=%filename:"=%
set absolutesourcepath=%5
set absolutesourcepath=%absolutesourcepath:"=%

set servernumber=1
set fail=0
rem ****** Look up size of queue of each server ******
:queueLookup
"C:\Program Files\MCEBuddy2x\mcebuddy.usercli.exe" --command=query --action=queuelength --server=!server%servernumber%! >"%monitorpath1%\queue!servernumber!.txt"
findstr /C:"queue = " "%monitorpath1%\queue!servernumber!.txt">"%monitorpath1%\queue!servernumber!a.txt"
move /y "%monitorpath1%\queue!servernumber!a.txt" "%monitorpath1%\queue!servernumber!.txt"
set /p queue!servernumber!=<"%monitorpath1%\queue!servernumber!.txt"
set "queue!servernumber!=!queue%servernumber%:~-1!
if "!queue%servernumber%!"=="~-1" (
set queue!servernumber!=
set /a fail=fail+1
if !fail! EQU 3 goto:skip
SET /A rdm=%RANDOM% * 5 / 32768 + 1
timeout !rdm!
goto:queueLookup
)
:skip
set fail=0
set /a servernumber=servernumber+1
if defined server!servernumber! (
goto:queueLookup
) else (
set servernumber=1
goto:queueCompare
)

rem ****** Check if each server is "started" and compare queue length to defined threshold. ******
rem ****** If secondary server is "stopped", "paused", or "scheduled" it will be skipped. ******
rem ****** Transfer file to first server where conditions are met and trigger rescan of that server. ******
:queueCompare
if !servernumber!==1 (
if !queue%servernumber%! LEQ !minQueue%servernumber%! (
if !minQueue%servernumber%! GEQ 1 goto:ending
)
)
set /a servernumber=servernumber+1
if defined server!servernumber! (
goto:nextserver
) else (
goto:ending
)
:nextserver
"C:\Program Files\MCEBuddy2x\mcebuddy.usercli.exe" --command=query --action=enginestate --server=!server%servernumber%!>"%monitorpath1%\status!servernumber!.txt"
set "Server!servernumber!Status="
findstr /C:"started" "%monitorpath1%\status!servernumber!.txt"
if "%ERRORLEVEL%"=="0" set Server!servernumber!Status=1
findstr /C:"progress" "%monitorpath1%\status!servernumber!.txt"
if "%ERRORLEVEL%"=="0" set Server!servernumber!Status=1
if defined Server!servernumber!Status (
if !minQueue%servernumber%! GTR !queue%servernumber%! (
rem *******could also say something like: if !queue%servernumber%! LEQ !queue1!
md "!monitorpath%servernumber%!\%relativesourcepath%" 2>nul
move /y %1 "!monitorpath%servernumber%!\%relativesourcepath%\%filename%"
"C:\Program Files\MCEBuddy2x\mcebuddy.usercli.exe" --command=engine --action=rescan --server=!server%servernumber%!
goto:ending
) else (
goto:queueCompare
)
)

:ending

Updated 3/19/2019 to fix some bugs in the script.

1 Like

[FEATURE] MCEBuddy Status - create option to monitor multiple MCEBuddy's in one MCEBuddy Status window