[BUG] MINOR: Media with different show name character set encodings (apostrophe = â€tm (aEtm))

General Support / Questions Metadata and Renaming
1

I record a show that has an apostrophe in the show name on both my HDHR and on my Tivo.
I use KMTTG to transfer my files in TS from the Tivo and transcode them to raw MP4 (no compression, only because I can’t figure out how to skip that and let MCEBuddy process the raw TS file). I use KMTTG because I only want to process some shows from the Tivo.

What I end up with is two directories with the files from the same show:
Christopher Kimballâ€tms Milk Street Television
and also
Christopher Kimball’s Milk Street Television
I think the Tivo files end up in the folder with the messed up show name.
On the filesystem, it is even weirder. I get THREE directories:

  1. Christopher Kimball’s Milk Street Television (proper apostrophe - likely HDHR source)
  2. Christopher Kimball’s Milk Street Television (mistranslated UTF-8 - Tivo source, from internal metadata)
  3. Christopher Kimballs Milk Street Television (apostrophe removed entirely - Also from Tivo, but the KMTTG transcoded MP4 version, but KMTTG apparently does not retain the internal metadata).

There could even be different guide data for the show name depending on which channel it was recorded on, or if the special naming correction rules I have setup are forcing TVDB info to be used for some recordings and that is the source of the missing apostrophe.

I found an article for PHP that might have a solution:

and to use “iconv(“UTF-8”, “CP1252”, $data)”.
The problem seems to be my filesystem (Windows) uses Windows Code Page 1252 (close enough to ISO-1252 character set to be functionally equivalent).

I suspect that the Tivo is using filenames (or shownames in the metadata) encoded with UTF-8 and that causes the weirdness with the “smart quote” apostrophe being translated differently between the two media file sources.

2

More info - the weird ’ is appearing in the show name used in the filename, the metadata description, and the internal metadata for “Showname”.
I will track down the log file and the media file for uploading if you want to see the flow through MCEBuddy.

I’ve also narrowed it down to these files are coming from the Tivo, after extraction by KMTTG.

3

I’ve narrowed down the source to the files extracted by KMTTG, however, there appears to be additional issues when MCEBuddy processes them.
KMTTG extracts the raw .TS file, and then transcodes it (raw) to an .MP4 file. So there are two files in the output “Tivo” directory that is monitored by MCEBuddy in my Tivo profile.
Here is an extract of the log for the .MP4 file. Notice that KMTTG has stripped the apostrophe entirely from the show name, so that explains that resulting processing from MCEBuddy into an MKV.

However, there appears to be a problem in the Exception processing (from the MP4 file log):

INFORMATION> --> Task -> MKV-TV-HD
Profile -> HEVC MKV
Source File -> F:\Recorded TV\Tivo\Christopher Kimballs Milk Street Television-S03E06-Chicken Around the World-2019-10-11.mpg

No Apostrophe in “Kimball’s” - removed by KMTTG before MCEBuddy even sees it.
Here’s the section where I think the filename/metadata is not working as I think it should.

Download Series Details -> True
Download Banner -> True
Overwrite metata from Title -> False
Overwrite metata from Internet -> Default
Metadata Correction => Option 0
[...]
Original Title -> Christopher Kimball*
Corrected Title -> 
TVDB Series Id -> 334523
IMDB Series Id ->

Notice the wildcard is placed to ignore whether or not the apostrophe is there (or am I not doing the regex correctly?). I have this same issue with “America’s Test Kitchen” as well for the same reasons.

Here’s the log section of the processing that I think shows that the TVDB info is not used (meaning the override didn’t work as expected).

INFORMATION> 2019-10-19T16:33:37 MCEBuddy.Engine.ConversionJob --> Current System language is English (United States) (eng)
INFORMATION> 2019-10-19T16:33:37 MCEBuddy.Engine.ConversionJob --> Converting file -> F:\Recorded TV\Tivo\Christopher Kimballs Milk Street Television-S03E06-Chicken Around the World-2019-10-11.mpg
INFORMATION> 2019-10-19T16:33:40 MCEBuddy.Engine.ConversionJob --> Running custom commands
INFORMATION> 2019-10-19T16:33:40 MCEBuddy.Transcode.CustomCommand --> No custom commands found
INFORMATION> 2019-10-19T16:33:40 MCEBuddy.Engine.ConversionJob --> Getting show information and banner from Internet sources
INFORMATION> 2019-10-19T16:33:40 MCEBuddy.MetaData.VideoMetaData --> Extracting Generic Tags
INFORMATION> 2019-10-19T16:33:40 MCEBuddy.MetaData.VideoMetaData --> Downloading Series details
INFORMATION> 2019-10-19T16:33:40 MCEBuddy.MetaData.VideoMetaData --> Recording Type Show
INFORMATION> 2019-10-19T16:33:40 MCEBuddy.MetaData.VideoMetaData --> Checking TheTVDB
WARNING> 2019-10-19T16:33:54 MCEBuddy.MetaData.VideoMetaData --> TheTVDB failed
INFORMATION> 2019-10-19T16:33:54 MCEBuddy.Engine.ConversionJob --> Checking for destination file skip reprocessing
WARNING> --> Cannot find AirYear
WARNING> --> Cannot find AirMonth
WARNING> --> Cannot find AirDay
INFORMATION> 2019-10-19T16:33:54 MCEBuddy.Engine.ConversionJob --> Destination file F:\Video\TV Shows\Christopher Kimballs Milk Street Television\Season 03\Christopher Kimballs Milk Street Television-S03E06-Chicken Around the World 2019 10 11---.mkv does not exist, continuing with conversion
INFORMATION> 2019-10-19T16:33:54 MCEBuddy.Engine.ConversionJob --> Running custom commands
WARNING> --> Cannot find AirYear
WARNING> --> Cannot find AirMonth
WARNING> --> Cannot find AirDay
WARNING> --> Cannot find AirYear
WARNING> --> Cannot find AirMonth
WARNING> --> Cannot find AirDay
INFORMATION> 2019-10-19T16:33:54 MCEBuddy.Transcode.CustomCommand --> No custom commands found

Then, what is even more weird, is that for some reason, MCEBuddy looks at the .TS file during the conversion job. (See below).

INFORMATION> 2019-10-19T16:34:46 MCEBuddy.Engine.ConversionJob --> Analyzing video information
INFORMATION> 2019-10-19T16:34:46 MCEBuddy.VideoProperties.VideoInfo --> Reading MediaInfo from F:\Temp\working0\Christopher Kimballs Milk Street Television-S03E06-Chicken Around the World-2019-10-11.ts
INFORMATION> 2019-10-19T16:34:53 MCEBuddy.VideoProperties.VideoInfo --> Supplementing Media information using FFMPEG
INFORMATION> 2019-10-19T16:34:55 MCEBuddy.VideoProperties.VideoInfo --> Reading Original File Media information

Any ideas? I will look into either KMTTG not transcoding the .TS file into an MP4, or to delete it when it is done.

It seems either way, I’ve got a broken process - the MP4 file has no apostrophe at all, and the .TS file gets the weird UTF8 apostrophe that doesn’t get converted to CP1252 (mostly the usual ASCII we’re used to).

4

Here’s the MCEBuddy processing log section for the .TS version of the file. This is the one that appears to be getting renamed and uses the incorrectly converted UTF8 apostrophe character (I can’t tell if it is an apostrophe or a “smart quote”, meaning a close-single-quote mark instead of an actual apostrophe).

INFORMATION> 2019-10-19T15:35:40 MCEBuddy.Engine.ConversionJob --> Converting file -> F:\Recorded TV\Tivo\Christopher Kimballs Milk Street Television-S03E06-Chicken Around the World-2019-10-11.TiVo
INFORMATION> 2019-10-19T15:35:43 MCEBuddy.Engine.ConversionJob --> Running custom commands
INFORMATION> 2019-10-19T15:35:43 MCEBuddy.Transcode.CustomCommand --> No custom commands found
INFORMATION> 2019-10-19T15:35:43 MCEBuddy.Engine.ConversionJob --> Getting show information and banner from Internet sources
INFORMATION> 2019-10-19T15:35:43 MCEBuddy.MetaData.VideoMetaData --> Extracting Generic Tags
WARNING> 2019-10-19T15:35:43 MCEBuddy.MetaData.VideoMetaData --> Unable to extract meta data from file F:\Recorded TV\Tivo\Christopher Kimballs Milk Street Television-S03E06-Chicken Around the World-2019-10-11.TiVo. F:\Recorded TV\Tivo\Christopher Kimballs Milk Street Television-S03E06-Chicken Around the World-2019-10-11.TiVo (taglib/tivo)
INFORMATION> 2019-10-19T15:35:43 MCEBuddy.MetaData.FileExtractMetadata --> Extracting TiVO metadata using TDCat
INFORMATION> 2019-10-19T15:35:45 MCEBuddy.MetaData.FileExtractMetadata --> Extracting XML Tags
INFORMATION> 2019-10-19T15:35:45 MCEBuddy.MetaData.VideoMetaData --> Downloading Series details
INFORMATION> 2019-10-19T15:35:45 MCEBuddy.MetaData.VideoMetaData --> Recording Type Show
INFORMATION> 2019-10-19T15:35:45 MCEBuddy.MetaData.VideoMetaData --> Checking TheTVDB
WARNING> --> No match found on TVDB for language en
INFORMATION> 2019-10-19T15:35:46 MCEBuddy.Engine.ConversionJob --> Checking for destination file skip reprocessing
INFORMATION> 2019-10-19T15:35:46 MCEBuddy.Engine.ConversionJob --> Destination file F:\Video\TV Shows\Christopher Kimball’s Milk Street Television\Season 03\Christopher Kimball’s Milk Street Television-S03E06-Chicken Around the World-2019-10-12.mkv does not exist, continuing with conversion

Note that the original .TiVo file (.TS) doesn’t have the apostrophe, but that it appears that this gets inserted into the show name when the metadata is extracted from the TiVo (.TS) file.

Either way, there are problems here too with it 1) not applying the naming/TVDB matching override, nor with 2) using nVidia GPU HW transcoding for the H.265 HEVC codec.

5

Need the full log to see what’s going on. Most likely it’s coming from the TiVO metadata extraction. Having the original MPG file along with the MAK key would be even better to check it out.