NextPVR - Three questions concerning VAP and having it Parse from XML files. Thanks!

[jtcreate]

Hi... I started to raise this topic up a few months back but I got too busy. So I am just going to open a fresh thread on this. I have three questions...

1# NextPVR and MediaPortal both provide options to output XML data along with the .ts recording file. What I was asking before was to have VAP be able to grab and parse the recording data from the XML file. The samples I attached below are a very simple format. I was also going to also ask if VAP could have an option of generating an XML file from the data it gathers?

2# When VAP used the autoprocess batch script to process a file through Handbrake, I can see you are assigning a value of %~1 for the new filename and %~3 for the path. Do you have any assignments for the original file name? Would it be hard to incorporate?

3# You might already have this option, but I wasn't sure. I was asking if we can choose how VAP parses data between different sources of metadata. For example, I wanted to obtain the "season and episode numbers" from TVDB, but keep the "eptitle" and the "Original Air date" from the recording source (.wtv file or generated XML).



Thanks.

 

[dlflannery]

Hi... I started to raise this topic up a few months back but I got too busy. So I am just going to open a fresh thread on this. I have three questions...

1# NextPVR and MediaPortal both provide options to output XML data along with the .ts recording file. What I was asking before was to have VAP be able to grab and parse the recording data from the XML file. The samples I attached below are a very simple format.

I'm considering doing this. You said samples (plural) -- I only found one sample attached. Also is this just for TV Shows or also for Movies? I would need a movie sample too.

I was also going to also ask if VAP could have an option of generating an XML file from the data it gathers?

Would this be for all videos, or just for those that had an input XML file? What XML elements are required? For example, your sample file includes channel_OID, uid, and some other elements that may only be available if there is an input xml file providing them. Please eliminate any elements that aren't essential.

2# When VAP used the autoprocess batch script to process a file through Handbrake, I can see you are assigning a value of %~1 for the new filename and %~3 for the path. Do you have any assignments for the original file name? Would it be hard to incorporate?

Look in VAP-ReadMe.pdf, section "Things To Know", item 17, where all nine arguments passed to the script are described. The second argument is what you want.

3# You might already have this option, but I wasn't sure. I was asking if we can choose how VAP parses data between different sources of metadata. For example, I wanted to obtain the "season and episode numbers" from TVDB, but keep the "eptitle" and the "Original Air date" from the recording source (.wtv file or generated XML). .......

This requires more thought. I don't want to have to make an entire configuration panel where you select which source you want for each meta item. I'm thinking along the lines of a default action where the two choices you mention automatically happen if an input xml file is available. Will this work for you?

 

[jtcreate]

A lot of good questions. Let me put some thoughts together on those. I'll also gather and send you a variety of XML samples too.

As far as having VAP parse metadata between different sources like TVDB, I am fine your approach as far as XML files, but what about other files like .wtv, .mp4 or .mkv? I agree with you about not having to assemble another panel over this. I was thinking more on the lines of adapting one of the current panels. Maybe the "MetaData Templates" panel, just under the "Tag Grabber". You can create a section called "TVDB searches". Under it, you simply list all the tags and put a "true/false" check box next to each one.OR, perhaps, a drop down menu that does the same thing. How does that sound?

 

[dlflannery]

A lot of good questions. Let me put some thoughts together on those. I'll also gather and send you a variety of XML samples too.

As far as having VAP parse metadata between different sources like TVDB, I am fine your approach as far as XML files, but what about other files like .wtv, .mp4 or .mkv? I agree with you about not having to assemble another panel over this. I was thinking more on the lines of adapting one of the current panels. Maybe the "MetaData Templates" panel, just under the "Tag Grabber". You can create a section called "TVDB searches". Under it, you simply list all the tags and put a "true/false" check box next to each one.OR, perhaps, a drop down menu that does the same thing. How does that sound?

OK I'll be looking for your answers, in particular in response to this question trying to clarify when the XML output file is desired and what is needed in it:

Would this be for all videos, or just for those that had an input XML file? What XML elements are required? For example, your sample file includes channel_OID, uid, and some other elements that may only be available if there is an input xml file providing them. Please eliminate any elements that aren't essential.

At first, at least until more users show interest in this topic, I would like to just implement a default behavior which would be the current behavior except override episode title and original air date with values from the input WTV or XML file (when available). I need to know which (WTV or XML) values are preferred in case both are available.

 

[jtcreate]

Here are some samples to get yourself familiar. I'll give you a more detailed response to your questions shortly... Thanks


NextPVR Samples

https://1drv.ms/u/s!AnHKMcbNlt04pAbJwV7kfO1i8hRU


MediaPortal Samples

https://1drv.ms/u/s!AnHKMcbNlt04pAcTfoh3wl18KBof

 

[jtcreate]

Sorry, I thought I would have gotten back to you sooner. I've been busy with work and other personal issues. I hope all is well with you. Since its been a month, I might miss some of your earlier questions. I'll try to revisit them. The file samples in the previous post should still be active.

Would this be for all videos, or just for those that had an input XML file?

It would be for all output videos. Mp4 is what I use but its limited in metadata. That's why I am favoring the XML file companion option.

What XML elements are required? For example, your sample file includes channel_OID, uid, and some other elements that may only be available if there is an input xml file providing them. Please eliminate any elements that aren't essential.

Perhaps its easier to just tell you that I am only concerned with what VAP presently works with; "Title", "OAD", "Eptitle","SeasonNum","EPNum", "Network", and "Wildcard" or "Recorded Date and Time" (in that order of importance). Everything else is irrelevant. However, if an XML input file is provided, I don't see why any additional data needs to be lost. VAP could just make a copy of the input file and impliment any neede changes to it. Pass it along as an output XML file. Just a thought.

Btw, Has VAP added a "Channel" or "Network" tag yet for Network affiliates?

At first, at least until more users show interest in this topic, I would like to just implement a default behavior which would be the current behavior except override episode title and original air date with values from the input WTV or XML file (when available). I need to know which (WTV or XML) values are preferred in case both are available.

Perhaps a more simplified approach? How about a simple True/false setting that establishes "TVDB" as a backup metadata source instead of a primary metadata source? If the metadata in any of the Tags cannot be located in an XML, Wtv or mp4 input file, VAP uses TVDB to parse it? My wtv files never provide info for tags SeasonNum and EPnum. Most of the time the XML file does, but not always. Both always provide the "Eptitle". Would that be easier?

If you do follow that approach, there is one little catch. sometimes Schedules Direct makes an error. Instead of provide a "s02e04", it will just place a #14 in the "Epnum" and leave the "SeasonNum" empty. Perhaps if Vap sees any "#" characters in the "SeasonNUm" or "EpNum" tags, it deafults to TVDB.

 

[dlflannery]

Several issues after reviewing the sample xml files you sent:

NextPVR and MediaPortal have completely different XML formats. VAP can adapt to sense which format and read either but: (1) which one do you want for the output format? and (2) If your answer is "same format as input xml", which format when there isn't any input xml?

VAP currently determines whether a file is movie or TV episode and doesn't attempt to do theTVDB search if it's a movie. I think I see data patterns in both xml formats that will imply movie or TV show. When VAP thinks it's a movie, the input xml data will just be passed along to the output xml file.

Just FYI, the MediaPortal TV xml's have some weird non-numeric episode numbers, e.g., "6ARG08".

Are you processing .mp4 input files? I believe all the samples you furnished were .ts files, which have no embedded metadata, although they do contain H.264 video, which is commonly contained in .mp4 files. If you are actually processing .mp4 input files, could you please furnish an example or two so I can test getting metadata from them?

 

[jtcreate]

Several issues after reviewing the sample xml files you sent:

NextPVR and MediaPortal have completely different XML formats. VAP can adapt to sense which format and read either but: (1) which one do you want for the output format? and (2) If your answer is "same format as input xml", which format when there isn't any input xml?

VAP currently determines whether a file is movie or TV episode and doesn't attempt to do theTVDB search if it's a movie. I think I see data patterns in both xml formats that will imply movie or TV show. When VAP thinks it's a movie, the input xml data will just be passed along to the output xml file.

Just FYI, the MediaPortal TV xml's have some weird non-numeric episode numbers, e.g., "6ARG08".

Are you processing .mp4 input files? I believe all the samples you furnished were .ts files, which have no embedded metadata, although they do contain H.264 video, which is commonly contained in .mp4 files. If you are actually processing .mp4 input files, could you please furnish an example or two so I can test getting metadata from them?

I only process MP4 files once in a while. Its not that important. My main concern is the .ts files and the .wtv files. I output in MP4.

As far as weird episode numeric with MediaPortal, I believe MediaPortal is not responsible for it. I just remembered I was having trouble getting Schedules Direct to work on MediaPortal. So I downloaded the XMLTV Guide info from Zap2it.com. Sorry! I completely forgot to mention that too. There is suppose to be a program or approach that you add as a plugin to mediaportal that deciphers that episode information into "SeasonNUM" and "Epnum". I haven't dug it up yet. The problem with MediaPortal is that it doesn't list an OAD element anywhere. It tends to post the original air date in Parathensis at the end of the show's description (and, I have no idea why).

Honestly, I only included the MediaPortal XML at the time as I was unsure if I was going to use it. Since then, I've been doing more with NextPVR. It would be good to be able to work with both especially since MediaCenter is a near death issue. Perhaps, its best to focus on NextPVR for now. Down the road, I can always revisit MediaPortal and solve that episode numbering issue.

VAP currently determines whether a file is movie or TV episode and doesn't attempt to do theTVDB search if it's a movie. I think I see data patterns in both xml formats that will imply movie or TV show. When VAP thinks it's a movie, the input xml data will just be passed along to the output xml file.

NextPVR is the better XML solution and I believe its closer to the Schedule's Direct format which is a good format. It also stores OAD information too. I wish there was a way that these DVR software programs would leave an ID of themselves in the XML file. Something that tells VAP where the XML file came from.

 

[dlflannery]

For cases where VAP has to create an output xml file with no input xml file available, I need to know what elements must be present. The following elements that I see in your NextPVR TV samples cannot be obtained from theTVDB.com

status
startTime & endTime
channel_OID
uid
firstrun
audio

Thus to create a useable output xml file we either need to deternine that an element may be absent or provide a default fixed value for it. Could you experiment with one of your current files by deleting, or finding an acceptable fixed default value for, each of those elements and verifying the xml file still serves your puupose? BTW, what program is reading these files?

What are channel_OID and uid? I searched a little via google and the nextPVR wiki and didn't find anything on those.

 

[dlflannery]

Assuming we resolve the issues in my previous post about what elements can be left out or defaulted to fixed values, here is a proposed behavior:

Initially at least, this new feature will be activated by a Command-line option, -nextPVRmeta, not case sensitive. If several users show interest in the feature it can be controlled by a checkbox in Advanced Configuration but that tab is already pretty busy.

For movies, if an input xml exists it will be meta-renamed and copied into the output folder.

For TV Shows:

If an input xml file exists but no other metadata is available, it will be meta-renamed and copied into the output folder.

If an input xml file exists and other metadata (e.g. from theTVDB.com) is found, the xml file will be read and a meta-renamed output xml file containing merged data will be created in the output folder. The merge strategy will be that item values that are available in the input xml will be substituted for the corresponding item values from other sources.

If no input xml file exists but other metadata is available, a meta-renamed output xml file will be created.

In all cases

The filename xml element in any output xml file will be modified to match output video file path, meta-renamed if applicable.

 

[jtcreate]

For cases where VAP has to create an output xml file with no input xml file available, I need to know what elements must be present. The following elements that I see in your NextPVR TV samples cannot be obtained from theTVDB.com

status
startTime & endTime
channel_OID
uid
firstrun
audio


Thus to create a useable output xml file we either need to deternine that an element may be absent or provide a default fixed value for it.

Good question. When the XML file is a movie file, the season and episode elements are missing. With that exception, much of the XML template is the same for both. With the exception of the end time, I have been trying to think of keeping things easy. How about this?


Status | Not important at all - Fixed element "ready"
StartTime | When it comes to wtv files, maybe it can be parsed from WTV file name. Its available in the Ts file name as well.
endTime | Ts file has it in the filename. Wtv has the "total video length time" in the metadata which can be added to the start time with an easy math equation. If that video length time can't be read, we an simply default "00000". Does that help?

Channel_OID | not important default fixed value "0000"
UID | Not important. We can set a fixed value "EP0000000False" for TV and "MV0000000False" for movies. Basically saying not available.
FirstRun | Definitely not important.
Audio | All this does is just identifies if the incoming audio is in Stereo or 5 channel. That info is not available in wtv? If not, just fixed value: Stereo

Could you experiment with one of your current files by deleting, or finding an acceptable fixed default value for, each of those elements and verifying the xml file still serves your puupose? BTW, what program is reading these files? What are channel_OID and uid? I searched a little via google and the nextPVR wiki and didn't find anything on those.

UID stands for "Unique Identifier". Each asset in the EPG gets its own ID num no matter the type of programming (I believe TV shows start with an "EP" while movies start with a "MV"). If you know how a database works, then you already got the picture what the UID's roll iS. Btw, Its the UID that's popping up in the MediaPortal Season/number instead of the right S/E information.

OID I am not so sure about. Its probably a technical ID of the actual channel or transponder on Directv.

Neither are really important for NextPVR or anything else I am doing. But its good to preserve it if its already available. In case I am wrong.



The workflow you laid out looks fine. I didn't see any mention of adding a "Network" tag to VAP. Would it be possible to add a "network" and "Wildcard2" tag?

 

[dlflannery]

.... I didn't see any mention of adding a "Network" tag to VAP. Would it be possible to add a "network" and "Wildcard2" tag?

VAP metadata structure has channelNumber, callSign, network and wildcard2 tags. The network tag is the network affiliation (e.g., CBS). Do you want that to be in the channel element of the output xml flag? What I see in the samples looks like station call signs being put in the channel element, e.g., KCAL, and I don't see any network, call sign, or wildcard2 elements in your sample xml files. I would be surprised if NextPVR xml uses wildcard2 elements. The VAP wildcard tags are used only for parsing metadata from input video filenames with string templates defined by the user. How are you envisioning using wildcard2?

Can you point me to any documentation defining the elements used in these xml files? As I said, I couldn't find anything.

Still wondering what program(s) is(are) taking these xml files as input?

 

[jtcreate]

Oh, I didn't explain this better...Completely my bad. I just ran you through a gautlett on those tags for no reason. Sorry about that. I was considering Network, Channel and callsign to be the same thing. In the XML file, Callsign is simply the "channel" element. In the .WTV metadata, its called "Station callsign". Its all the same. Which ever one you want to use is fine. As long as it containts the four letter callsign.

As far as Wildcard2, I only requested that for parsing filenames through VAP. I don't expect any elements in XML to fill it. I'm playing with a separate idea. If it works, Ill share it with you.

As far as the output XML, I don't have a specific software in mind yet for the XML file. I'm mainly using it for library purposes for now. Why I want to have the XML file is that its highly customizeable and I can store my own data on it. Notes. Ideas. Stuff like that. Especially when I write on my spare time

So turning on the XML input feature is "-nextPVRmeta". Cool! Can't way to see what you have done.


Thanks for the effort. Its nice to see VAP expand so much after all these years as well.

 

[dlflannery]

......
As far as Wildcard2, I only requested that for parsing filenames through VAP. I don't expect any elements in XML to fill it. I'm playing with a separate idea. If it works, Ill share it with you.

At present I can't define any modification you are requesting re: wildcard behavior in VAP. If that is incorrect, you will need to clarify it.

..... As far as the output XML, I don't have a specific software in mind yet for the XML file. I'm mainly using it for library purposes for now. Why I want to have the XML file is that its highly customizeable and I can store my own data on it. Notes. Ideas. Stuff like that. Especially when I write on my spare time

Hmm... I'm not enthusiastic about creating output xml files to an undefined specification. Not sure what you mean by "highly customizable". Changing the format in code is more than a trivial exercise.

So turning on the XML input feature is "-nextPVRmeta". Cool! Can't way to see what you have done.
.......

So far all I've done is develop a routine to read in the NextPVR xml files per your examples. That was relatively easy. The code to create output xml files, handling the three possible cases defined in a prior post, will require a lot more work. Have you looked at the pyTivo .txt metadata files that VAP will output if the CL option -metapytivo is applied? Since you don't have a specific xml format in mind, perhaps that file would suffice, although it isn't xml format. Since it is just text, it also would be "highly customizable" by you just using a text editor. The definitions of the tags in this type file are found here:
https://pytivo.sourceforge.io/wiki/index.php/Metadata

 

[jtcreate]

Sorry I got swamped again but I did take your advice and looked over you suggestions. I think metapytivo and the metaBTV are just fine. I suppose the -metaNEXTPVR command would mainly be a trigger for XML input?

 

[dlflannery]

Reading in a NextPVR xml file could be default (i.e., automatic) behavior activated by the simple presence of the appropriately named file, not requiring a command-line option. Those metadata items, when available, would take precedence over those obtained by other means.

 

[jtcreate]

Reading in a NextPVR xml file could be default (i.e., automatic) behavior activated by the simple presence of the appropriately named file, not requiring a command-line option. Those metadata items, when available, would take precedence over those obtained by other means.

If you feel that is the better way to go, I am fine with it.

 

[dlflannery]

OK, substitute the VAP executable (not an installer package) in the zip downloaded from here:

VAPexe133T1.zip

If a NextPVR xml file was found and successfully read, the VAP log file will give a message to that effect.

 

[jtcreate]

OK, substitute the VAP executable (not an installer package) in the zip downloaded from here:

VAPexe133T1.zip

If a NextPVR xml file was found and successfully read, the VAP log file will give a message to that effect.

Great...I will give it a test and see how it goes.

 

[jtcreate]

Ok....Potential is definitely there. A few bugs and a file maintenance issue I overlooked. Sorry for the long list.


1# {title} tag is not populating. Instead, the "EpTitle" information is appearing in the title tag instead. This is occuring even when I parse from the .ts file name in advanced settings (i.e. .ts {title}_{wildcard} ). The Season and Episodes numbers are not double digits each. s02e08 instead of s2e8. Could cause issues when organizing a library by alpha. Perhaps VAP can make a simple correction when it detects a single digit.

This is my current naming convention:

{title} ~s{seasonnum}e{epnum}~ OAD {year} {month}-{day} ''{eptitle}'' [RDT- {callsign} {wildcard}]

This is what VAP is outputting.

A Problem Like Maria ~s1e17~ OAD 2017 03-08 ''A Problem Like Maria'' [RDT- 20170613_20002100]


2# Original XML file fate should be the same as the Original .ts file. If VAP deletes or moves the original .TS file from the 'monitor' folder to a another destination folder or recycle bin, the same should happen with the original XML file too.

3# Some XML files like "Access Hollywood" are missing "EPtitle" info and TVDB has no listing. I'm still thinking this one over. Perhaps copy and paste title into eptitle as a last resort.

4# VAP Missing a "Callsign" tag.


5# I did notice that VAP was trying to do TVDB searches on the .ts files before the XML file arrived into the 'Source' directory. Perhaps the NEXTPVR switch is necessary so that VAP knows to wait for the XML file to arrive once the TS file finishes transfering. Although once in a blue moon, a TS file may not have an XML companion. As a failsafe, VAP can procede TS file processing without the XML file after a certain amount of time has passed.



I made some more samples and put them in the link below and I am including my VAP log from the test to help. Let me know if there is something I can do. Thanks.

https://1drv.ms/f/s!AnHKMcbNlt04pAVfUJow_hYSLAXf