NextPVR - Three questions concerning VAP and having it Parse from XML files. Thanks!

[dlflannery]

Take a look at the thread titled "Attention: BeyondTV users". Started it a month ago. 94 views and no one seems to care about Beyond TV. It wouldn't be that hard to resurrect even if I do kill it.

Just to be clear here is what could be possible now that we aren't constrained to a BTV format:
1. Specify any xml output format you want, hopefully not TOO complex. It could be just a small augmentation or modification of the current BTV format, or it could be a completely different format. Thought occurs that an xml version of the current pyTivo .txt metadata file might be interesting.
2. VAP currently only outputs a BTV xml file if it can determine the input video is either a movie or a TV series show. It could be modified to produce the new xml file format even if the only metadata it has is the input NextPVR xml file and it can't determine whether it's a TV series or a movie. (Of course meta-based folder/file renaming will fail if the meta items defined in the template aren't available.)

 

[jtcreate]

Take a look at the thread titled "Attention: BeyondTV users". Started it a month ago. 94 views and no one seems to care about Beyond TV. It wouldn't be that hard to resurrect even if I do kill it.

Just to be clear here is what could be possible now that we aren't constrained to a BTV format:
1. Specify any xml output format you want, hopefully not TOO complex. It could be just a small augmentation or modification of the current BTV format, or it could be a completely different format. Thought occurs that an xml version of the current pyTivo .txt metadata file might be interesting.
2. VAP currently only outputs a BTV xml file if it can determine the input video is either a movie or a TV series show. It could be modified to produce the new xml file format even if the only metadata it has is the input NextPVR xml file and it can't determine whether it's a TV series or a movie. (Of course meta-based folder/file renaming will fail if the meta items defined in the template aren't available.)

Hey Dlflannery. Sorry to have been absent and leave you hanging on this. Had some real family medical problems get in the way and the Job has been hitting me a bit. Anyway, if no one is working with the BTV XML and you feel comfortable altering it, I am ok with that. I need to refresh myself with this thread real quick and do some more tests. I'll get back to you later today or tomorrow.

 

[jtcreate]

I've been Running some tests. Off and on, Windows would flag me that VAP would stop working. I purged out all of the VAP profiles and settings in the user profile, started from scratch, and so far, the issue hasn't returned. However, some of the incoming .TS files I recorded got stuck in "Transferring" status. I just mentioned this on another thread where another user has this issue. I'll try to send you some samples shortly.

 

[dlflannery]

I've been Running some tests. Off and on, Windows would flag me that VAP would stop working. I purged out all of the VAP profiles and settings in the user profile, started from scratch, and so far, the issue hasn't returned. However, some of the incoming .TS files I recorded got stuck in "Transferring" status. I just mentioned this on another thread where another user has this issue. I'll try to send you some samples shortly.

Since this is off topic for this thread, let's pursue it on the other thread rather than here.

 

[jtcreate]

The issue hasn't returned since purging the VAP configurations, however, Atomic Parsley stops working from time to time and the VAP closes out. If you are running multiple VAP configs, it doesn't close out all the VAPS. Just the config that gets the error. I'll gather my logs and open another thread on this.

 

[dlflannery]

The issue hasn't returned since purging the VAP configurations, however, Atomic Parsley stops working from time to time and the VAP closes out. If you are running multiple VAP configs, it doesn't close out all the VAPS. Just the config that gets the error. I'll gather my logs and open another thread on this.

Actually there might be basic problems running Atomic Parsley in multiple VAP instances simultaneously. AP is an old command line program that VAP runs as a process. I'm not sure exactly what system resources it needs and what issues might arise running multiple instances of it, i.e, could be very difficult to debug.

How many simultaneous instances of VAP do you run? I have no idea how many user do that -- I don't. Possibly VAP could detect already running instances and delay starting the new AP process until the existing one finishes, similar to what it does to prevent running simultaneous instances of a particular VAP configuration. Thus I would be interested in seeing any pertinent log extracts or error messages.

 

[jtcreate]

Actually there might be basic problems running Atomic Parsley in multiple VAP instances simultaneously. AP is an old command line program that VAP runs as a process. I'm not sure exactly what system resources it needs and what issues might arise running multiple instances of it, i.e, could be very difficult to debug.

How many simultaneous instances of VAP do you run? I have no idea how many user do that -- I don't. Possibly VAP could detect already running instances and delay starting the new AP process until the existing one finishes, similar to what it does to prevent running simultaneous instances of a particular VAP configuration. Thus I would be interested in seeing any pertinent log extracts or error messages.

I have two instances on one system but only one instance needs to run atomic parsley. The one that processes broadcast based NextPVR TS files. The second VAP does commercial ad tracking and creates a VPRJ file. I do the actual cuts. I do this because I want to cut the video after its compressed into an MP4. I have three computers configured this way and they've been this way for a couple of years.. But only one has the atomic parsley issue and its only recent.

A delay is a good precaution. Also having a switch that turns off AP in VAP would also be good.

 

[dlflannery]

VAP uses AP for two purposes: (1) reading metadata from input .mp4 or .mv4 files and (2) inserting metadata into output .mp4 or .m4v files. Taking that into account, are you still sure only one VAP instance uses AP?

In either case I will be curious to see any diagnostic info you come up with including, if feasible, an output file that AP failed to insert metadata into.

 

[jtcreate]

VAP uses AP for two purposes: (1) reading metadata from input .mp4 or .mv4 files and (2) inserting metadata into output .mp4 or .m4v files. Taking that into account, are you still sure only one VAP instance uses AP?

In either case I will be curious to see any diagnostic info you come up with including, if feasible, an output file that AP failed to insert metadata into.

I think AP is stopping during the Post-processing phase using the handbrake script. Haven't seen it in the log but it appears to be happening when handbrake creates the MP4 and atomic is trying to insert the Metadata.

 

[dlflannery]

In an earlier post you said "VAP closes out". Please describe exactly what that means and any messages or log entries that appear at that time. Also, please attach a copy of the post-process script.

Also not certain whether you have ruled out the possibility that more than one instance of AP could attempt to run simultaneously. If this is happening with the AP instance being launched in the post-process script, it may be either difficult or impossible to detect and prevent.

Curious as to why you prefer to use Handbrake rather than VRD for the .mp4 encoding?

 

[jtcreate]

Take a look at the thread titled "Attention: BeyondTV users". Started it a month ago. 94 views and no one seems to care about Beyond TV. It wouldn't be that hard to resurrect even if I do kill it.

Just to be clear here is what could be possible now that we aren't constrained to a BTV format:

1. Specify any xml output format you want, hopefully not TOO complex. It could be just a small augmentation or modification of the current BTV format, or it could be a completely different format. Thought occurs that an xml version of the current pyTivo .txt metadata file might be interesting.

I've been going back and forth on this for a while and I completely agree with you. No need to complicate the whole thing. We can add some elements and that should be sufficient. However, I have another issue. The TVDB hasn't been lacking these days in providing timely and accurate info. I do try to correct it. But, at this point, would it be too much just to parse the Season, Episode, Episodetitle (aka subtitle), Original Air date, Station, description, filename and uid (unique identifier) directly from the NextPVR XML file? The NextPVR XMl file has been more reliable. The rest can still come from theTVDB. Maybe a switch command that can decide between the two processes?

2. VAP currently only outputs a BTV xml file if it can determine the input video is either a movie or a TV series show. It could be modified to produce the new xml file format even if the only metadata it has is the input NextPVR xml file and it can't determine whether it's a TV series or a movie. (Of course meta-based folder/file renaming will fail if the meta items defined in the template aren't available.)

I know I was focus on trying to identify TV from movies before, but I think we don't need to worry about it at this point. We'll just add some missing elements. I'm trying to database my recordings and I've grown to respect how import the uid in segregating duplicate recordings and reruns(unique identifier) is...lol.

Here is the BTV sample:


<?xml version="1.0" encoding="utf-8"?>
<episode-properties>
<property name="EpisodeDescription">With the playoffs looming, the couples must choose between their romances and their careers; new alliances are formed; a bombshell revelation may derail the future of the L.A. Devils</property>
<property name="Title">Hit the Floor</property>
<property name="DisplayTitle">Hit the Floor</property>
<property name="EpisodeTitle">20180814 Hot Streak</property>
<property name="OriginalAirDate">20180814</property>
<property name="TargetStart">131787826330000000</property>
<property name="Genre">Drama</property>
<property name="StationCallSign">BET</property>
<property name="Channel">10605</property>
</episode-properties>


These are the properties that need to be added. "UID" and "RecordedFilename" would definitely have to be parsed from the NextPVR XML file.

<property name="uid"> ?? </property>
<property name="Season"> ?? </property>
<property name="episode"> ?? </property>
<property name="RecordingFile"> ?? </property>


Here is the original NEXTPVR XML that created the BTV File above.


<?xml version="1.0" encoding="UTF-8"?>
<recording>
<filename>D:\RecordDirectory\MCE-02\Hit the Floor_20180814_19002000.ts</filename>
<channel>BET</channel>
<status>READY</status>
<startTime>2018-08-14 19:00:01</startTime>
<endTime>2018-08-14 20:02:00</endTime>
<channel_OID>7508</channel_OID>
<title>Hit the Floor</title>
<subtitle>Hot Streak</subtitle>
<description>With the playoffs looming, the couples must choose between their romances and their careers; new alliances are formed; a bombshell revelation may derail the future of the L.A. Devils.</description>
<uid>EP017330650041</uid>
<firstrun>true</firstrun>
<original_air_date>2018-08-14T00:00:00.0000000</original_air_date>
<season>4</season>
<episode>6</episode>
<rating>TV14</rating>
<genres>
<genre>Drama</genre>
</genres>
</recording>

Thank you.

 

[dlflannery]

@jtcreate

I have this ready to test, pending a couple of questions:

1. In your output format request you have "Season". Do you really want that capitalized?

2. For the season and episode properties do you want at least two digits, e.g., "04" instead of "4"?

Also, is this output file properly called a "BTV" file or what would be a better thing to call it? It appears to be a custom format now. And same question about the input XML file. Is NextPVR the best label for it? (We need something to put in the changelog.)

 

[jtcreate]

@jtcreate

I have this ready to test, pending a couple of questions:

1. In your output format request you have "Season". Do you really want that capitalized?

2. For the season and episode properties do you want at least two digits, e.g., "04" instead of "4"?

1. Capitalization is not important when it comes to the elements. You can use the NEXTPVR xml as a reference.
2. Two digits regardless. We had that problem before and it was a nightmare when it came to archiving the files in Alphabetical order.

Also, is this output file properly called a "BTV" file or what would be a better thing to call it? It appears to be a custom format now. And same question about the input XML file. Is NextPVR the best label for it? (We need something to put in the changelog.)

I don't think the additions to the .BTV file changes the structure that much. I'll give it some thought but I guess we can keep it BTV for now.
NextPVR is also recognized as NPVR. So we can use NPVR or NVPR XML

I forgot another feature. VAP crashes if a TS file is at zero bytes. This happens if the hard drive runs out of space. NextPVR will still continue recording but will generate a TS file with 0 bytes. VAP needs to ignore these files when trying to parse them on startup so it doesn't crash. In fact, I would have it ignore anything that is under 100 bytes to be safe. Maybe it flags them instead by changing color of the name in the view screen.

Also, I notice that there is no flag on files that stall and get kicked to the stall folder.

Thank you.

 

[dlflannery]

Try test executable (NOT AN INSTALLER) Ver. 1.38T2 in this downloaded zip:
https://vap.videoredo.net/VAPexe138T2.zip

Video files in the monitor folder that don't exceed 100 bytes in length will stay labeled as TRANSFERRING forever (and won't be processed).

Use the -metaBTV command line option (not case sensitive) to invoke processing input .xml files and producing output .xml files. XML input data will preempt TVDB data when available.

 

[jtcreate]

I tested it. Most of the XML looks very good. Only one error. The "RecordingFileName" needs to reflect the incoming TS file, not the VAP output name convention.

Also, I'm curious where the "TargetStart" data comes from.

Thanks

 

[dlflannery]

I tested it. Most of the XML looks very good. Only one error. The "RecordingFileName" needs to reflect the incoming TS file, not the VAP output name convention.

Also, I'm curious where the "TargetStart" data comes from.

Thanks

RecordingFileName is taken from the "filename" element of the input xml file. I can't see any way in the VAP code that it could be set to anything else. (??)

What is "TargetStart" supposed to be? I found the following in the BTV documentation on this web page:
http://code.snapstream.com/api/btv47/html/6d1c1eea-b2c5-1653-dbb6-147a54c26717.htm

The value of this property contains a 64-bit integer representing the targeted start of the recording in UTC filetime. This value differs from the ActualStart propertyin that the TargetStart represents when the recording should start according to guide information,and the ActualStart property represents when the recording actually did start, modifiedby factors such as global padding of recording times, etc.

How should I get this from the NPVR data? Or is this saying it must be parsed from the input file name?

 

[jtcreate]

RecordingFileName is taken from the "filename" element of the input xml file. I can't see any way in the VAP code that it could be set to anything else. (??)

My mistake. Its working. I just didn't expect the path to be included.

What is "TargetStart" supposed to be? I found the following in the BTV documentation on this web page:
http://code.snapstream.com/api/btv47/html/6d1c1eea-b2c5-1653-dbb6-147a54c26717.htm

Not important. Just curious.



I did some testing last night. There are some bugs to work out. Uid and RecodingFilename go missing on some of the BTV XML. They are in the original NPVR XML. I attached some samples for your review. Also, the show "25 Words or Less" doesn't parse the XML data. So I included the original file on that as well. That you.


25 Words or Less" ~.se.~ OAD - '''' [RDT-_20180820_18301900]

Thank you

View attachment source xml (good).zipView attachment source xml (Problem).zipView attachment 25 Words or Less_20180820_18301900__(DVR1-NY)__(DVR1-NY)__(DVR1-NY).txt

 

[dlflannery]

My mistake. Its working. I just didn't expect the path to be included.




Not important. Just curious.



I did some testing last night. There are some bugs to work out. Uid and RecodingFilename go missing on some of the BTV XML. They are in the original NPVR XML. I attached some samples for your review. Also, the show "25 Words or Less" doesn't parse the XML data. So I included the original file on that as well. That you.


25 Words or Less" ~.se.~ OAD - '''' [RDT-_20180820_18301900]

Thank you

View attachment 2348View attachment 2349View attachment 2350

Totally confused about the "25 Words or Less .." stuff. Here is what is in your attachment:

XML#: NO 
DVR#: DVR1-NY 
PROC: MCE-01 
 
 
 ------- W R A P P E R   L O G -------- 
 Mon 08/20/2018 
 
Mon 08/20/2018 :: Old Name: "25 Words or Less_20180820_18301900__(DVR1-NY)__(DVR1-NY)" 
Mon 08/20/2018 :: New Name: "25 Words or Less_20180820_18301900__(DVR1-NY)__(DVR1-NY)" 
        1 file(s) moved.
Mon 08/20/2018 :: Successful move .TS to wrapper: "25 Words or Less_20180820_18301900__(DVR1-NY)__(DVR1-NY)" 
Mon 08/20/2018 :: failed to move .xml to wrapper: "25 Words or Less_20180820_18301900__(DVR1-NY)__(DVR1-NY)" 
Mon 08/20/2018 :: Successful Rename .TS to: "25 Words or Less_20180820_18301900__(DVR1-NY)__(DVR1-NY)__(DVR1-NY)" 
 Mon 08/20/2018 :: 
 Mon 08/20/2018 :: 
 Mon 08/20/2018 :: First stage Wrapper complete 
 Mon 08/20/2018 :: 
 Mon 08/20/2018 :: 
Mon 08/20/2018 :: DETERMINE PROCESSING THROUGH FILE LOCATION  
Mon 08/20/2018 :: MCE-01 PROCESS DETERMINED 
Mon 08/20/2018 :: FINAL MANAGE PROCESS STAGE 
Mon 08/20/2018 :: DETECTING AND REMOVING DUPLICATES FROM LOCAL LIBRARY 
Mon 08/20/2018 :: MOVING .TS COPY TO LOCAL LIBRARY 
Mon 08/20/2018 :: Successful .TS copy to Local Library: "25 Words or Less_20180820_18301900__(DVR1-NY)__(DVR1-NY)__(DVR1-NY)" 
Mon 08/20/2018 :: Failed to copy .XML file to Local Library: "25 Words or Less_20180820_18301900__(DVR1-NY)__(DVR1-NY)__(DVR1-NY)" 
Mon 08/20/2018 :: RENAMING .TS COPY IN LOCAL LIBRARY WITH X IDENTIFIER 
Mon 08/20/2018 :: Successful Rename: "25 Words or Less_20180820_18301900__(DVR1-NY)__(DVR1-NY)__(DVR1-NY)-X1.ts" 
Mon 08/20/2018 :: RENAMING .XML COPY IN LOCAL LIBRARY WITH X IDENTIFIER 
Mon 08/20/2018 :: failed Rename: "25 Words or Less_20180820_18301900__(DVR1-NY)__(DVR1-NY)__(DVR1-NY)-X1.XML"

This has no meaning to me -- not anything related to VAP as far as I can tell, and not recognizable to me as a valid NPVR input xml file.

So you don't care about the TargetStart element? If so, I'll just remove it from the output xml.

I'll be looking at the sample good and issues files.

 

[dlflannery]

OK some kind of mix up here with the VAP executable you're running. Two indications are:

1. When I run with exactly the same input xml for your "burden of proof …" example, I get the uid and recordingFileName elements in the output
2. In my output the season and episode numbers are 2-digits whereas they are single digits in your output.

So either one of two things has happened:
1. I provided the wrong executable in VAPexe138T2.zip
(or)
2. You didn't get the new executable installed in the runtime folder. Before any further changes, run VAP and see what version is displayed at the top.

Windows can make it tricky to replace files in the Program Files (86) folder tree. To be absolutely sure I usually force delete the .exe being replaced, which will require admin permission. Then copy in the new .exe.

Since I can't absolutely rule out that I provided a bogus .exe, I have placed a new one, Ver. 1.38T3, at this download link:
https://vap.videoredo.net/VAPexe138T3.zip

 

[jtcreate]

I sent you the wrong file. Not sure were the original XML went. So, I am not going to worry about it. Maybe I am mistaken and NPVR never generated one.