Handbrake Encode Option?

[tannebil]

Things have been pretty quiet lately so I figure you must be getting bored. How about an option to use Handbrake to do H.264 encoding? I don't want to diss on VRD but my experience is that Handbrake produces faster, smaller, and higher quality than the encoder used in VRD. That said, VRD is much better in other areas, e.g. TiVo files, WTV files, QSF fixing files that HB can't handle, editing, commercial scaning, while VAP fills in the missing holes of workflow and metadata.

The ideal combination for me would be if I could configure VAP to do a QSF in VRD and use Handbrake to do the encodes.

Just to be crystal clear, I'm looking for an encoding alternative to VRD, not a replacement for VRD.

Any interest?

 

[dlflannery]

I can't see myself wanting to handle the complexities of setting up x264 command line parameters. However, using the Handbrake command line program for H.264 encoding in a VAP post process script has been common for a while. The HB parameters can be perfected using the GUI version and pasted into the script. The catch is getting the metadata into the resulting .mp4 (or .m4v) file. There are two options for that, both of which would require you to configure VAP to produce pyTivo metadata files. These are programs that read pyTivo MD files and insert the MD into mpeg4 files:
pyTivoParsley and pyTiVoMetaPopper (available on VAP download site).

pyTiVoParsley:
The documentation is in the zip download. I think it needs a little updating to get the optimum MD structure and it will never quite equal the MD insertion done by VAP because the pyTivo MD file doesn't capture as much info.

pyTiVoMetaPopper:
The documentation is in this thread. This program works on both mpeg4 and wtv/dvr-ms targets and thus carries extra baggage if all you want is mpeg4 insertion. It also needs to be updated slightly so it doesn't require the input video file to be present in order to run. A major factor to be considered is that it works on all file-sets in the input and output folders each time it is invoked. Thus if you want to use it on a per-file basis you have to move or delete one of the files after each invocation.

My guess is you would prefer to use an updated version of pyTivoParsley, although the current version is plenty good for test purposes.

You can download an example post process script for HB encoding here. It's a little outdated but the required mods should be fairly obvious.

I'll be glad to help as needed as you get further into this.

Another approach comes to mind:
VAP uses AtomicParsley to insert MPEG4 metadata so it already composes a string of command line options for AP. It could dump that string to a text file (invoked by a new VAP CL arg) in the output folder. Then the postprocess script could run AP directly using those options, thus resulting in metadata insertion exactly the same as what VAP now does. Since VAP already distributes the AP executable, the user would not even have to download it.

 

[tannebil]

I used to do everything with command files but switched to VAP because you write C far better than I write command files.

I was always figuring that I would have to hand code the x264 parameters (which I'd use HB to create) and VAP would just provide the calling framework.

 

[dlflannery]

So let me know which of the three options you prefer and I will do the clean up or VAP mods for them. I like the third one best since it will give more complete metadata insertion..

 

[tannebil]

I think #3 is the best way to handle the metadata but I'm not quite ready to pull the trigger on making any changes quite yet.

Is this the workflow you are talking about?

1. VAP detects the new file
2. VAP does the metadata lookup and transformations
3. VAP runs a QSF and dumps the metadata into an intermediate file.
4. VAP calls VAPpostprocess.cmd
5. VAPpostprocess runs HB
6. VAPpostprocess loads the metadata using AP
7. VAPpostprocess runs something (adscan.vbs?)that does ad detection, creates a .vprj file, and creates a .drx file
8. VAPpostprocess run Drax to load the chapter marks

How do errors get trapped and alerted? How do results get logged? I'm doing 0-6 programs per day and sometimes can't check things for a week.

 

[dlflannery]

I envision this process flow:
1. VAP detects the new file
2. VAP does the metadata lookup and transformations
3. VAP runs a QSF and dumps the metadata into an intermediate file.
3a. VAP does ad detection, creates a .vprj file, and creates a .drx file
4. VAP calls VAPpostprocess.cmd
5. VAPpostprocess runs HB
6. VAPpostprocess loads the metadata using AP
8. VAPpostprocess runs Drax to load the chapter marks

Steps 1 thru 3a are the same as you run now I think, but VAP needs mod so it will produce the .drx file when you configure output profiles other than MPEG4.

As for logging of errors or other items: I believe there is a feature already in VAP that allows postProcess scripts to insert messages in the VAP log file by echoing to a specific text file name. This feature was in TVAP and I need to verify it migrated into VAP and add it to the VAP readme file. Here is the description in the TVAP readme file:

If your TVAPpostProcess or TVAPcleanUp batch file writes text lines to a file named TVAPlog.txt located in the folder containing the batch script, these text lines will be added to the TVAP log display and log file AFTER the batch completes. TVAP will prepend "BATCH MSG: " before each log line.

 

[tannebil]

So (3) becomes

3. VAP runs an QSF with an mpg output profile, runs an ad scan against it, and creates a .drx file

Since the drx file contains time offsets, it should work equally well with the HB output file.

The only changes to VAP will be to:

1. Create the intermediate metadata file
2. Create a .drx file file with additional output profiles (I think I only need mpg if that makes a difference).

I'll need (9) to delete the mpg file and the drx file.

 

[dlflannery]

Attached are VAP exe 0.69T1 and a sample portion of a postprocess script that shows how to run AtomicParsley.

On the "Other" tab of advanced configuration, check both drax generation and atomic parsley option file generation. Then configure QSF only and your mpeg2 output profile. The AP option file with filename and extension .apo appears in your output folder before the postprocess script is run.

I don't have HB latest version running CLI yet so for testing I created a .mp4 file in the output folder beforehand using VRD. The attached script also shows how to send log messages back to VAP with timestamps.

 

[tannebil]

Works great so far. Of course, my command file is much uglier but it gets to job done.

If you are wondering if it's worth the trouble, here are two clips showing the difference.

VRD: http://dl.dropbox.com/u/164545/720p clip - VRD iPhone.mp4

Handbrake: http://dl.dropbox.com/u/164545/720p clip - HB iPhone.m4v

Original: http://dl.dropbox.com/u/164545/720p clip.mpg

The HB clip is 50% larger but when I encode an entire show, HB and VRD produce files with an almost identical size, bit rate, and resolution. The unusually small size of the VRD clip and the dramatic reduction is quality makes me think it might be a bug in the encoder. It only seems to happen a once or twice during a program with most action scenes being of similar quality.

The Handbrake output files are not without issues. While they play fine, VRD hates them. On opening some files, VRD reports a seek error. Or VRD will go non-responsive for long periods of time when moving around the file. Or the thumbnails will stop moving instead of changing as you move around in the file. In addition, VRD always reports the frame rate as 90000 fps. I've seen these issues on most of the Handbrake files produced so far.

Of course, there's always the possibility that I'm doing something stupid or there is a bug related to the processor type. I'm actually changing my machine to an Sandy Bridge i5 2500 this week so it will be interesting to compare run times and see if the issue might go away.

I don't actually want to use Handbrake as VRD/VAP are a great combination but the quality hit was driving me crazy.

 

[dlflannery]

Interesting results -- thanks. I would be interested in your HB command line options if you don't mind posting them.

Is the HB CLI exe download distinct from the one packaged in the GUI HB download? At one time they were the same exe then I think they became distinct.

The VRD/HB comparisons and issues are bothersome. Guess you just have to be flexible and keep trying til you get something acceptable.

The two clips give the impression of being different format sizes or letterboxed differently? They are all playing at 100% size in WMP (so it says) but the two mpeg4 versions are much smaller than 720 while the mpg looks like 720. Were the MPEG4's resized to 240? Would be curious as to what mediaInfo says about them.

In my tests, the thumbnail insertion part of the AP runs was working properly. Does it work for you?

I googled Sandy Bridge (wondering if it was an Aussie PC brand ). I found this recent PC World article you might want to be aware of:

http://www.pcmag.com/article2/0,2817,2379241,00.asp

 

[tannebil]

HandbrakeCLI is included with the GUI download. My CLI arguments are largely non-existent with -i <input file name> -o <output file name> --preset=""iPhone & iPod Touch". I am doing some experimentation with adding -D to try and boost the volume but that's it.

The H264 clips are lower resolution so that they will run on on an 2nd generation iPod Touch which has a maximum horizontal resolution of 480. I never understood why VRD takes 16:9 material and encodes it to 480x320 rather than 480x272 but the aspect ratio looks fine and the file size is pretty much the same so I'm guessing it's just encoding the letterboxing black bars. The Handbrake preset says 480x240 but the output is 480x272 which I assume is the result of "Keep Aspect Ratio" being checked.

I've tried changing the Aspect Ratio in the VRD output profile but the file always seems to come out 480x320. But I know that there is all kinds of complicated stuff related to aspect ratios, the 16x16 pixel size of the blocks used in H.264, auto cropping, anamorphic encode and zillion other factors. In the end, I just try to find something that works.

The issues with the Handbrake files are hard to duplicate in the short clips but tend to be quite pronounced on the full-length versions. The thumbnail issue I mentioned is the thumbnail strip shown in the VRD GUI between the video and the time slider. Adding cover art thumbnails via VAP works fine.

The Sandy Bridge issue was in the supporting chipset so it was on the motherboards rather than the CPU. The problem was corrected and the manufacturers are shipping boards with the fix. I'm projecting at least 2x increase in encoding performance in Handbrake and something closer to 3x in VRD. VRD doesn't seem to scale well on AMD quad-cores with the cores usually only running at 60-70% compared to HB which runs the cores at 95-100% which is why I'm hoping for the bigger bump.

There are lots of different multimedia extensions on AMD and Intel processors which is why I have a small hope that's the source of my issue. I don't see anybody else complaining about CPU under-utilization or quality glitches.

Try encoding the MPG flip using VRD and the iPhone/iPod Touch preset. I'd be interested in the quality of your result.

I'm not Australian. I'm American and live on the USA West Coast.

 

[dlflannery]

Sorry, I must have confused you with another user or misread one of your posts to think you were an Aussie.

I had an issue with low VRD CPU usage some months ago and even had it addressed as a support issue. It was ~35% on my i7 quad core. The bottom line was that the resizer wasn't optimized for multi-threading. I don't know if VRD claims to have fixed this or not but IIRC I've still seen low usage on recent encodes. Encodes that don't do resizing achieve much higher usage.

 

[tannebil]

No problem. I'm not offended being mistaken for an Aussie.

All my encodes are resizing so that's bad news about the resizer not being multi-core optimized.

On raw video encoding benchmarks the i5-2500 is twice as fast as my Athlon 9850. The Quick Sync capability promises another 5x improvement in encoding performance but I expect it will be at least 6 months before it finds its way into the products that I use.

 

[dlflannery]

Just thought I would point out the new feature to produce options files for AtomicParsley works no matter what the process flow is, not just when producing drax chapter files.

 

[tannebil]

Ran into my first issue. I was experimenting with the Handbrake preset for the Apple TV 2 and found that AtomicParsley was not up to the job of adding the medatadata for the program. AP throws an error about not having full 64-bit support and that I must be crazy to think that it would work with a 2.47GB file. I always thought that 4GB was the magic file size limit for 64-bit (I'm running Win 7 64-bit).

Have you done any large H264 transcodes where you've seen this problem?

Any idea what the magic number is for AP as far as maximum file size?

 

[dlflannery]

I googled "AtomicParsley 64 bit" and quickly found this:

http://tntluoma.com/technology/atomicparsley-handbrake-appletv-bug/

This looks like the problem you encountered. The file doesn't have to be greater than 4 GB for this to be a problem for AP. Note that the suggested solution, creating a modified Apple TV preset, will not work for command line HB since it won't use custom presets. However you should be able to get the CL options for the Apple TV built-in preset, modify them, then paste into your script file.

I looked for any available newer versions of AP that might handle the 64-bit atom but didn't find anything. AP is old and has some limitations (including this one and not supporting long description) but it is solid and has a command line implementation. I can't identify any better approach.

 

[tannebil]

I'll redo it from the GUI without that setting and see what happens with AP.

The Handbrake 480x272 encodes for my Touch actually look pretty good when watched on my 61" JVC through my Apple TV although they will never be confused with watching the HD program on my TiVo S3. The VRD Touch encodes look pretty ragged in comparison. The differences when watching on the Touch are not really that noticable except during scenes like the one I posted.

Just out of curiosity, what are you doing with VAP/VRD? I assume you are encoding TiVo to H.264 but what are your target devices? What output profiles are you using?

 

[dlflannery]

I'm not doing much encoding for myself lately other than for test purposes. I spend more time modifying, fixing and testing VAP, which I (preversely?) enjoy doing. A couple of years ago I was transcoding SD movies from the TCM channel to mpeg4 using HB and building an archive of 640x480 videos to view via pyTivo or StreamBaby. In July 2009 I got an HD TV, replaced my S2 TiVo with a TiVo HD, and switched from analog to digital cable --- and my video world changed in more ways than one. First, Time Warner copy protects all digital channels except locals so my only source of videos to transcode almost dried up. Also, it has been a minor hobby project just to keep my TiVo and Tuning Adapter working -- still missing some SDV recordings because of the TA tuning failure problem. I gather some folks get their videos via BitTorrent but I decline to do that. I'm kind of relying on my 1 TB TiVo capacity (which could easily be upgraded to 2 TB for around $100) plus Netflix and Amazon, and don't have as much interest in archiving mpeg4's anymore. One deterrent is the time it takes to transfer HD via TTG. The TiVo Premiere transfers 2 or 3 times faster which would help there. I think the future of TV will be Internet VOD and it can't replace cable fast enough for me!

I test with a lot of the built-in profiles and have a 1280x720 H.264 custom profile that I developed for archiving (but haven't used much).

 

[tannebil]

Sounds like our hobbies complement each other. You like building a workflow for recording and encoding while I like using it.

I've heard about issues with Time Warner excessive use of the "copy once" flag where "once" means the original copy on the DVR. I guess I'm lucky to have Comcast which is better behaved in that regard. We've also been spared SDV so far but I'm sure our luck will run out eventually. There was even a period of time where Comcast was using clear QAM for everything but the non-premium stations which was sweet for recording using my HD HomeRun. It's just the broadcast stations now so I have to use use TiVo for about half my shows.

I copy everything on my primary TiVo to my PC as both a backup and to keep my TiVo from running out of space. It's got the original 250GB drive plus a 1GB external but space becomes a problem if I'm gone for a couple of weeks. I make a second backup by copying everything from the Windows machine to a Mac server (just an old Powerbook with lots of external disk space). The transfer speed makes a new Premiere attractive.

Anyway, turning off "large disk support" resolved the problem with AP.

The next problem I'm working on is trying to figure out why the ATV2 is only playing stereo rather than surround sound even when the file contains both stereo and 5.1 tracks. It could be my new ATV2, my old Denon receiver, some interaction between them, or even something related to the way Handbrake created the audio tracks. Should be fun to figure out.

 

[dlflannery]

Let me know when this test version has enough mileage on it to justify making a release of it.