Support for tivo tystreams

[AlphaWolf]

Hey, I just discovered this editor recently, so far it looks like it does an excellent job with pretty much any mpeg converted stream I throw at it.

Seeing as you already are supporting the dishpvr streams, I am wondering if you guys will consider supporting tivo tystreams as well, both from directv tivos (satellite receiver integrated) and standalone tivos. As you probably know by now, tivo tystreams are mpeg-2 transport streams (although I have been told they more resemble program streams) packaged in a somewhat proprietary format.

I know your project is commercial and that you probably want to go wherever the money is, but I am not a market expert, so with that in mind I am not quite sure what to say about the commercial market for tivo owners. However I can tell you that there is a very good sized tivo community called deal database with many people interested in video extraction and editing. You can find it here (http://www.dealdatabase.com/forum/forumdisplay.php?f=43).

Now there currently are a few freeware tystream conversion tools. One called tystudio is open source and it does a good job of processing the streams into good standard mpeg but it hasn't seen any updates in a while. Another called tytool is closed source, sees occasional updates, and has many features but it can't always convert tystreams to proper mpeg-2, and the author hasn't shown any interest in correcting these issues either, thus you can't use any of the streams it creates in any other editor without encountering severe problems (except videoredo of course, yours seems to do a good job of fixing the mpeg errors that tytool produces, although it does an even better job with the tystudio processed streams as I mentioned here (http://www.dealdatabase.com/forum/showpost.php?p=190143&postcount=32).)

Over the last few days I have been giving your software quite a bit of praise (both here (http://www.dealdatabase.com/forum/showthread.php?t=38704) and here (http://www.dealdatabase.com/forum/showthread.php?t=38510)) as it seems to work quite well with tystudio processed streams and a few people seem rather interested. Based on what I have seen so far, you can probably do a much better job with the raw unprocessed tystreams than all of the freeware editors currently do, and seeing how you make frequent updates it would probably generate a lot of interest in the deal database community.

I know you already have a lot on your hands with some other features you already mentioned you are going to add, but if it isn't too far out of your way I would like you to take this into consideration. I have a few sample streams I can send you if you are interested.

[Anonymous]

I second AlphaWolf's request for consideration of Tivo support. The DealDatabase Tivo Extraction Forum was the one I mentioned in my email. I had planned to use one of the freeware demuxers for tivo files, then edit the output with VideoReDo, but more direct support of Tivo-derived transport streams (or tystreams) would be even better.

[DanR]

I am an avid Tivo user, we have two standalone units here (3 actually, but one has a very wierd software problem so its not being used), but must admit I rarely edit Tivo programs. If there's something we want to archive, we will use one of the digital capture cards we have, (Hauppauge, ATI, NVida). That being said, I have used Ty Studio on occasion without any problems.

Our support for DishPVR streams is similar to our Tivo support. A conversion program creates PS streams which we then edit. A big plus for the DishPVR users is our ability to handle 3:2 pulldown and correct sync which are issues with the Dish streams.

tystream conversion tools. One called tystudio is open source and it does a good job of processing the streams into good standard mpeg but it hasn't seen any updates in a while. Are there any issues with tystudio? At least with respect to extracting streams and creating an MPEG PS?

We won't access the Tivo to extract the .ty streams like tystudio can unless we receive permission from Tivo which is unlikely. We will look at the possibility of supporting .ty streams once you've imported them to your PC, so long as there is no decryption involved.

[AlphaWolf]

Are there any issues with tystudio? At least with respect to extracting streams and creating an MPEG PS?

There are a few of them. Namely it's support for the series 2 streams is mostly broken, and although I haven't seen it myself, many people report the occasional stream going out of sync with tystudio. There also is one little bug that hasn't really bothered me yet, but it seems to cut off the last few seconds of any tystream you run through it (which are usually credits or something, but still it probably shouldn't do that.) And lastly, tystudio does not support high definition tivo streams.

We will look at the possibility of supporting .ty streams once you've imported them to your PC, so long as there is no decryption involved.

That will work fine.

In fact to further that I think if it was really wanted, there are probably numerous ways that the freeware scene could go about creating a wrapper so that your software doesn't know much of the difference between whether the tystream is on a tivo itself or is on the local PC. The only problem here is if videoredo needs to do anything beyond reading the stream itself in real-time (e.g. creating key files or something the way that tystudio does before it can even open the stream.) Barring network speed there wont be any problem.

Also decryption will never be an issue.

EDIT: Something else I almost forgot about, and this would help your entire userbase abroad rather than just tivo owners. Back in the day, I remember Olaf_SC talking about being able to upscale video from 480x480 to 720x480 in the compressed space, or without re-encoding with a quick and lossless method, and he was talking about adding this to tystudio, but unfortunately nobody has heard from him in well over a year. Perhapse you might be able to do that?

Combined with your gop resizing feature, this could make all tivo tystreams as well as SVCDs 100% compatible with DVD spec.

[rc3105]

We won't access the Tivo to extract the .ty streams like tystudio can unless we receive permission from Tivo which is unlikely.
I wrote a tivo-side ftp server (http://www.dealdatabase.com/forum/showthread.php?t=21915) that allows any standard client - including browsers - to retrieve shows. you could support direct tivo extraction merely by adding the ablility to access standard ftp servers ;)

maybe not particularly high on anybody's wishlist, but it'd be usefull enough for non-tivo people to justify implementation (not to mention the same sort of ftpd is possible for dish, myth & other video collections and sources)


--
Riley

[DanR]

I wrote a tivo-side ftp server that allows any standard client - including browsers - to retrieve shows. you could support direct tivo extraction merely by adding the ablility to access standard ftp servers Does your server convert the program content into standard MPEG2 PS?

[rc3105]

nope, been down that road & the tivo really doesn't have the horsepower. I was a tystudio contributor, the tivo port topped out at about 750K/sec doing ty->mpg so it didn't seem particularly usefull

everybody and their dog seems to want to invent new extraction protocols (tivo included) my public utils just move data from point a to point b & allow for restoring archived recordings

[DanR]

Do we need additional software on the Tivo, besides your FTP server to access the TY stream? Its been a couple of years since I loaded TYstudio on my standalone Tivo. Can't remember exactly what I did.

Maybe you can help me with my standalone Tivo problem. I have a Phillips Series 1Tivo, upgraded to TurboNet 10/100 after the modem died and 100 hours of disk space. The TurboNet works great, I can FTP and telnet to my hearts content. But, the Tivo will no longer access the guide data. Even if I do a test call, the test gets hung up while "hanging up". It says, setting clock, then "hanging up" but never goes any further. Can still access it via ethernet, but can't get guide data to load. So, I broke down and bought another Tivo, 40 hr, Series II. But, I'd love to get the orignal one working again. Any ideas?

[rc3105]

mfs_ftp requires a few support files that're included in the download, but not any other utilities to be running

with s2 versions of the support binaries mfs_ftp works on everything from the original sa2 through the hdtivo & pioneer/humax dvd burner combo units

---

couple of ways to get the s1 goin, listed from easiest to hardest

change the daily call from phoneline to ethernet & route it through your home network - use dialing prefix ",#401"

connect an external modem to the serial port - use dialing prefix ",#396"

run a serial cable from tivo -> pc & configure it as a serial ppp server - use dialing prefix ",#296" (big pain)

generate your own guide data - not particularly difficult, but gotta be done on a weekly basis

---

here's a bit of a pie-in-the-sky feature request

lot of people are interested in putting mpg from other sources into their tivo. a "convert-to-ty" feature would sell a LOT of copies of VideoReDo

*converting mpg/vob/ts into tivo compliant ty isn't hard, unfortunatly everybody wants support and nobody wants to pay for it - allready done my good deed for the millenium :P

[DanR]

couple of ways to get the s1 goin, listed from easiest to hardest

change the daily call from phoneline to ethernet & route it through your home network - use dialing prefix ",#401" Its already configured as ,#401. Worked for just over a year without problem. Then we had a power failure here and it never worked again :( Thought it would be the TurboNet card, but that's working just fine, everything else about the Tivo box is working OK as well. Couldn't get any help on any of the Tivo forums.

lot of people are interested in putting mpg from other sources into their tivo. a "convert-to-ty" feature would sell a LOT of copies of VideoReDo Is it hard to stuff the info back into the MFS? Why not simply purchase the Home Media option and play your files stored elsewhere on your home network?

[rc3105]

hard to say then. if the networking & gateway config are all correct you should be able to ping or http_get files off the net. if http_get works but it still can't retrieve guide data then perhaps something has gone awry in mfs, check your logs after a "dialin" attempt to see exactly where it failed

trying a mfs cleanup with "mfsassert -please", clear & delete, or loading a fresh image to another drive (to test w/o losing current recordings) might be the next steps

---

it's not hard to put content into mfs. HMO is free now, but it's pretty limited and tivo2go isn't much better. doesn't work with any s1 or any dtivo model, can't access video on anything except another tivo and can't use files from any other source

[AlphaWolf]

Is it hard to stuff the info back into the MFS? Why not simply purchase the Home Media option and play your files stored elsewhere on your home network?

Actually getting stuff onto the tivo is the easy part (in fact you can transfer recordings from your S2 to your S1 if you want to, even though the S1 doesn't support HMO.) The currently impossible part is converting regular mpeg into a tystream.

If you could do that, then you aren't just limited to moving content from tivo to tivo, you can put just any arbitrary content on the tivo (or at least a directivo anyways, which doesn't have any mpeg encoder hardware.) Not only that but you could also edit the commercials out of the tivo streams, and then put them back on the tivo. IIRC a lot of people were looking for a way to do that a while back.

BTW, is it conceivable that you'll ever add a compressed space upscaling feature like what I mentioned?

[rc3105]

mpg -> ty is easier than ty -> mpg. the tivo is pretty flexible, just need to wrap the mpg packets in ty chunk headers & create a master chunk if you want ff/rw/skipto to work

if you use sa2 packets as the ideal standard to construct ty they play back in everything. using the 810 packet format lets you burn from dvd combo units but they won't play in s1 hardware

[AlphaWolf]

They would also probably have to be in 720x480 in order for the 810 to burn them. Unless it is possible to change the recording resolution on the 810 and still be able to record DVDs with it?

[Dan203]

They would also probably have to be in 720x480 in order for the 810 to burn them. Unless it is possible to change the recording resolution on the 810 and still be able to record DVDs with it?

Actually the Pioneer support both 720x480 and 352x480, so either should work.

However I'm not sure about the audio format. By default the Pioneer records audio in AC3 format. I'm not sure if they would accept files with MP2 or PCM audio even if they were properly converted to Ty streams.

Dan

[Rowan]

There are a few of them. Namely it's support for the series 2 streams is mostly broken, and although I haven't seen it myself, many people report the occasional stream going out of sync with tystudio. There also is one little bug that hasn't really bothered me yet, but it seems to cut off the last few seconds of any tystream you run through it (which are usually credits or something, but still it probably shouldn't do that.) And lastly, tystudio does not support high definition tivo streams.

I am looking into the sync and last few seconds of the stream problems. I have no S2 streams or HD streams to play with but once I come up to speed I may try and get some.


EDIT: Something else I almost forgot about, and this would help your entire userbase abroad rather than just tivo owners. Back in the day, I remember Olaf_SC talking about being able to upscale video from 480x480 to 720x480 in the compressed space, or without re-encoding with a quick and lossless method, and he was talking about adding this to tystudio, but unfortunately nobody has heard from him in well over a year. Perhapse you might be able to do that?

I remember that as well and have been looking through the old emails but have not found it yet, may be on sourceforge. That would be very cool if we could do that along with VideoReDo to cut out unwanted parts and finish making the stream DVD compliant.

[DanR]

about being able to upscale video from 480x480 to 720x480 I've been having a similar conversation with someone from the Dish community offline about the very same question. His idea was to pillar box the edges of the video and then use the pan and scan features of a DVD to expand the original material onto a full 4:3 screen at playback.

Technically that's feasible and could be done in the compressed space, but I think we are back to the problem we have discussed in other threads which is how to get the DVD players to honor the pan & scan output.

So here's the question, has anyone gotten the P&S working on their DVD authoring program and player reliably? If so, then we could probably do the pillar box without too much effort and convert it to 704 / 720 x 480.

[AlphaWolf]

I am pretty sure that olaf was talking about real scaling and not doing any pan and scan tricks. I *think* he mentioned something about documentation of the algorithm/process for doing it in the compressed space existing somewhere but no code for it had actually been written.

IIRC this was part of an IRC discussion though so it probably wont be on any web forums.

EDIT: Heres a mention of it: http://www.dealdatabase.com/forum/showpost.php?p=84630&postcount=4

A search for his specific terms yields these:

http://www.cs.ccu.edu.tw/~cwlin/pub/dctxcodervis02.pdf
http://www.hpl.hp.com/personal/Susie_Wee/PAPERS/hpidc97/hpidc97.html

Theres also one I found in a google summary that specifically mentions upsizing, but you need a login/password to get into that site.

EDIT2: Also found this: http://jpegclub.org/jidctred/index2.html

Hope that helps.

[DanR]

Those are some interesting papers, thanks for the links. They seem to be more concerned with downscaling rather than upscaling, but similar concepts might apply.

There's actually some interesting things to think about here. A DVD stream can have about 9+ Mbps of video bandwidth, yet Tivo (and Dish for that matter) strems are much, much lower. This means that we can increase the overall video bandwidth by not optomizing the motion vectors in the transcoding. We can utilize the original vectors appropriately scaled to the new frame size. The benefit of this is speed, since motion vector search is the most time consuming aspect of MPEG encoding. The downside is a possible increase in bandwidth without a comensurate quality benefit.

[AlphaWolf]

Would be one killer feature for *all* users nonetheless, especially if you can preserve the picture quality as olaf mentions. At the very least you could directly convert SVCD to DVD compliant video if you were so inclined.

[AlphaWolf]

I noticed in one of the documents they were talking about DCT conversion of I frames to P frames. Perhapse that would provide a 100% lossless (and faster) method of GOP resizing?

[DanR]

I noticed in one of the documents they were talking about DCT conversion of I frames to P frames. Perhapse that would provide a 100% lossless (and faster) method of GOP resizing?That's a good thought as well. I'm also starting to look into other things that can be done in the DCT space. This could be a lot of fun.

[Anonymous]

Just curious, have you guys decided whether or not you are going to support this yet?

[DanR]

We haven't even begun to research this yet. Honestly, I would expect anything until at least Q2 '05.

[Anonymous]

I bought this product specifically to help me convert from my tivo files to useable MPGs for archving on DVD. Currently I have to use TyTools to convert into a "dirty" mpg and then I clean it with VideoReDo - it would be great to cut out the middle step....

[gyre]

I stumbled over this product while trying to fix up the mpegs producted from tystudio and tytool.

I'm just plain amazed at how well and fast it works. Video streams that completely baffled my toshiba DVD player now just work.

Like the last poster, if this could handle tystreams (or even just extracted .ty files) natively it would save so much extra hassle and time for me.

As it is, I already consider this a killer application for tivo users.

-- gyre --

[DanR]

Can one extract from the Series II or is extraction limited to the Series I?

Also I looked at the source for the Ty to MPEG conversion and while its not rocket science it isn't trivial either. If someone wants to help out on the programming for this we could really accelerate the implementation.

[Rowan]

You can extract from both series I and series II including the HD (High def) DVD HR10-250. There are two main camps out there the TyTool and TyStudio, currently TyTool is the only supported tool (by Jdiner), the TyStudio project has not been worked on in quite some time and only partially supports series II streams. Since the ‘ty’ format is not officially documented the programs need to be changed as new issues are found, D* is always changing the way they compress things so it takes work to keep up.

TyTool fully support’s most of the tivo streams the only part that is missing is full support for the HD unit, I would say it is about 85% there. The only problem with this tool is no source code is available and Jdiner does things at his own pace and refuses to let anyone help most of the time, he did accepted my code to transcode the audio but was the most I was allowed to help and it was an external DLL.

TyStudio works properly with only series I streams and has partial support for series II streams, there are some known bugs, some of them I know about and have possible fixes others I don’t. I was one of the people that originally worked on this project and did all of the audio work on it but did not do much of the video side of things. It would take a lot of work to get this program in shape to support all of the different Tivo versions again.

[DanR]

Thanks for the information. If this is a continously moving target its going to be very difficult for us to find the time to support this by ourselves.

[Rowan]

Thanks for the information. If this is a continously moving target its going to be very difficult for us to find the time to support this by ourselves.

It is not that bad, I would say maybe once or twice a year there may be something that comes up. The main thing is stream corruption caused by rain fade, bad transfers or just missing packets. Currently the TyTool has over a year on TyStudio on fixing all the issues so that is were the work would really be.

[DanR]

Here's some info that you might find interesting if you hadn't seen it before. I plan on stopping by the Tivo booth at the CES to learn more about it.

http://news.zdnet.com/2100-1040_22-5510240.html?tag=zdnn.alert

[AlphaWolf]

Hate to resurrect a dead thread, but have any of the VideoReDo developers made any headway on either tivo streams, or DCT compressed space video manipulation?

FWIW, TivoToGo has many limitations. The biggest being that it doesn't work on directv streams (only standalone users can use it, satellite users can't) and thus it also doesn't work with high definition tivo streams.

[DanR]

No work has been done on TyStreams nor is it something we are actively pursuing at this point. We are in the process of creating an "open interface" which will let ourselves and 3rd party developers add additional MPEG stream reading capabilities. Perhaps once this is published, someone who is familiar with TyStreams would like to use it to add that support to VideoReDo.

DCT-spaced maniuplation is still on our development list, nothing further to report at this point.