Vprj file containing Unicode stars in filename fails to open

SnoopyDog

Member
Hi

A comskip generated Vprj file dropped onto VRD latest beta containing the filename 20180601_20-59-02_Kanal 5_Gravity - ★★★★★★★★☆☆.ts fails to open. Dropping the actual .ts file with the stars onto VRD opens it ok though.

<VideoReDoProject Version="3">
<Filename>F:\TV\Recording Service\20180601_20-59-02_Kanal 5_Gravity - ★★★★★★★★☆☆.ts</Filename><CutList>
<InputPIDList><VideoStreamPID>1059</VideoStreamPID>
<AudioStreamPID>1058</AudioStreamPID><SubtitlePID1>0</SubtitlePID1></InputPIDList>
<Cut><CutTimeStart>0</CutTimeStart> <CutTimeEnd>832800000</CutTimeEnd> </Cut>
<Cut><CutTimeStart>13209200000</CutTimeStart> <CutTimeEnd>17501600000</CutTimeEnd> </Cut>
<Cut><CutTimeStart>29575200000</CutTimeStart> <CutTimeEnd>34057600000</CutTimeEnd> </Cut>
<Cut><CutTimeStart>45644400000</CutTimeStart> <CutTimeEnd>50384000000</CutTimeEnd> </Cut>
<Cut><CutTimeStart>63358400000</CutTimeStart> <CutTimeEnd>64755600000</CutTimeEnd> </Cut>
</CutList>
<SceneList>
<SceneMarker Sequence="0" Timecode="0:00:58.28">582800000</SceneMarker>
<SceneMarker Sequence="1" Timecode="0:01:13.51">735200000</SceneMarker>
<SceneMarker Sequence="2" Timecode="0:01:23.28">832800000</SceneMarker>
<SceneMarker Sequence="3" Timecode="0:22:00.87">13208800000</SceneMarker>
<SceneMarker Sequence="4" Timecode="0:22:22.68">13426800000</SceneMarker>
<SceneMarker Sequence="5" Timecode="0:22:25.92">13459200000</SceneMarker>
<SceneMarker Sequence="6" Timecode="0:22:30.92">13509200000</SceneMarker>
<SceneMarker Sequence="7" Timecode="0:22:34.51">13545200000</SceneMarker>
<SceneMarker Sequence="8" Timecode="0:22:50.55">13705600000</SceneMarker>
<SceneMarker Sequence="9" Timecode="0:23:11.16">13911600000</SceneMarker>
<SceneMarker Sequence="10" Timecode="0:23:41.16">14211600000</SceneMarker>
<SceneMarker Sequence="11" Timecode="0:24:16.11">14561200000</SceneMarker>
<SceneMarker Sequence="12" Timecode="0:24:31.16">14711600000</SceneMarker>
<SceneMarker Sequence="13" Timecode="0:25:06.03">15060400000</SceneMarker>
<SceneMarker Sequence="14" Timecode="0:25:20.40">15204000000</SceneMarker>
<SceneMarker Sequence="15" Timecode="0:25:46.16">15461600000</SceneMarker>
<SceneMarker Sequence="16" Timecode="0:26:01.11">15611200000</SceneMarker>
<SceneMarker Sequence="17" Timecode="0:26:11.27">15712800000</SceneMarker>
<SceneMarker Sequence="18" Timecode="0:26:30.92">15909200000</SceneMarker>
<SceneMarker Sequence="19" Timecode="0:27:00.92">16209200000</SceneMarker>
<SceneMarker Sequence="20" Timecode="0:27:16.20">16362000000</SceneMarker>
<SceneMarker Sequence="21" Timecode="0:27:26.16">16461600000</SceneMarker>
<SceneMarker Sequence="22" Timecode="0:27:46.16">16661600000</SceneMarker>
<SceneMarker Sequence="23" Timecode="0:27:48.59">16686000000</SceneMarker>
<SceneMarker Sequence="24" Timecode="0:28:01.03">16810400000</SceneMarker>
<SceneMarker Sequence="25" Timecode="0:28:14.64">16946400000</SceneMarker>
<SceneMarker Sequence="26" Timecode="0:28:16.03">16960400000</SceneMarker>
<SceneMarker Sequence="27" Timecode="0:28:26.03">17060400000</SceneMarker>
<SceneMarker Sequence="28" Timecode="0:28:50.31">17303200000</SceneMarker>
<SceneMarker Sequence="29" Timecode="0:29:00.96">17409600000</SceneMarker>
<SceneMarker Sequence="30" Timecode="0:29:05.87">17458800000</SceneMarker>
<SceneMarker Sequence="31" Timecode="0:29:10.16">17501600000</SceneMarker>
<SceneMarker Sequence="32" Timecode="0:49:17.48">29574800000</SceneMarker>
<SceneMarker Sequence="33" Timecode="0:49:22.75">29627600000</SceneMarker>
<SceneMarker Sequence="34" Timecode="0:49:41.27">29812800000</SceneMarker>
<SceneMarker Sequence="35" Timecode="0:49:44.51">29845200000</SceneMarker>
<SceneMarker Sequence="36" Timecode="0:49:49.51">29895200000</SceneMarker>
<SceneMarker Sequence="37" Timecode="0:50:19.51">30195200000</SceneMarker>
<SceneMarker Sequence="38" Timecode="0:50:35.48">30354800000</SceneMarker>
<SceneMarker Sequence="39" Timecode="0:50:39.51">30395200000</SceneMarker>
<SceneMarker Sequence="40" Timecode="0:50:50.15">30501600000</SceneMarker>
<SceneMarker Sequence="41" Timecode="0:50:52.07">30520800000</SceneMarker>
<SceneMarker Sequence="42" Timecode="0:51:04.51">30645200000</SceneMarker>
<SceneMarker Sequence="43" Timecode="0:51:14.51">30745200000</SceneMarker>
<SceneMarker Sequence="44" Timecode="0:51:23.11">30831200000</SceneMarker>
<SceneMarker Sequence="45" Timecode="0:51:24.51">30845200000</SceneMarker>
<SceneMarker Sequence="46" Timecode="0:51:54.75">31147600000</SceneMarker>
<SceneMarker Sequence="47" Timecode="0:52:04.92">31249200000</SceneMarker>
<SceneMarker Sequence="48" Timecode="0:52:09.63">31296400000</SceneMarker>
<SceneMarker Sequence="49" Timecode="0:52:26.11">31461200000</SceneMarker>
<SceneMarker Sequence="50" Timecode="0:52:30.03">31500400000</SceneMarker>
<SceneMarker Sequence="51" Timecode="0:52:39.07">31590800000</SceneMarker>
<SceneMarker Sequence="52" Timecode="0:52:43.11">31631200000</SceneMarker>
<SceneMarker Sequence="53" Timecode="0:52:59.15">31791600000</SceneMarker>
<SceneMarker Sequence="54" Timecode="0:53:19.67">31996800000</SceneMarker>
<SceneMarker Sequence="55" Timecode="0:53:49.59">32296000000</SceneMarker>
<SceneMarker Sequence="56" Timecode="0:54:09.63">32496400000</SceneMarker>
<SceneMarker Sequence="57" Timecode="0:54:39.51">32795200000</SceneMarker>
<SceneMarker Sequence="58" Timecode="0:54:57.03">32970400000</SceneMarker>
<SceneMarker Sequence="59" Timecode="0:55:24.51">33245200000</SceneMarker>
<SceneMarker Sequence="60" Timecode="0:55:59.55">33595600000</SceneMarker>
<SceneMarker Sequence="61" Timecode="0:56:14.48">33744800000</SceneMarker>
<SceneMarker Sequence="62" Timecode="0:56:33.51">33935200000</SceneMarker>
<SceneMarker Sequence="63" Timecode="0:56:45.75">34057600000</SceneMarker>
<SceneMarker Sequence="64" Timecode="1:16:04.39">45644000000</SceneMarker>
<SceneMarker Sequence="65" Timecode="1:16:08.51">45685200000</SceneMarker>
<SceneMarker Sequence="66" Timecode="1:16:26.15">45861600000</SceneMarker>
<SceneMarker Sequence="67" Timecode="1:16:29.39">45894000000</SceneMarker>
<SceneMarker Sequence="68" Timecode="1:16:34.39">45944000000</SceneMarker>
<SceneMarker Sequence="69" Timecode="1:17:04.39">46244000000</SceneMarker>
<SceneMarker Sequence="70" Timecode="1:17:19.39">46394000000</SceneMarker>
<SceneMarker Sequence="71" Timecode="1:17:28.96">46489600000</SceneMarker>
<SceneMarker Sequence="72" Timecode="1:17:34.39">46544000000</SceneMarker>
<SceneMarker Sequence="73" Timecode="1:17:38.00">46580000000</SceneMarker>
<SceneMarker Sequence="74" Timecode="1:17:54.03">46740400000</SceneMarker>
<SceneMarker Sequence="75" Timecode="1:18:14.39">46944000000</SceneMarker>
<SceneMarker Sequence="76" Timecode="1:18:24.39">47044000000</SceneMarker>
<SceneMarker Sequence="77" Timecode="1:18:39.39">47194000000</SceneMarker>
<SceneMarker Sequence="78" Timecode="1:18:54.39">47344000000</SceneMarker>
<SceneMarker Sequence="79" Timecode="1:19:23.15">47631600000</SceneMarker>
<SceneMarker Sequence="80" Timecode="1:19:39.47">47794800000</SceneMarker>
<SceneMarker Sequence="81" Timecode="1:20:09.15">48091600000</SceneMarker>
<SceneMarker Sequence="82" Timecode="1:20:29.51">48295200000</SceneMarker>
<SceneMarker Sequence="83" Timecode="1:20:49.39">48494000000</SceneMarker>
<SceneMarker Sequence="84" Timecode="1:21:09.80">48698000000</SceneMarker>
<SceneMarker Sequence="85" Timecode="1:21:49.64">49096400000</SceneMarker>
<SceneMarker Sequence="86" Timecode="1:21:59.64">49196400000</SceneMarker>
<SceneMarker Sequence="87" Timecode="1:22:09.27">49292800000</SceneMarker>
<SceneMarker Sequence="88" Timecode="1:22:49.43">49694400000</SceneMarker>
<SceneMarker Sequence="89" Timecode="1:23:04.31">49843200000</SceneMarker>
<SceneMarker Sequence="90" Timecode="1:23:19.43">49994400000</SceneMarker>
<SceneMarker Sequence="91" Timecode="1:23:44.47">50244800000</SceneMarker>
<SceneMarker Sequence="92" Timecode="1:23:48.64">50286400000</SceneMarker>
<SceneMarker Sequence="93" Timecode="1:23:58.39">50384000000</SceneMarker>
<SceneMarker Sequence="94" Timecode="1:44:29.96">62699600000</SceneMarker>
<SceneMarker Sequence="95" Timecode="1:45:35.80">63358000000</SceneMarker>
<SceneMarker Sequence="96" Timecode="1:46:00.64">63606400000</SceneMarker>
<SceneMarker Sequence="97" Timecode="1:47:55.55">64755600000</SceneMarker>
</SceneList>
</VideoReDoProject>
 

Dan203

Senior Developer
Staff member
XML containing unicode characters requires a special header. It's the only way the parser knows to treat every character as 2 bytes. We support both the BOM character or the UTF8 or UTF16 encoding types declared in the XML header like so...

Code:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
 

SnoopyDog

Member
Thanks Dan. This is hardcoded in comskip. I will ask erik if he can add it. Just to make sure the vprj format hasnt changed between V5 and V6, right..
 

SnoopyDog

Member
Wow, the Unicodification is totally awesome :D Although it took way too long, now this software is actually usable for us non native english speakers. Hope it will payoff for you...you sold one to me anyways;) Thanks for the great work guys.
 

Danr

Administrator
Staff member
No. v5 and v6 use the same project format.
Just be clear. V5 assumed ASCII characters which range in value from 0-255. V6 VPRJ typically use UTF-8, although UTF16 should be supported as well (see more below). V5 also had an error as it wrote a UTF-8 declaration at the front, but actually ignored it and treated all characters as ASCII. What this means is that a V5 VPRJ file that had ASCII characters > 127 will now be treated as UTF-8 by V6 and it won't be able to open the file.

Regarding UTF1-16, when we write an XML file in V6 we always write it as UTF-8, which means that the support we have for UTF-16 XML receives much less testing.
 

dlflannery

Moderator
This got me wondering about the COM objects called via vbscript by VAP6. In V6, what encoding is recognized for the file names? When I try to process a file name with the stars (as per the first post here) my COM calls fail. "Normal" file names without special chars process fine. Arguments are passed to the VBScripts in the default C# Unicode UTF-16 string encoding.
 

SnoopyDog

Member
Edit!!!
Nevermind my problem. My misstake trying to envoke vp.vbs from a UTF-8 no BOM encoded script. Converting it to UTF-8 with BOM the star file passes alright to v6 with no problem.

dlflannery I get the same result as you trying to pass the star file to V6 with script through vp.vbs. Fails to open while it works fine with non Unicode file names.

CMD.exe says
? Unable to open file/project: F:\TV\Recording Service\20180601_20-59-02_Kanal 5_Gravity - ★★★★★★★★☆☆.VPrj
I copied the
Code:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
on top in the project file in post 1 here.

AHK call used
Code:
RunWait, %comspec% /k cscript.exe //nologo "C:\Program Files (x86)\VideoReDoTVSuite6\vp.vbs" "F:\TV\Recording Service\20180601_20-59-02_Kanal 5_Gravity - ★★★★★★★★☆☆.VPrj" "F:\Edited Recordings\20180601_20-59-02_Kanal 5_Gravity - ★★★★★★★★☆☆.ts" "/p H.264 Transport Stream" /q, ,
 
Last edited:

Danr

Administrator
Staff member
I'm not sure if you can use unicode characters from the command line in cmd.exe. Not sure, but Powershell might work better. Try creating a .vbs script with the unicode file name embedded in the script and see if that works.
 
Last edited:

SnoopyDog

Member
Yes CMD.exe accept unicode chars but I forgot about the importance of the Byte Order Mark (BOM). Invoking it with no BOM encoded script you get the strange chars in the stdout quote above. If using the BOM it looks like this:
Code:
Output complete to: F:\Edited Recordings\20180601_20-59-02_Kanal 5_Gravity - ★★★★★★★★☆☆.ts
 

Dan203

Senior Developer
Staff member
This got me wondering about the COM objects called via vbscript by VAP6. In V6, what encoding is recognized for the file names? When I try to process a file name with the stars (as per the first post here) my COM calls fail. "Normal" file names without special chars process fine. Arguments are passed to the VBScripts in the default C# Unicode UTF-16 string encoding.
Do you get an error when you open? Save? Somewhere else in the middle?
 

dlflannery

Moderator
For testing, I call a .vbs file that VAP6 uses to determine the encoding of a video file using V6 COM, calling it from a .bat file. i get the error message that file <so-and-so> was not found, and the <so-and-so> has mangled char sequences in place of. the star chars.

In actual VAP6 use, the .vbs file is run via a process object and the file name is passed to it as part of an argument string (C# unicode). This works fine for file names that don’t include weird chars like the star. So apparently in the process of going from a C# unicode string into the .vbs script then being passed to the COM function, the file name doesn’t have the special-character encoding needed for the COM function.
 

tobyW

Member
Wow, the Unicodification is totally awesome :D Although it took way too long, now this software is actually usable for us non native english speakers. Hope it will payoff for you...you sold one to me anyways;) Thanks for the great work guys.
Congratulations to the VRD team for making the decision and for carrying it through. I've been round that one myself: it's not a trivial undertaking.
 

SnoopyDog

Member
Try creating a .vbs script with the unicode file name embedded in the script and see if that works.
No no no, I wont go there cuz I wanna spare myself a world of trouble ;) I abandoned VBScript because of poor Unicode handling over a decade ago. Evidently thats what dlflannery is fighting with.
 
Top Bottom