cpg command not superceding fcharset command

Question

cpg command not superceding fcharset command

abLoftware opened this issue 5 years ago · comments

Japanese_UTF8.rtf.txt
Japanese_JIS.rtf.txt

It looks like fcharset is not being over-ridden when cpg is also provided,
From RTF Specification version 1.9.1 pg 20:
"If the \cpgN does appear, it supersedes the code page corresponding to the \fcharsetN."

test cases attached
TestBug2.java.txt

Jon Iles · Answer 1 · Fri Mar 20 2020 21:19:57 GMT+0800 (China Standard Time)

Thanks for the bug report. Could you provide the two samples as standalone RTF files which I can open with Microsoft Word or Wordpad? Thanks!

abLoftware · Answer 2 · Fri Mar 20 2020 21:42:31 GMT+0800 (China Standard Time)

Hi Jon, Thanks for taking a look at it! I’ve attached 2 RTF files, one for JIS and one for UTF8 Note that if you do a save after opening it in WordPad then WordPad Will strip out “\fcharset128\cpg65001” and change the hex to Unicode For the UTF8 file From: Jon Iles <notifications@github.com> Sent: Friday, March 20, 2020 9:20 AM To: joniles/rtfparserkit <rtfparserkit@noreply.github.com> Cc: Andre Boutin <ABoutin@loftware.com>; Author <author@noreply.github.com> Subject: Re: [joniles/rtfparserkit] cpg command not superceding fcharset command (#22) Thanks for the bug report. Could you provide the two samples as standalone RTF files which I can open with Microsoft Word or Wordpad? Thanks! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#22 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIEWFCMEYXI42O4XJDGWSGDRINUQXANCNFSM4LOTGMAA>.

Jon Iles · Answer 3 · Sat Mar 21 2020 23:54:09 GMT+0800 (China Standard Time)

H Andre, unfortunately I can't see any attachments - cold you link them directlyto the GitHub issue?

Thanks!

Jon

abLoftware · Answer 4 · Mon Mar 23 2020 20:25:11 GMT+0800 (China Standard Time)

Hi Jon, I have attached them to the issue in GitHub Thanks! Andre From: Jon Iles <notifications@github.com> Sent: Saturday, March 21, 2020 11:54 AM To: joniles/rtfparserkit <rtfparserkit@noreply.github.com> Cc: Andre Boutin <ABoutin@loftware.com>; Author <author@noreply.github.com> Subject: Re: [joniles/rtfparserkit] cpg command not superceding fcharset command (#22) H Andre, unfortunately I can't see any attachments - cold you link them directlyto the GitHub issue? Thanks! Jon — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#22 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIEWFCJCHBJ4RT4ZDRM3JUDRITPK5ANCNFSM4LOTGMAA>.

Jon Iles · Answer 5 · Thu Mar 26 2020 21:51:23 GMT+0800 (China Standard Time)

I've had a chance to take a quick look. Before I make any changes to the code, I wanted to validate what Microsoft products made of the sample RTF files you provided.

This is what Wordpad makes of the JIS file

and here's what Wordpad makes of the UTF8 file

Here's what Word makes of the JIS file

and here's what Word makes of the UTF8

Based on these results I'm inclined to think that the UTF8 version of the file isn't correct as it stands. If we can get to the point with the UTF8 file where it renders consistently when opened in a Microsoft product and uses the cpg command, I can make a stab at getting the parser to work with it appropriately.

abLoftware · Answer 6 · Thu Mar 26 2020 22:07:15 GMT+0800 (China Standard Time)

I am able to get consistent results with both word and wordpad, Being very careful not to make any changes within either since they will re-write how it is it stored Note for each of the images below I had to add spaces so I could move the caret Out of the way so that the caret wouldn’t be in image, so I had to be sure not to save The file when closing it, which would then change the rtf as it originally was written In fact even though I SWEAR I did not save the UTF-8 yet it still seems to have been rewritten on me, and I ended up with something similar to your word with utf-8 Here is each file in wordpad/word being super careful that the file is not modified Wordpad JIS ![image](https://user-images.githubusercontent.com/34169481/77658487-2741a580-6f4d-11ea-8f53-07d24810c03b.png) Word JIS ![image](https://user-images.githubusercontent.com/34169481/77658563-3e809300-6f4d-11ea-8d3e-16046219caea.png) Wordpad UTF-8 ![image](https://user-images.githubusercontent.com/34169481/77658611-50facc80-6f4d-11ea-8f74-20285c36ed0c.png) Word UTF-8 ![image](https://user-images.githubusercontent.com/34169481/77658658-6243d900-6f4d-11ea-9916-f512cb5cf765.png) From: Jon Iles <notifications@github.com> Sent: Thursday, March 26, 2020 9:52 AM To: joniles/rtfparserkit <rtfparserkit@noreply.github.com> Cc: Andre Boutin <ABoutin@loftware.com>; Author <author@noreply.github.com> Subject: Re: [joniles/rtfparserkit] cpg command not superceding fcharset command (#22) I've had a chance to take a quick look. Before I make any changes to the code, I wanted to validate what Microsoft products made of the sample RTF files you provided. This is what Wordpad makes of the JIS file [image]<https://user-images.githubusercontent.com/4912864/77653652-eb680980-6f67-11ea-83ac-408fe5925cfd.png> and here's what Wordpad makes of the UTF8 file [image]<https://user-images.githubusercontent.com/4912864/77653728-0470ba80-6f68-11ea-97e7-9be15e1fb6cd.png> Here's what Word makes of the JIS file [image]<https://user-images.githubusercontent.com/4912864/77653950-48fc5600-6f68-11ea-822e-cea424349129.png> and here's what Word makes of the UTF8 [image]<https://user-images.githubusercontent.com/4912864/77654047-6b8e6f00-6f68-11ea-82b3-591933e8bb48.png> Based on these results I'm inclined to think that the UTF8 version of the file isn't correct as it stands. If we can get to the point with the UTF8 file where it renders consistently when opened in a Microsoft product and uses the cpg command, I can make a stab at getting the parser to work with it appropriately. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#22 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIEWFCN2XNY42ADOD2HH2MLRJNMWVANCNFSM4LOTGMAA>.

Jon Iles · Answer 7 · Thu Mar 26 2020 22:23:24 GMT+0800 (China Standard Time)

Thanks for the reply. Unfortunately emailing responses back to this issue drops any embedded images or files. Can you add the images via the GitHub UI?

abLoftware · Answer 8 · Thu Mar 26 2020 22:36:31 GMT+0800 (China Standard Time)

Should be updated now From: Jon Iles <notifications@github.com> Sent: Thursday, March 26, 2020 10:24 AM To: joniles/rtfparserkit <rtfparserkit@noreply.github.com> Cc: Andre Boutin <ABoutin@loftware.com>; Author <author@noreply.github.com> Subject: Re: [joniles/rtfparserkit] cpg command not superceding fcharset command (#22) Thanks for the reply. Unfortunately emailing responses back to this issue drops any embedded images or files. Can you add the images via the GitHub UI? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#22 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIEWFCPKAT5FW7J3LRACDDLRJNQOXANCNFSM4LOTGMAA>.

Jon Iles · Answer 9 · Mon Mar 30 2020 17:39:00 GMT+0800 (China Standard Time)

Hi! That was interesting, I got different results from Wordpad in Windows 8.1 and Windows 10 . I could see the files both rendering the same with the Windows 10 version. Anyway, I've applied a fix and released a new version - hopefully that'll work for you!

abLoftware · Answer 10 · Mon Mar 30 2020 21:38:14 GMT+0800 (China Standard Time)

Awesome! Thanks for looking into it! Andre From: Jon Iles <notifications@github.com> Sent: Monday, March 30, 2020 5:39 AM To: joniles/rtfparserkit <rtfparserkit@noreply.github.com> Cc: Andre Boutin <ABoutin@loftware.com>; Author <author@noreply.github.com> Subject: Re: [joniles/rtfparserkit] cpg command not superceding fcharset command (#22) Hi! That was interesting, I got different results from Wordpad in Windows 8.1 and Windows 10 . I could see the files both rendering the same with the Windows 10 version. Anyway, I've applied a fix and released a new version - hopefully that'll work for you! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#22 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIEWFCJJH235T6YD2VBXICDRKBSEFANCNFSM4LOTGMAA>.