(some?) utf8 encoded characters are transformed into latin1

Question

(some?) utf8 encoded characters are transformed into latin1

rjoberon opened this issue 7 years ago · comments

In combination with pdflatex and \usepackage[T1]{fontenc} utf-8 characters in notes are converted to latin1 which is a problem for pdfpc.

With this input (test.tex):

\documentclass{beamer}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[ngerman]{babel}
\usepackage{../includes/pdfpcnotes}

\begin{document}
\begin{frame}
  schön
  \pnote{- schön}
\end{frame}
\end{document}

the following output (test.pdfpc) is produced:

[file]
test.pdf
[notes]
### 1
- sch\366n

Note that I have inserted the \366 here manually as my browser converted it to the letter "ö" - hexdump -C test.pdfpc shows that this is latin1:

00000000  5b 66 69 6c 65 5d 0a 74  65 73 74 2e 70 64 66 0a  |[file].test.pdf.|
00000010  5b 6e 6f 74 65 73 5d 0a  23 23 23 20 31 0a 2d 20  |[notes].### 1.- |
00000020  73 63 68 f6 6e 0a                                 |sch.n.|
00000026

With this file pdfpc does not show any notes.

After recode latin1..utf8 test.pdfpc this is indeed utf-8:

00000000  5b 66 69 6c 65 5d 0a 74  65 73 74 2e 70 64 66 0a  |[file].test.pdf.|
00000010  5b 6e 6f 74 65 73 5d 0a  23 23 23 20 31 0a 2d 20  |[notes].### 1.- |
00000020  73 63 68 c3 b6 6e 0a                              |sch..n.|
00000027

And pdfpc correctly shows "schön" as a note.

Note that \usepackage[T1]{fontenc} has an influence on that. Removing that line from test.tex, results in the following test.pdfpc:

[file]
test.pdf
[notes]
### 1
- sch\protect \unhbox \voidb@x \bgroup \U@D 1ex{\setbox \z@ \hbox {\char 127}\dimen@ -.45ex\advance \dimen@ \ht \z@ \fontdimen 5\font \dimen@ }\accent 127\fontdimen 5\font \U@D o\egroup n

... which does not work with pdfpc either. ;-)

This happens with the following version of pdflatex:

> pdflatex -version
pdfTeX 3.14159265-2.6-1.40.17 (TeX Live 2016/Debian)
kpathsea version 6.2.2
Copyright 2016 Han The Thanh (pdfTeX) et al.
There is NO warranty.  Redistribution of this software is
covered by the terms of both the pdfTeX copyright and
the Lesser GNU General Public License.
For more information about these matters, see the file
named COPYING and the pdfTeX source.
Primary author of pdfTeX: Han The Thanh (pdfTeX) et al.
Compiled with libpng 1.6.28; using libpng 1.6.28
Compiled with zlib 1.2.8; using zlib 1.2.8
Compiled with poppler version 0.48.0

My current workaround is to (automatically) call recode after each run of pdflatex ... so it's not a big problem.

Carsten Brandt · Answer 1 · Sun Oct 15 2017 04:42:36 GMT+0800 (China Standard Time)

Note that I have inserted the \366 here manually as my browser converted it to the letter "ö" - hexdump -C test.pdfpc shows that this is latin1:

73 63 68 f6 6e 0a - this looks like UTF8 to me:

I tried your example tex file and it works fine in pdfpc for me.

Carsten Brandt · Answer 2 · Sun Oct 15 2017 04:49:09 GMT+0800 (China Standard Time)

Okay, seems my knowledge about encoding is a bit limited, c3b6 is listed as the UTF-8 encoding for ö https://unicode-table.com/en/search/?q=U%2B00f6, still, opening the file in my text editor showed it as "UTF-8 encoded".

Carsten Brandt · Answer 3 · Sun Oct 15 2017 04:52:33 GMT+0800 (China Standard Time)

Here is my version of pdflatex:

$ pdflatex --version
pdfTeX 3.14159265-2.6-1.40.15 (TeX Live 2015/dev/Debian)
kpathsea version 6.2.1dev
Copyright 2014 Peter Breitenlohner (eTeX)/Han The Thanh (pdfTeX).
There is NO warranty.  Redistribution of this software is
covered by the terms of both the pdfTeX copyright and
the Lesser GNU General Public License.
For more information about these matters, see the file
named COPYING and the pdfTeX source.
Primary author of pdfTeX: Peter Breitenlohner (eTeX)/Han The Thanh (pdfTeX).
Compiled with libpng 1.6.13; using libpng 1.6.13
Compiled with zlib 1.2.8; using zlib 1.2.8
Compiled with poppler version 0.26.5

it produces the correct UTF8 encoding:

$ hexdump -C out/test.pdfpc 
00000000  5b 6e 6f 74 65 73 5d 0a  23 23 23 20 31 0a 2d 20  |[notes].### 1.- |
00000010  73 63 68 c3 b6 6e 0a                              |sch..n.|
00000017

Robert Jäschke · Answer 4 · Sun Oct 15 2017 17:54:56 GMT+0800 (China Standard Time)

True, with the latest version this works indeed (and also with the version from October 2013.
This version causes the problem. It seems I got lost in all the different versions I have tried. Thus, the issue remains relevant for pull request #15 only. (The improved overlay handling in Xaf's fork is crucial for me, though.)