File detected as Windows-1250, but is UTF-8

Question

File detected as Windows-1250, but is UTF-8

tobbi opened this issue 4 years ago · comments

Tobias Markus commented 4 years ago

I'm using UTF.Unknown 2.3.0
The following file is detected as Windows-1250, but is UTF-8:

csv_test_correct_GZ.zip

Rustam · Answer 1 · Fri Feb 21 2020 00:26:35 GMT+0800 (China Standard Time)

Hello, @tobbi !

Thank you for the report.

Could you add a text file? Why did you choose zip? Do you submit this to input?

Tobias Markus · Answer 2 · Fri Feb 21 2020 01:31:28 GMT+0800 (China Standard Time)

Sorry, my bad, it used to be a csv file and github wouldn't accept those. Here's the file with the extension changed to .txt:

csv_test_correct_GZ.txt

Rustam · Answer 3 · Fri Feb 21 2020 02:14:43 GMT+0800 (China Standard Time)

Thanks for clarifying.

At first glance, I think the result is normal. Why? The algorithm by which detected is statistical, and, accordingly, the more different input data, the more accurate the final result. Details can be found in the "A composite approach to language/encoding detection" article.

But, we need to try to improve the result :)

Status Logs:

SBCS: Detected windows-1250 with confidence of 0.7738685

Get confidence:
-- new match found: confidence 0.01, index 0, charset windows-1251.
-- new match found: confidence 0.18598664, index 6, charset iso-8859-7.
-- new match found: confidence 0.7133932, index 15, charset iso-8859-1.
-- new match found: confidence 0.71340704, index 18, charset iso-8859-1.
-- new match found: confidence 0.76677626, index 23, charset iso-8859-1.
-- new match found: confidence 0.7738685, index 86, charset windows-1250.
Get confidence done.
SBCS Group Prober --------begin status
SBCS 0.01: [windows-1251]
SBCS: 0.01 [windows-1251]

SBCS 0.01: [koi8-r]
SBCS: 0.01 [koi8-r]

SBCS 0: [iso-8859-5]
SBCS: 0.00 [iso-8859-5]

SBCS 0.01: [x-mac-cyrillic]
SBCS: 0.01 [x-mac-cyrillic]

SBCS 0.01: [ibm866]
SBCS: 0.01 [ibm866]

SBCS 0.01: [ibm855]
SBCS: 0.01 [ibm855]

SBCS 0.18598664: [iso-8859-7]
SBCS: 0.1859866 [iso-8859-7]

SBCS 0.18598664: [windows-1253]
SBCS: 0.1859866 [windows-1253]

SBCS 0: [iso-8859-5]
SBCS: 0.00 [iso-8859-5]

SBCS 0.01: [windows-1251]
SBCS: 0.01 [windows-1251]

SBCS 0: [windows-1255]
HEB: 0 - 0 [Logical-Visual score]

SBCS 0: [windows-1255]
SBCS: 0.00 [windows-1255]

SBCS 0.09991017: [tis-620]
SBCS: 0.09991017 [tis-620]

SBCS 0.09991017: [iso-8859-11]
SBCS: 0.09991017 [iso-8859-11]

SBCS 0.7133932: [iso-8859-1]
SBCS: 0.7133932 [iso-8859-1]

SBCS 0.6674997: [iso-8859-15]
SBCS: 0.6674997 [iso-8859-15]

SBCS 0.7133932: [windows-1252]
SBCS: 0.7133932 [windows-1252]

SBCS 0.71340704: [iso-8859-1]
SBCS: 0.713407 [iso-8859-1]

SBCS 0.67082536: [iso-8859-15]
SBCS: 0.6708254 [iso-8859-15]

SBCS 0.71340704: [windows-1252]
SBCS: 0.713407 [windows-1252]

SBCS 0.6861101: [iso-8859-2]
SBCS: 0.6861101 [iso-8859-2]

SBCS 0.6861101: [windows-1250]
SBCS: 0.6861101 [windows-1250]

SBCS 0.76677626: [iso-8859-1]
SBCS: 0.7667763 [iso-8859-1]

SBCS 0.76677626: [windows-1252]
SBCS: 0.7667763 [windows-1252]

SBCS inactive: [iso-8859-3] (i.e. confidence is too low).
SBCS inactive: [iso-8859-3] (i.e. confidence is too low).
SBCS 0.717128: [iso-8859-9]
SBCS: 0.717128 [iso-8859-9]

SBCS inactive: [iso-8859-6] (i.e. confidence is too low).
SBCS 0: [windows-1256]
SBCS: 0.00 [windows-1256]

SBCS 0.40016073: [viscii]
SBCS: 0.4001607 [viscii]

SBCS 0.44124976: [windows-1258]
SBCS: 0.4412498 [windows-1258]

SBCS 0.71854687: [iso-8859-15]
SBCS: 0.7185469 [iso-8859-15]

SBCS 0.7641578: [iso-8859-1]
SBCS: 0.7641578 [iso-8859-1]

SBCS 0.7641578: [windows-1252]
SBCS: 0.7641578 [windows-1252]

SBCS 0.71640146: [iso-8859-13]
SBCS: 0.7164015 [iso-8859-13]

SBCS 0.6377162: [iso-8859-10]
SBCS: 0.6377162 [iso-8859-10]

SBCS 0.6736411: [iso-8859-4]
SBCS: 0.6736411 [iso-8859-4]

SBCS 0.71818155: [iso-8859-13]
SBCS: 0.7181816 [iso-8859-13]

SBCS 0.6363546: [iso-8859-10]
SBCS: 0.6363546 [iso-8859-10]

SBCS 0.6753149: [iso-8859-4]
SBCS: 0.6753149 [iso-8859-4]

SBCS 0.666065: [iso-8859-1]
SBCS: 0.666065 [iso-8859-1]

SBCS 0.666065: [iso-8859-9]
SBCS: 0.666065 [iso-8859-9]

SBCS 0.62630904: [iso-8859-15]
SBCS: 0.626309 [iso-8859-15]

SBCS 0.666065: [windows-1252]
SBCS: 0.666065 [windows-1252]

SBCS inactive: [iso-8859-3] (i.e. confidence is too low).
SBCS 0.6366351: [windows-1250]
SBCS: 0.6366351 [windows-1250]

SBCS 0.6366351: [iso-8859-2]
SBCS: 0.6366351 [iso-8859-2]

SBCS 0.72143143: [x-mac-ce]
SBCS: 0.7214314 [x-mac-ce]

SBCS 0.72143143: [ibm852]
SBCS: 0.7214314 [ibm852]

SBCS 0.6434225: [windows-1250]
SBCS: 0.6434225 [windows-1250]

SBCS 0.64008415: [iso-8859-2]
SBCS: 0.6400841 [iso-8859-2]

SBCS 0.7291228: [x-mac-ce]
SBCS: 0.7291228 [x-mac-ce]

SBCS 0.7253399: [ibm852]
SBCS: 0.7253399 [ibm852]

SBCS 0.58494663: [windows-1250]
SBCS: 0.5849466 [windows-1250]

SBCS 0.5881849: [iso-8859-2]
SBCS: 0.5881849 [iso-8859-2]

SBCS 0.61615247: [iso-8859-13]
SBCS: 0.6161525 [iso-8859-13]

SBCS 0.58494663: [iso-8859-16]
SBCS: 0.5849466 [iso-8859-16]

SBCS 0.66285837: [x-mac-ce]
SBCS: 0.6628584 [x-mac-ce]

SBCS 0.65958494: [ibm852]
SBCS: 0.6595849 [ibm852]

SBCS 0.7628341: [iso-8859-1]
SBCS: 0.7628341 [iso-8859-1]

SBCS 0.71730226: [iso-8859-4]
SBCS: 0.7173023 [iso-8859-4]

SBCS 0.71730226: [iso-8859-9]
SBCS: 0.7173023 [iso-8859-9]

SBCS 0.7628341: [iso-8859-13]
SBCS: 0.7628341 [iso-8859-13]

SBCS 0.71730226: [iso-8859-15]
SBCS: 0.7173023 [iso-8859-15]

SBCS 0.7628341: [windows-1252]
SBCS: 0.7628341 [windows-1252]

SBCS 0.76252055: [iso-8859-1]
SBCS: 0.7625206 [iso-8859-1]

SBCS inactive: [iso-8859-3] (i.e. confidence is too low).
SBCS 0.76252055: [iso-8859-9]
SBCS: 0.7625206 [iso-8859-9]

SBCS 0.71700746: [iso-8859-15]
SBCS: 0.7170075 [iso-8859-15]

SBCS 0.76252055: [windows-1252]
SBCS: 0.7625206 [windows-1252]

SBCS 0.6695262: [windows-1250]
SBCS: 0.6695262 [windows-1250]

SBCS 0.6695262: [iso-8859-2]
SBCS: 0.6695262 [iso-8859-2]

SBCS 0.7052443: [iso-8859-13]
SBCS: 0.7052443 [iso-8859-13]

SBCS 0.6695262: [iso-8859-16]
SBCS: 0.6695262 [iso-8859-16]

SBCS 0.7587035: [x-mac-ce]
SBCS: 0.7587035 [x-mac-ce]

SBCS 0.7587035: [ibm852]
SBCS: 0.7587035 [ibm852]

SBCS 0.76380235: [windows-1252]
SBCS: 0.7638023 [windows-1252]

SBCS 0.76380235: [windows-1257]
SBCS: 0.7638023 [windows-1257]

SBCS 0.71821266: [iso-8859-4]
SBCS: 0.7182127 [iso-8859-4]

SBCS 0.76380235: [iso-8859-13]
SBCS: 0.7638023 [iso-8859-13]

SBCS 0.71821266: [iso-8859-15]
SBCS: 0.7182127 [iso-8859-15]

SBCS 0.6575037: [iso-8859-1]
SBCS: 0.6575037 [iso-8859-1]

SBCS 0.6575037: [iso-8859-9]
SBCS: 0.6575037 [iso-8859-9]

SBCS 0.61825883: [iso-8859-15]
SBCS: 0.6182588 [iso-8859-15]

SBCS 0.6575037: [windows-1252]
SBCS: 0.6575037 [windows-1252]

SBCS 0.7738685: [windows-1250]
SBCS: 0.7738685 [windows-1250]

SBCS 0.7738685: [iso-8859-2]
SBCS: 0.7738685 [iso-8859-2]

SBCS 0.7738685: [iso-8859-16]
SBCS: 0.7738685 [iso-8859-16]

SBCS 0.75962406: [ibm852]
SBCS: 0.7596241 [ibm852]

SBCS 0.66994256: [windows-1250]
SBCS: 0.6699426 [windows-1250]

SBCS 0.66994256: [iso-8859-2]
SBCS: 0.6699426 [iso-8859-2]

SBCS 0.66994256: [iso-8859-16]
SBCS: 0.6699426 [iso-8859-16]

SBCS 0.75917524: [x-mac-ce]
SBCS: 0.7591752 [x-mac-ce]

SBCS 0.75917524: [ibm852]
SBCS: 0.7591752 [ibm852]

SBCS 0.76376295: [iso-8859-1]
SBCS: 0.763763 [iso-8859-1]

SBCS 0.7181756: [iso-8859-4]
SBCS: 0.7181756 [iso-8859-4]

SBCS 0.76376295: [iso-8859-9]
SBCS: 0.763763 [iso-8859-9]

SBCS 0.7181756: [iso-8859-15]
SBCS: 0.7181756 [iso-8859-15]

SBCS 0.76376295: [windows-1252]
SBCS: 0.763763 [windows-1252]

SBCS Group found best match [windows-1250] confidence 0.7738685.

MBCS: Detected utf-8 with confidence of 0.7525

Get confidence:
-- new match found: confidence 0.7525, index 0, charset utf-8.
Get confidence done.
MBCS Group Prober --------begin status
MBCS 0.7525: [utf-8]

MBCS 0.01: [shift-jis]

MBCS 0.01: [euc-jp]

MBCS 0.01: [gb18030]

MBCS 0.01: [euc-kr]

MBCS 0.01: [cp949]

MBCS 0.01: [big5]

MBCS inactive: euc-tw (i.e. confidence is too low).
MBCS Group found best match [utf-8] confidence 0.7525.

Latin1Prober: Detected windows-1252 with confidence of 0.43269232

Latin1Prober: 0.43269232 [windows-1252]