Giters

JacobDev1 / xl-converter

Easy-to-use image converter for modern formats. Supports multithreading, drag 'n drop, and downscaling.

Home Page:https://codepoems.eu/xl-converter

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

On Windows, doesn't support Unicode paths (at least for JXL ouput)

Ofunniku opened this issue · comments

commented

When trying to tell it to process files in a path or a filename that has Japanese characters (like numbered files that have "cover" at the start of the name but in a directory called "劇場版") it just ends almost instantly and processes nothing but when I rename the folder or files to have nothing but ANSI characters and it works as expected.

I haven't tried the other formats but this happens with JXL for sure.

commented

Hello,

thank you for using my application.

I have been able to reproduce the issue on Windows. The Linux version seems unaffected.

I could not immediately find a solution, but I will be working more on this problem in the future.

~The Developer

commented

cjxl and avifenc encoders do not support these characters on Windows. src

There doesn't seem to be anything I can do about it.

If you need this to work, use the Linux version instead.

~The Developer

commented

Update: libavif added UTF-8 support.

Next version of XL Converter will support UTF-8 for AVIF.

As for JPEG XL, we will have to wait until libjxl devs add it.

commented

As a temporary solution, have your XL Converter create a temporary folder while running, for example "C:\JXLtemp", which 100% will not cause problems, copy the files there, then convert and move the new files to the original folder. I think it’s not too difficult and you will no longer be hostage to the restrictions of any third-party codecs.

commented

That's exactly how it's laid out in my TODO list. However, there is a reason why I did not implement this yet.

Here are the locations I considered

  • %USERPROFILE%\AppData\Local\Temp - this may not work if the user account has UTF-16-specific characters in it
  • C:\temp - needs admin privileges

I cannot use admin privileges, because Windows would make drag 'n drop stop working (not a bug, just the way Windows works).

The current plan is:

  1. Try to use the user-based tmp location
  2. If that fails, prompt the user to choose a different one

There would have to be a wall of text explaining everything. I'm worried about users getting confused.

I'm working on another feature at the moment. However, this workaround is important, so I will get to it eventually.

commented

Hmm, then here's an idea for you: in the Windows console there is a "subst" command that allows you to temporarily create a virtual disk that will actually point to a folder. Only there are some restrictions that I don't understand, please read the description of this command.

Or you can use one of the already existing folders, for example "ProgramData", "Users", creating "C:\ProgramData\JXLtemp" there (I don't know for sure if Windows will allow access there). I asked ChatGPT and he assured that there would be no problem using "C:\Users\Public\Temp". I don't know if this is true.

The converter may also ask the user to manually create this folder. There's nothing complicated about it and you don't have to worry about people getting confused.

I also wanted to ask you, does your program provide for monitoring the success of the conversion? Moreover, this control must be quite reliable and based on several factors:

  1. Codec report.
  2. Availability of the converted file on disk.
  3. It is also advisable to provide an additional option that would use a special tool that would verify the file. This may sound a little paranoid, but I have encountered similar situations more than once.

P.S. Sorry for my bad English, I have to use Google Translate.

commented

Hmm, then here's an idea for you: in the Windows console there is a "subst" command that allows you to temporarily create a virtual disk that will actually point to a folder.

I am not planning on using subst because this may come off as shady.

I asked ChatGPT and he assured that there would be no problem using "C:\Users\Public\Temp". I don't know if this is true.

The C:/Users/Public location only exists in the English installation of Windows. It gets localized and often features non-ANSI symbols.

I also wanted to ask you, does your program provide for monitoring the success of the conversion? Moreover, this control must be quite reliable and based on several factors:

Yes, XL Converter does check If the conversion was successful. The error reporting is quite thorough.

  1. It is also advisable to provide an additional option that would use a special tool that would verify the file. This may sound a little paranoid, but I have encountered similar situations more than once.

Verifying images (outside of checksums) is not possible to do reliably.

commented

You are partly wrong about "C:\Users\Public". As far as I know, Windows simultaneously uses a dual name: the main one is saved, and the localized version is displayed in Explorer. A similar situation occurs with the “C:\Program Files” and “C:\Program Files (x86)” folders. Just think: if the original name were not retained, many programs would stop working altogether, that would be too reckless on Microsoft's part. That is, you can safely ignore localizations and use only the standard names of all these folders.

commented

You appear to be correct. This is only how Windows Explorer presents it. I could perhaps use this folder, although it is a public folder... I will look into it.

As for system locations, programmers use environmental variables (like %PUBLIC%), which then are translated by the OS.

commented

Please read this information, perhaps this will be the solution to the problem - libjxl/libjxl#683 (comment)

commented

Please read this information, perhaps this will be the solution to the problem - libjxl/libjxl#683 (comment)

The user can set this globally for the whole system with the "Beta: Use Unicode UTF-8 for worldwide language support" setting in the Control Panel.

I tested this method and it is unacceptable, because then in localized programs the font is replaced with strange characters. In addition, I remembered that several years ago I already tried to use it, with a similar result.

commented

mes with Japanese or Chinese characters.

That's exactly how it's laid out in my TODO list. However, there is a reason why I did not implement this yet.

Here are the locations I considered

  • %USERPROFILE%\AppData\Local\Temp - this may not work if the user account has UTF-16-specific characters in it
  • C:\temp - needs admin privileges

I cannot use admin privileges, because Windows would make drag 'n drop stop working (not a bug, just the way Windows works).

The current plan is:

  1. Try to use the user-based tmp location
  2. If that fails, prompt the user to choose a different one

There would have to be a wall of text explaining everything. I'm worried about users getting confused.

I'm working on another feature at the moment. However, this workaround is important, so I will get to it eventually.

I've tried many other JXL format conversion software. They can support file and folder names with Japanese or Chinese characters.They might have employed a similar approach.

commented

They can support file and folder names with Japanese or Chinese characters

@orbit666 Can you list these programs? I have also tried many (Windows only) and none work properly.

@JacobDev1 Is there any progress in solving this problem?

commented

If you want this to work, go ask the libjxl developers about it. UTF-8 support is broken in all libjxl tools (cjpegli, cjxl, djxl, jxlinfo etc.)

I saw some progress on the original libjxl issue, so I decided to work on other aspects of the program for a while.

I have prototyped a few solutions to this. I got it mostly working, but I kept running into edge cases. I got a very bad feeling about having to maintain this code and worried about the potential bugs.

This should really be fixed at the encoder level.

commented

Of course, I asked them about this, it was six months ago, several people discussed it a little and there was no practical result. Then I started searching on the internet about this problem and saw how other people had contacted them many times asking them to fix the bug; this started almost from the moment the format was created. As you can see, nothing has changed. So I think it makes no sense to hope that one day they will release a working version of the encoder. I can only hope that one of the creators of converter programs will fix their bug.

commented

Update: I have built libjxl with the fix suggested in the libjxl #683 issue. It works. Full UTF-8 support.

xl_converter_utf8

However, it is an unofficial hotfix and I did get some errors during build, so I'm unsure how stable it really is.

I'll be tinkering with it some more. I'm hoping it will get included in the next libjxl release.

commented

Can you give me this version? I'll try to test it - mainly on Japanese characters.

commented

Do not use these binaries for anything important - only testing. This is not a fix.

  1. Download the binaries
  2. Unpack them with 7zip
  3. Copy all files to xl-converter/_internal/bin/win, overwrite when prompted
  4. Launch XL Converter, go to "Settings" and enable: "libjxl - Disable UTF-8 Check"

Be sure to test it thoroughly and tell me If something breaks. Anyone who's reading this should feel invited to test it and share their feedback.

Feel free to copy the xl-converter folder and work on that instead. The application is self-contained.

Here is the patch file for programmers.

Running the patched cjxl with Wine works, but also displays this error: 0094:err:rpc:I_RpcReceive we got fault packet with status 0x1c010003.

commented

So, I tested several folders with files in all possible combinations where Japanese characters were contained in the file name and/or in the folder name. At first glance, everything went well.

The only strange thing is that the PhotoQt viewer cannot open files encoded with "Modular". I first thought that the reason was the file/folder name, but no. I checked the "standard" version of your converter, same thing. Other viewers that I use - ImageGlass, IfranView - open files perfectly, regardless of the "Modular" encoding or the presence of these characters.

Thank you.

commented

If you find broken images, try to investigate the official JPEG XL tools (cjxl, djxl and so on). XL Converter uses them to process your images.

If something goes wrong during encoding, I can't fix it. I'm not a C developer. I'm just relying on the official implementation.

When you're using JPEG XL expect bugs like this to occur. It is a new and still developing format.

I checked and PhotoQt indeed doesn't open them. Strangely enough, when modular images are encoded on Linux, they do open fine.

Image viewers may not be able to open every JPEG XL image. For example, some images encoded with libjxl 0.10.* would fail to open. Most likely a metadata issue.

This GitHub issue is related to UTF-8 support. If you want to discuss something unrelated, feel free to open another GitHub issue.

commented

This experimental UTF-8 patch does not affect generated images, which is good. The encoder seems stable as far as I've tested.

What I find peculiar is how using cjxl from the same libjxl version generates images with different checksums depending on the build environment.

Checksums differ:

Checksums match:

This is purely a technical difference because images look and work identically.

commented

Update: Another way to enable UTF-8 support is to do it system-wide. It works with all versions of XL Converter.

Disclaimer: This might cause older programs to break.

  1. Go to Control Panel -> Change date, time or number formats -> Administrative -> Change system locale...
  2. Check Beta: Use Unicode UTF-8 for worldwide language support
  3. Go to XL Converter -> Settings
  4. Check libjxl - Disable UTF-8 Check
  5. Restart Windows.

UTF-8 should be supported now.

commented

XL Converter 1.0 will support UTF-8 for JPEG XL and JPEGLI out of the box.

I just published this patch for libjxl. You can use it to enable UTF-8 support.

  1. Download the 7z file
  2. Unpack it with 7zip
  3. Copy all files to xl-converter/_internal/bin/win, overwrite when prompted
  4. Launch XL Converter, go to "Settings" and enable: "libjxl - Disable UTF-8 Check"

If you already swapped the binaries with the previous ones, you can keep using them.

commented

They can support file and folder names with Japanese or Chinese characters

@orbit666 Can you list these programs? I have also tried many (Windows only) and none work properly.

@JacobDev1 Is there any progress in solving this problem?

https://gitee.com/Insestars/jxlgui-qt

commented

@orbit666

XnConvert supports JPEG XL If you want an alternative. Apart from that, IrfanView has a plugin for JPEG XL. However, these implementations are not fully featured.

The program you listed is all in Chinese and hasn't been updated in 9 months... Feel free to suggest software, but make sure it offers an English version.

If you really need UTF-8 support, there is a way.

commented

@orbit666

XnConvert supports JPEG XL If you want an alternative. Apart from that, IrfanView has a plugin for JPEG XL. However, these implementations are not fully featured.

The program you listed is all in Chinese and hasn't been updated in 9 months... Feel free to suggest software, but make sure it offers an English version.

If you really need UTF-8 support, there is a way.

Thank you so much