[BUG] ValueError: Unknown encoding cl100k_base. Plugins found: []

[kyle-kw]

Please read this first before report any issue
https://pyarmor.readthedocs.io/en/latest/questions.html

If this issue has been documented and there is solution in FAQ, it will be marked as documented and be closed it immediately.

Do not ask questions here but in discussions, it will be closed immediately.

A good report should have

• A clear title
• Reproduced steps
• Expected results and actual results

Please also provide necessary log (but not full log), for example, the whole command options pyarmor gen and first 4 logs in the console, paste the text directly, DO NOT paste IMAGE

---

test.py content:

import tiktoken

def get_token_num(txt: str):
    encoding = tiktoken.get_encoding('cl100k_base')
    token = encoding.encode(txt)
    return len(token)

print(get_token_num('hello world'))
# output : 2

error output:

$ pyarmor gen --pack onefile test.py
INFO     Python 3.9.19
INFO     Pyarmor 8.5.8 (group), 006279, jfh
INFO     Platform linux.x86_64

$ ./dist/test
Traceback (most recent call last):
  File "<frozen __main__>", line 3, in <module>
  File "<frozen test>", line 14, in <module>
  File "<frozen test>", line 8, in get_token_num
  File "tiktoken/registry.py", line 68, in get_encoding
ValueError: Unknown encoding cl100k_base. Plugins found: []
[345813] Failed to execute script 'test' due to unhandled exception!

How to add custom parameters for pyinstaller?
--hidden-import=tiktoken_ext.openai_public --hidden-import=tiktoken_ext

reference: openai/tiktoken#80

[Jondy]

Please check online documentation inside into pack command