Giters

deepset-ai / FARM

:house_with_garden: Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.

Home Page:https://farm.deepset.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

current process gets forked, after parallelism has already been used

EddyDavies opened this issue · comments

commented

Describe the bug
When using the DataSilo function following this article the second cell under Data Processor produce this error code.
Error message

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

Expected behavior
No error message

To Reproduce
Run the Jupyter Notebook from the article

System:

  • OS: Mac 11.5
  • GPU/CPU: 2 GHz Quad-Core Intel Core i5
  • FARM version: 0.8.0
commented

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 21 days if no further activity occurs.