Giters

dipu-bd / lightnovel-crawler

Generate and download e-books from online sources.

Home Page:https://pypi.org/project/lightnovel-crawler/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Help cant search Fix this bug

Nephalemzz opened this issue · comments

commented

Describe the bug

Searching: 53%|██████████████████████████████████▌ | 145/273 [01:22<03:05, 1.45s/source]
! Error: decoding to str: need a bytes-like object, NoneType found
<class 'TypeError'>
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\app.py", line 103, in search_novel
search_novels(self)
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\novel_search.py", line 125, in search_novels
app.search_results = _combine_results(results)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\novel_search.py", line 46, in _combine_results
key = slugify(item.title)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\daved\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\slugify\slugify.py", line 104, in slugify
text = _unicode(text, 'utf-8', 'ignore')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

! Error: Cancelled by user

Exception ignored in: <module 'threading' from 'C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\threading.py'>
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 1592, in _shutdown
atexit_call()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\concurrent\futures\thread.py", line 31, in _python_exit
t.join()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 1147, in join
self._wait_for_tstate_lock()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 1167, in wait_for_tstate_lock
if lock.acquire(block, timeout):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core_init.py", line 25, in destroy
sys.exit(1)
SystemExit: 1

Let us know

Does not matter which novel i try and search it gives this error. did not have it on 3.4.2.

App source: EXE | pip install clone
App version: 3.5.1
Your OS: Windows

commented

Tried to downgrade to python 3.11 and error still occurs

                      [#] Lightnovel Crawler v3.5.1
              https://github.com/dipu-bd/lightnovel-crawler

-> Press Ctrl + C to exit

? Enter novel page url or query novel: super god gene
Searching: 34%|██████████████████████▋ | 93/270 [00:11<00:31, 5.60source/s]Failed to get any results, likely auth failure
Searching: 73%|███████████████████████████████████████████████▍ | 197/270 [01:10<00:28, 2.58source/s]
! Error: decoding to str: need a bytes-like object, NoneType found
<class 'TypeError'>
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\app.py", line 103, in search_novel
search_novels(self)
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\novel_search.py", line 125, in search_novels
app.search_results = _combine_results(results)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\novel_search.py", line 46, in _combine_results
key = slugify(item.title)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\daved\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\slugify\slugify.py", line 115, in slugify
text = str(text, 'utf-8', 'ignore')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  • https://github.com/dipu-bd/lightnovel-crawler/issues
    ================================================================================
    Searching: 100%|█████████████████████████████████████████████████████████████████| 270/270 [01:10<00:00, 3.85source/s]
commented

lightnovel-crawl> python lncrawl -s https://boxnovel.com/novel/reincarnation-of-the-strongest-sword-god/

                      [#] Lightnovel Crawler v3.5.1
              https://github.com/dipu-bd/lightnovel-crawler

-> Press Ctrl + C to exit

Retrieving novel info...
Exception in thread Thread-1 (read_novel_info):
Traceback (most recent call last):
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\templates\madara.py", line 66, in select_chapter_tags
raise Exception("No chapters on first URL")
Exception: No chapters on first URL

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 1045, in _bootstrap_inner
self.run()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 982, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\templates\soup\general.py", line 34, in read_novel_info
for item in self.parse_chapter_list(soup):
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\templates\soup\chapter_only.py", line 13, in parse_chapter_list
for tag in self.select_chapter_tags(soup):
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\templates\madara.py", line 70, in select_chapter_tags
response = self.submit_form(
^^^^^^^^^^^^^^^^^
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\scraper.py", line 230, in submit_form
return self.post_response(url, data=data, headers=headers, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\scraper.py", line 211, in post_response
return self.__process_request(
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\scraper.py", line 130, in __process_request
raise e
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\scraper.py", line 123, in __process_request
response.raise_for_status()
File "C:\Users\daved\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\requests\models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://boxnovel.com/wp-admin/admin-ajax.php

! Error: No chapters found
<class 'Exception'>
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\bots\console\integration.py", line 107, in start
raise e
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\bots\console\integration.py", line 101, in start
_download_novel()
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\bots\console\integration.py", line 85, in _download_novel
self.app.get_novel_info()
File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\app.py", line 137, in get_novel_info
raise Exception("No chapters found")

commented

Error: decoding to str: need a bytes-like object, NoneType found This error is fixed. But the error which occurs when you try to download from URL : novel_url. Yeah, I am not sure how to fix this.

commented

Thanks zGadli, Well one down is better then nothing. What is the fix to this or the article with the fix steps.

commented

Just tried to do a serch and still showing the error; decoding to str
-> Press Ctrl + C to exit

? Enter novel page url or query novel: super god gene
Searching: 36%|███████████████████████▍ | 102/283 [00:07<00:17, 10.13source/s]Failed to get any results, likely auth failure
Searching: 73%|███████████████████████████████████████████████▎ | 206/283 [01:09<00:29, 2.60source/s]
! Error: decoding to str: need a bytes-like object, NoneType found
<class 'TypeError'>
File "C:\Users\daved\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\lncrawl\core\app.py", line 103, in search_novel
search_novels(self)
File "C:\Users\daved\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\lncrawl\core\novel_search.py", line 125, in search_novels
app.search_results = _combine_results(results)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\daved\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\lncrawl\core\novel_search.py", line 46, in _combine_results
key = slugify(item.title)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\daved\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\slugify\slugify.py", line 115, in slugify
text = str(text, 'utf-8', 'ignore')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Searching: 100%|█████████████████████████████████████████████████████████████████| 283/283 [01:20<00:00, 2.60source/s]

commented

@Nephalemzz If you want the fix you will have to clone the dev branch of this repo to your local machine and run it.