Giters

packt-cli / Packt-Publishing-Free-Learning

Scripts that automatically claim and download free daily eBooks from https://www.packtpub.com/packt/offers/free-learning

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Unable to retrieve or download books

StinThiago opened this issue · comments

commented

Hi, I'm unable to download books again. Particularly speaking I think that packet has changed the way that is requested follow my errors.

image

image

commented

Looks like you get further than I do:

python packtPublishingFreeEbook.py -da
[INFO] - Creating session...
[INFO] - Session created, logged in successfully!
[INFO] - Getting data of all your books...
[INFO] - Opened 'https://www.packtpub.com/account/my-ebooks' successfully!
[ERROR] - Exception occurred 'NoneType' object has no attribute 'find_all'
commented

It's because Packtpub has changed how their website works. They not only have changed the layout, but now they are also using JSON API with JWT (which seems to be an advantage as we can receive JSON from their API instead of parsing HTML as we did).

The different error output from your posts is because you use different script options.

I'm working on fixing the script so that it will work with new Packtpub website (so far I've got logging in and parsing products list), but I'm not sure when it'll be ready (should be before Christmas, but I cannot say whether it'll be today, tomorrow or on Monday).

commented

I can't even find the free book of the day page. It doesn't seem available yet in the new website.

commented

Thanks. Sounds like the new approach will be even better.

commented

Downloading books should already work on my setup branch (see last commit this pull request). The code is still ugly and requires adding error handling.

For further fixes we'll have to wait till Pack website will be fully operational.

commented

I can't even find the free book of the day page. It doesn't seem available yet in the new website.

@AghilasMessara A few times per year Packt are temporarily disabling free books of the day, especially around Christmas.

As quoted from their page:

"FREE LEARNING IS TAKING A SHORT BREAK - CHECK OUT OUR BRILLIANT $5 SALE BESTSELLERS INSTEAD:"

commented

Hi

How to fix this issue Command "git checkout -q setup" failed with error code 1 in /tmp/pip-req-build-FDw4322

Thank you

(venv) root@server:/var/www/books# git --version
git version 2.7.4

(venv) root@server:/var/www/books# python --version
Python 2.7.12

(venv) root@server:/var/www/# pip install git+https://github.com/igbt6/Packt-Publishing-Free-Learning.git@setup
Collecting git+https://github.com/igbt6/Packt-Publishing-Free-Learning.git@setup
  Cloning https://github.com/igbt6/Packt-Publishing-Free-Learning.git (to revision setup) to /tmp/pip-req-build-JVo6P1
  Did not find branch or tag 'setup', assuming revision or ref.
error: pathspec 'setup' did not match any file(s) known to git.
Command "git checkout -q setup" failed with error code 1 in /tmp/pip-req-build-FDw4322

(venv) root@server:/var/www/books#
commented

@develsites Such branch does not exist in this repository, you have to either clone my fork or add new remote to use setup branch.

Also note that this repository owner has changed his username, I'm not sure if old repository URL will work.

commented

It seems Packt have changed all it's website structure again, and the script can't download the e-books anymore. I even noticed the e-book page url has changed from "/my-ebooks" to "/products". I think it requires a lot of efforts for this script to work again :(

commented

Let's wait till they finish an update of the page, After that we can start making some adaptation changes of the script

commented

Hi,
Monday (8th Jan) all my books vanished form the Products page, but today they are back on the https://account.packtpub.com/account/products page.

Not sure if they have finished the redesign or not.

commented

The offer seems to be over. I've updated the pull request, seems to be working, but I'll need to test it one more time tomorrow or this week.

Of course you can also verify if that works for you, the more of you will do it, the smaller the chance that something will be broken.

Linters are failing on #111, I'll fix them tomorrow.

commented

Hello
The offer seems to be back.
I tried your code (master branch), with no success. Do you have a better result ?
Thanks for your reply, regards
Laurent

commented

Laurent: Did you mean @mjenczmyk code or Luk6xff's code? Luk6xff's hasnt had an update yet, I was going to try and @mjenczmyk master tommorow if I get time.

commented

It's my setup branch, it's in #111 pull request. I keep my master branch the same as @luk6xff master. Make sure you've read README before you try it, there are major changes - I can help if something is not clear.

commented

Tried swapping out the main script with no luck, maybe missing some dependancy, I get the error
2019-01-31 07:17:01,888 - main - [ERROR] - Exception occurred argument of type 'NoneType' is not iterable
I have it on Windows at the moment, I intend to move it to a Ubuntu VM that's always on I was just being lazy and using my main PC first, Will see if re-installing it from your fork works better sometime over the weekend.

commented

Did you install it (either into your Python installation or better into virtualenv) with pip install git+https://github.com/mjenczmyk/Packt-Publishing-Free-Learning.git@setup. Did you run the script the traditional way or with packt-cli command?

commented

Did you install it (either into your Python installation or better into virtualenv) with pip install git+https://github.com/mjenczmyk/Packt-Publishing-Free-Learning.git@setup.

I get about 20 minutes in the morning over breakfast as I check email. I simply cloned your script over the top of the Luk6xff one and it failed.

I'll build a fresh VM and try this weekend.

commented

OK, I'll write once again how to test my changes (it's described partially in README.md on setup branch and in #111 pull request).

  1. Create new virtualenv for testing those changes (this step is optional, but you don't want to pollute your global Python installation packages), enter this new virtualenv.
  2. Install script by running pip install git+https://github.com/mjenczmyk/Packt-Publishing-Free-Learning.git@setup - this will change after merge to pip install git+https://github.com/luk6xff/Packt-Publishing-Free-Learning.git@master. You don't need to clone anything!
  3. Run packt-cli -g -c path/to/your/config/file.cfg - that should grab Free Learning book for the current day. Similarly packt-cli -da -c path/to/your/config/file.cfg should download all books from your account.

Any questions or doubts? I haven't check it on Windows (I have access only to Linux machine), but it should work the same there.

commented

built a fresh Ubuntu 18 minimal install VM, installed Pip and Git and ran the setup. Looks like its failing calling console_scripts on line 11. This may be as its a new build something is missing

commented

@mjenczmyk : read your README more accurately.. You are right, I was using the old way to install the package.
But are you sure about : pip install git+https://github.com/luk6xff/Packt-Publishing-Free-Learning.git@master
or shall we use : pip install git+https://github.com/**mjenczmyk**/Packt-Publishing-Free-Learning.git@**setup**
Thanks in advance, regards
Laurent

commented

It's git+https://github.com/mjenczmyk/Packt-Publishing-Free-Learning.git@setup, it'll change to git+https://github.com/luk6xff/Packt-Publishing-Free-Learning.git@master after merge.

In the future it would be nice to upload this packege to PyPi, so one could run just pip install packt-cli==x.y.z, where x.y.x is the version of the script.

commented

@laurentlemercier I don't think the offer is really back yet. I get the 24h timer but there is no book to download.

commented

@AghilasMessara Its back on I've claimed books for the last three days, I does not work on IE for me but I tend to use my Phone to claim them on my commune.

commented

Hello,
It works like a charm using python3, thanks a lot @mjenczmyk ! Using python2, it throws an error : "from urllib.parse import urljoin ImportError: No module named parse"

For those it may concern, this my "recipe" for installing this version, using virtualenvwrapper (pip install virtualenvwrapper + edit of the ~/.profile )

The new directory just contains the config files. The "old" packtPublishingFreeEbook.py can be found here:

  • ls $VIRTUAL_ENV/lib/python*/site-packages/packtPublishingFreeEbook.py

and to run the script; run :

  • packt-cli -sgd -sm

Regards

Laurent

commented

I’m using Python27. I’ll shift to Python 3 and try that.

commented

I've started with theVM but I get the "No module named parse" with my Python3 Install. I think the python 2.7 is still hanging around. Going to wipe the VM and start again...

Given up on Windows, I'm a window person but getting python and PIP to work seem beyond me....

commented

I've started with theVM but I get the "No module named parse" with my Python3 Install. I think the python 2.7 is still hanging around. Going to wipe the VM and start again...

Given up on Windows, I'm a window person but getting python and PIP to work seem beyond me....

What OS do you have in your VM? I have literally just tried in on Windows 10 and then deployed it to Linux, both work fine. The trick is to use virtualenv/virtualenvwrapper/virtualenvwrapper-win and create a env using mkvirtualenv <new_env> -p to make sure you are using the python 3. Then when you do pip install setup.zip or pip install git+xxxxxxx, it should automatically install all the packages.

commented

Should have noted this morning that in the text below I refer to a VM. I wiped the "old" VM I mentioned above and created a new one this morning (5th Feb). Installed Ubuntu 18.04 then allowed it to get all updates.

Its a freshly built Ubuntu 18.04. Part of the issue may be its newly built, in that something people install on each new box when they build it and they then forget installing it may be missing.,

Cant do it all this morning (not enough time) But so far - I've got my Fresh Ubuntu 18.04 box.

Python is 3.6.7.

Used sudo apt install python3-pip to bring in pip.
Pip is reporting as Pip (9.0.1) from /usr/lib/pythong3/disk-packages (Python 3.6)

Used Sudo -apt-get install virtualenv to bring in virtualenv

however mkvirtualenv and mkproject both gives a command not found. To be honest as this is on a VM and I'll only use if to grab the days book and download it I don't think I really need virtualenv, but I'm following the steps laid out.

Times up off to work :)

commented

I would like to try to merge it anywhere soon. Let's decide what's left to be done.

  1. Can we agree it works on Linux on Python 3 (both running manually and cron setup on Debian related distributions)?
  2. Shall I fix this so the script will work on Python 2? We want to drop support for Python2 soon anyway.
  3. Have we checked the script on Windows? @sigino says it works on Windows, did you try scheduled tasks setup also? Could you check it and propose a fix if that won't work as described in README.md (I don't have access to Windows mashine). Another question is if we need setup for scheduled tasks anyway.
  4. Do we need a BAT script if the script is a installable script anyway?
  5. @gjchester Are you using virtualenv? If so what which python returns?
commented

Should have noted this morning that in the text below I refer to a VM. I wiped the "old" VM I mentioned above and created a new one this morning (5th Feb). Installed Ubuntu 18.04 then allowed it to get all updates.

Its a freshly built Ubuntu 18.04. Part of the issue may be its newly built, in that something people install on each new box when they build it and they then forget installing it may be missing.,

Cant do it all this morning (not enough time) But so far - I've got my Fresh Ubuntu 18.04 box.

Python is 3.6.7.

Used sudo apt install python3-pip to bring in pip.
Pip is reporting as Pip (9.0.1) from /usr/lib/pythong3/disk-packages (Python 3.6)

Used Sudo -apt-get install virtualenv to bring in virtualenv

however mkvirtualenv and mkproject both gives a command not found. To be honest as this is on a VM and I'll only use if to grab the days book and download it I don't think I really need virtualenv, but I'm following the steps laid out.

Times up off to work :)

Do the following :)
Copy your configFile.cfg into your folder ~. Then

sudo apt install python3 python3-pip virtualenv virtualenvwrapper
source /usr/local/bin/virtualenvwrapper.sh
mkvirtualenv --python=/usr/bin/python3 packt-books-env
workon packt-books-env
cd ~
wget https://github.com/mjenczmyk/Packt-Publishing-Free-Learning/archive/setup.zip
pip3 install setup.zip
packt-cli -g
commented

I would like to try to merge it anywhere soon. Let's decide what's left to be done.

  1. Can we agree it works on Linux on Python 3 (both running manually and cron setup on Debian related distributions)?

Yes it works for me at least.

  1. Shall I fix this so the script will work on Python 2? We want to drop support for Python2 soon anyway.

I wouldn't think so. It is a maintenance burden to continue supporting both 2 and 3 when everyone should be switching to Python 3 IMHO.

  1. Have we checked the script on Windows? @sigino says it works on Windows, did you try scheduled tasks setup also? Could you check it and propose a fix if that won't work as described in README.md (I don't have access to Windows mashine). Another question is if we need setup for scheduled tasks anyway.

On Windows, I have only tried to launch it manually and it works. But, I am also wondering if it is installed into a virtual env, it does not mention in the README or the script that the env will need to be activated (I think). Personally speaking, I think the details of how to schedule the task can be left for the end users :) It is not really the meat of this project.

  1. Do we need a BAT script if the script is a installable script anyway?

Don't really think so.

  1. @gjchester Are you using virtualenv? If so what which python returns?
commented

Some of it is working now on Ubuntu 18 under Python3 thanks to Sigino. My virtualenvwrapper.sh was in a different place but that was an easy fix..

Books being claimed and downloaded OK, Mails not sending but I can probably fix that easy enough (probably password or email account setup details) and I want to get auto upload to google in place but its one step at a time...

I'd suggest Python 2 is dropped too, its almost retired and part of my setup issues may have been I had parts under Python 2.7

I didnt reset it up on Windows, I've one windows machine and as long as it works in one place thats all I need, having it on a ubuntu VM means it can run all the time, there no licence issue and I can forget about it.

commented

@luk6xff Are we ready to go? Can I merge the pull request as master is broken and this code seems to fix this?

commented

Still cannot get the Email working, how is it sending, POP3 /IMAP etc? nor I cant get the Google docs upload working either (complains about a folder error, but it created the PACKT_BOOKS folder OK when it asked for permissions) , but then I realised I have about 630 books on my account and not enough Google space anyway...

I've got a chunk downloaded but then I got a Invalid URL message.

[ERROR] - Invalid URL 'None': No schema supplied. Perhaps you meant http://None?

Anyone seeing this be aware its a timeout on the Captcha or the PACKT logon, Rerun the file and it will continue on, skipping the files you already have downloaded (assume you have not deleted them).

Next step Automate it with Cron...

I think the line I need is (I know, I'm not a Linux person)

0 12 * * * cd /home/packt-download/Packt_books/ && /usr/bin/python3 packt-cli -gd -f> /home/packt-download/packtPublishingFreeEbook.log 2>&1

As my Books live in there own folders under home /packt-download/Packt_Books/

commented

Still cannot get the Email working, how is it sending, POP3 /IMAP etc? nor I cant get the Google docs upload working either (complains about a folder error, but it created the PACKT_BOOKS folder OK when it asked for permissions) , but then I realised I have about 630 books on my account and not enough Google space anyway...

I've got a chunk downloaded but then I got a Invalid URL message.

[ERROR] - Invalid URL 'None': No schema supplied. Perhaps you meant http://None?

I have not tried the emailing or google drive uploading, nor have seen the timeout. But I would suggest even they are broken, let's still let the guys to get it merged. If they are broken, very likely they are not broken by this change, and the main purpose of this change is to fix the core functionality - well, that and some :). Leave them as untested features or known issues if needed.

commented

@luk6xff Are we ready to go? Can I merge the pull request as master is broken and this code seems to fix this?

@mjenczmyk yeah, feel free to merge, tested also nhere, looks ok, good job :) !

commented

@gjchester I'm using virtualenvwrapper, and such entry in crontab works for me: 30 2 * * * ~/.virtualenvs/packt/bin/packt-cli -c ~/.packt/config.cfg -g. If you have it in your global Python installation than I suppose you could simplify your entry to 0 12 * * * packt-cli -gd -f -c /home/packt-download/Packt_books/configFile.cfg > /home/packt-download/packtPublishingFreeEbook.log 2>&1

commented

I've got a chunk downloaded but then I got a Invalid URL message.

[ERROR] - Invalid URL 'None': No schema supplied. Perhaps you meant http://None?

OK, I see what's the issue here. It's because JWT expires after an hour, we need to add an API client that would keep JWT as its field and retry making the request if it results in 401 status code. It shouldn't be difficult, I'll try to make a pull request till Friday (later I've got my days off, so I won't be able to do that in the following week). Here is an issue for that. #120

I'll also check once again if Google Drive and email sending setup works - after that we'll be able to close the issue.

commented

@sigino : Absolutely merge them, this fixes the script 95%, The timeout was more a FYI, its only going to be a few people who see the issue due to number of books, or slow connection but I see @mjenczmyk is on it already. After all once they've all downloaded the first time its only an add on of one book, not a monster download again.
I know my ISP will reject email if the sender of the email is NOT the same as the logged in account, hence my asking about what it was using to send emails. Its not vital I get an email saying its done its job just a nice confirmation its working.

@luk6xff and @mjenczmyk BIG Thank You to you both for writing and updating the script, its far beyond my Python skill.

commented

My configuration, if it can help anyone : Python 3.4.2 & virtualenv 16.0.0.0 on raspberry pi3 & raspbian jessie.
It works using commandline or crontab :
5 6 * * * cd /home/pi/pyprojects/PPFL3 && /home/pi/.virtualenvs/PPFL3/bin/packt-cli -sgd -sm >> /home/pi/packt/packtPublishingFreeEbook.log 2>&1
@luk6xff and @mjenczmyk : thanks a lot for your work !

commented

No problem, thank you @laurentlemercier, @gjchester and @sigino for help!

commented

Wow, that's interesting. Even if books obtained by Packt Free Learning offer cannot be downloaded via their website (you can only download code and read them on the website, they offer you to actually buy them if you want to have e-book files) they can be still downloaded using our script via their API 😂

commented

Wow, that's interesting. Even if books obtained by Packt Free Learning offer cannot be downloaded via their website

Only for my latest ones If you go a page they are downloadable as epub/pdf/mobi etc. Or at least are for me....

commented

Directly from the website, I think you can download the old ones and they are still downloadable, but no the ones claimed this year.

commented

I asked about new Free Learning books not being downloadable and their response was:

We understand that you are unable to see the download option for the products that are in your account.
Please accept our apologies for the inconvenience caused to you in this matter.
We would like to inform you that we have made few changes on the Packt website as all the free learning eBooks will only be available to view online/reader.
We would like to inform you that the free learning books will be available in your account for lifetime.
Below are the benefits of the free learning.

The title is still free forever and they can access it whenever they like
They can download it to read offline in the Packt app
The code files are available for download from “My Owned Products”
We hope this helps.

commented

That being said, I just ran the updated script (thanks for everyone who got it fixed) to download all my books, and I got the various formats for my newly claimed Free Learning ebooks. Who knows if they will figure out the issue and block it.

commented

I'll close the issue, JWT expiration is being in #120 issue, if we notice any more problems we can make a new issue to resolve them.

commented

how to download book from packt in simple words please

commented
commented

i am new to github community. which script are we taking about.............
please help me from start

commented
commented
commented
commented

NOTE: Packt changed their website at the start of the year, to make it so you will only be able to download books you collected BEFORE the start of 2019, any after that can only be viewed online, not downloaded.

Actually we can download even those books that were claimed in 2019, as we are using API directly.

commented

I can confirm this is working on Windows 10 using Python 3.7.

commented