scrapinghub / shub

Scrapinghub Command Line Client

Home Page:https://shub.readthedocs.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

IndexError on `shub image deploy`

szeswee opened this issue · comments

Hello. I'm getting this error on shub image deploy, where it worked flawlessly before I updated to the latest shub version 2.10.0:

Traceback (most recent call last):
  File "/Users/szeswee/.virtualenvs/scrapers/bin/shub", line 8, in <module>
    sys.exit(cli())
  File "/Users/szeswee/.virtualenvs/scrapers/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/szeswee/.virtualenvs/scrapers/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/szeswee/.virtualenvs/scrapers/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/szeswee/.virtualenvs/scrapers/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/szeswee/.virtualenvs/scrapers/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/szeswee/.virtualenvs/scrapers/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/szeswee/.virtualenvs/scrapers/lib/python3.6/site-packages/shub/image/deploy.py", line 63, in cli
    apikey, insecure, async_)
  File "/Users/szeswee/.virtualenvs/scrapers/lib/python3.6/site-packages/shub/image/deploy.py", line 81, in deploy_cmd
    username, password, email)
  File "/Users/szeswee/.virtualenvs/scrapers/lib/python3.6/site-packages/shub/image/deploy.py", line 200, in _prepare_deploy_params
    metadata['scripts'] = _extract_scripts_from_project()
  File "/Users/szeswee/.virtualenvs/scrapers/lib/python3.6/site-packages/shub/image/deploy.py", line 246, in _extract_scripts_from_project
    _, kwargs = global_vars['__setup_calls__'][0]
IndexError: list index out of range

On inspection, it seems it was unable to get the arguments of the setup call. This is how it looks:

setup(
    name="xxx",
    version="xxx",
    author="szeswee",
    author_email="xxx@gmail.com",
    packages=find_packages(exclude=["tests*"]),
    entry_points={"scrapy": ["settings = scrapers.settings"]},
    python_requires="~=3.6.0",
    install_requires=[
        "awscli==1.16.255",
        "boto3==1.9.91",
        "click==6.7",
        "pytz==2017.2",
        "PyYAML==3.12",
        "s3fs==0.3.5",
        "scrapinghub==2.0.3",
        "Scrapy==1.5.0",
        "scrapy-crawlera==1.3.0",
        "scrapy-dotpersistence==0.3.0",
        "scrapy-magicfields==1.1.0",
        "scrapy-splash==0.7.2",
        "semver==2.7.9",
        "tqdm==4.19.1",
        "watchdog==0.8.3",
    ],
    extras_require={
        "tests": [
            "black==18.9b0",
            "coverage==4.5.1",
            "flake8==3.5.0",
            "ipdb",
            "isort==4.3.4",
            "nose==1.3.7",
            "shub",
            "Sphinx==1.7.2",
        ]
    },
    scripts=[
        "tools/cli/vault",
        "tools/cli/jl_to_csv.py",
        "tools/cli/state-snapshot",
        "tools/cli/notify",
    ],
    include_package_data=True,  # include files found in MANIFEST.in
)

These are the commands used in the Dockerfile:

COPY . /app
WORKDIR /app

# Setup environment variable SCRAPY_SETTINGS_MODULE used by Scrapinghub
ENV SCRAPY_SETTINGS_MODULE scrapers.settings

RUN pip install .[tests]
RUN python setup.py install

I'm using Python 3.6.4 on OSX 10.14.6