charm build process fails for python setuptools

[Taihsiang Ho]

Checklist

• Confirmed this is an issue with charm-tools, not charmstore-client
• Provide versions of tools used
• Described the feature or ways to replicate the issue

What version am I running?

I ran the following command: snap info charm and got the following ouput:

name:      charm
summary:   charm and charm-tools
publisher: Canonical✓
store-url: https://snapcraft.io/charm
contact:   https://discourse.juju.is/c/charming
license:   unset
description: |
  charmstore-client and charm-tools
commands:
  - charm
snap-id:      2Rryoc2ylScfbFl4eQtpntHD9iuZuMvt
tracking:     latest/stable
refresh-date: 2020-07-23
channels:
  latest/stable:    2.7.8               2020-07-21 (512) 117MB classic
  latest/candidate: 2.7.8               2020-07-21 (512) 117MB classic
  latest/beta:      2.7.8               2020-07-21 (512) 117MB classic
  latest/edge:      2.7.8+git-5-4a31de7 2020-09-05 (544) 116MB classic
installed:          2.7.8                          (512) 117MB classic

I am using: Ubuntu Mate (bionic)

Issue/Feature

charm build fails for python setuptools. For example, build the jenkins charm will fail by

1. git clone https://github.com/jenkinsci/jenkins-charm.git
2. cd jenkins-charm
3. charm build ./

I expect/expected the following

The charm build process will complete successfully.

What I got

jenkins-charm$ charm build ./
build: Build dir not specified via command-line or environment; defaulting to /tmp/charm-builds
build: Destination charm directory: /tmp/charm-builds/jenkins
build: Processing layer: layer:options
build: Processing layer: layer:basic
build: Processing layer: layer:status
build: Processing layer: layer:apt
build: Processing layer: jenkins (from .)
build: Processing interface: http
build: Processing interface: jenkins-slave
build: Processing interface: jenkins-extension
build: Processing interface: zuul
build: Processing interface: nrpe-external-master
build: Collecting pip<19.0,>=18.1
  Using cached pip-18.1.tar.gz (1.3 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Saved /tmp/tmppt4mqwa8/pip-18.1.tar.gz
Collecting Jinja2<=2.10.1
  Using cached Jinja2-2.10.1.tar.gz (260 kB)
  Saved /tmp/tmppt4mqwa8/Jinja2-2.10.1.tar.gz
Collecting PyYAML<=5.2
  Using cached PyYAML-5.2.tar.gz (265 kB)
  Saved /tmp/tmppt4mqwa8/PyYAML-5.2.tar.gz
Collecting setuptools<42
  Using cached setuptools-41.6.0.zip (852 kB)
    ERROR: Command errored out with exit status 1:
     command: /tmp/tmpz0l5203u/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-download-4vf4q03b/setuptools/setup.py'"'"'; __file__='"'"'/tmp/pip-download-4vf4q03b/setuptools/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'
"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-bvfvt8a6
         cwd: /tmp/pip-download-4vf4q03b/setuptools/
    Complete output (9 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-download-4vf4q03b/setuptools/setuptools/__init__.py", line 6, in <module>
        import distutils.core
      File "/tmp/tmpz0l5203u/lib/python3.6/site-packages/_distutils_hack/__init__.py", line 83, in create_module
        return importlib.import_module('setuptools._distutils')
      File "/snap/charm/512/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
    ModuleNotFoundError: No module named 'setuptools._distutils'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: You are using pip version 20.1.1; however, version 20.2.3 is available.
You should consider upgrading via the '/tmp/tmpz0l5203u/bin/python -m pip install --upgrade pip' command.

[Taihsiang Ho]

This issue shows up today. I ran the same command yesterday and did not get the same error. According to this setuptools issue, in short:

1. a workaround is export SETUPTOOLS_USE_DISTUTILS=stdlib (I have tested for both of snap and pypi version of charm-tools. This workaround will work for both of them.)
2. the latest setuptools has included the above workaround pypa/setuptools#2376
3. the root cause is in pip and people are addressing it