Clearing out .pyc files and __pycache__ directories on package removal

[wwuck]

I looked at Issue #45 where it's mentioned why .pyc files are bundled in with the .deb package. I ran into a case where I was installing an updated version of a .deb package generated with dh-virtualenv that had removed some of the python package requirements that existed in the previous .deb package. This generated some dpkg warnings when installing the new .deb file over the old package version.

dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/requests/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/requests': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/idna/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/idna': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/utils/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/utils': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/transforms/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/transforms': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/schemas/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/schemas': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/model/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/model': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/grid/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/grid': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/estimators/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/estimators': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/backend/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/backend': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o': Directory not empty

Is it possible for dh-virtualenv to generate a prerm script that will clear out these old files and directories?

It looks like dh_python2 and dh_python3 use the prerm-pyclean and prerm-py3clean autoscripts to do this. Could this be done with dh-virtualenv too? Is there any way to add these manually into the debian/ directory, or does this have to be activated from within dh-virtualenv?

/usr/share/debhelper/autoscripts/prerm-pyclean
/usr/share/debhelper/autoscripts/prerm-py3clean

Also, I notice there are autoscripts for postinst-pycompile and postinst-py3compile, so those could also be used instead of bundling the .pyc files in the .deb?

/usr/share/debhelper/autoscripts/postinst-pycompile
/usr/share/debhelper/autoscripts/postinst-py3compile

[Marcel Wiegand]

I had a similar problem. My package couldn't be removed completely because of existing .pyc files and __pycache__ directories.

I temporarily solved it by adding py3clean . to override_dh_virtualenv and providing a prerm script to call py3clean . before the package is removed.

[Jürgen Hermann]

For reference:

$ dpkg -S py3clean
python3-minimal: /usr/bin/py3clean
dh-python: /usr/share/debhelper/autoscripts/prerm-py3clean
python3-minimal: /usr/share/man/man1/py3clean.1.gz
$ cat /usr/share/debhelper/autoscripts/prerm-py3clean
if which py3clean >/dev/null 2>&1; then
	py3clean -p #PACKAGE# #ARGS#
else
	dpkg -L #PACKAGE# | perl -ne 's,/([^/]*)\.py$,/__pycache__/\1.*, or next; unlink $_ or die $! foreach glob($_)'
	find /usr/lib/python3/dist-packages/ -type d -name __pycache__ -empty -print0 | xargs --null --no-run-if-empty rmdir
fi

[wwuck]

Don't forget there is also a prerm-pypyclean and a postinst-pypycompile, if dh-virtualenv supports pypy.

[Jyrki Pulliainen]

Hi!

Thanks for the bug report. Feels definitely something we should work on. I wonder what is the general opinion on how to handle this: We could default on some py*clean step to the build chain, but unsure if we could target the right python version?

In any case it should be overridable

[Jürgen Hermann]

It is not a build thing, but a maintainer script thing. As to versions, the helper scripts have to be called explicitly with the venv binary, and the venv path – from debhelper snippets.

[wwuck]

I attempted to add the following to a prerm script as a quick way to get this functionality, but I'm not sure how I embed the value of $DH_VIRTUALENV_INSTALL_ROOT into the prerm script.

${DH_VIRTUALENV_INSTALL_ROOT}/<package-name>/bin/python3 /usr/bin/py3clean ${DH_VIRTUALENV_INSTALL_ROOT}/<package-name>

Any suggestions?

EDIT: I guess I could just hardcode the value manually as a quick fix.

[Jürgen Hermann]

you place your code under the #debhelper# thing.