certbot / certbot

This is a continuation of #7741 and #6872. The issue has been recently re-raised here: https://community.letsencrypt.org/t/issue-using-certbot-on-cis-ubuntu-image.

My operating system is (include version):

Ubuntu 20.04 with the cis_level1_server USG profile. (I think the important thing here is the default umask is 0027).

I installed Certbot with (snap, OS package manager, pip, certbot-auto, etc):

v1.30.0 via snap

I ran this command and it produced this output:

rm -rf /var/lib/letsencrypt /etc/letsencrypt/; 
certbot register -n --staging --register-unsafely-without-email --agree-tos;
stat /etc/letsencrypt/

Certbot's behavior differed from what I expected because:

Directory should be created with mode 0755 but ends up 0750:

# stat /etc/letsencrypt/
File: /etc/letsencrypt/
Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: fc01h/64513d    Inode: 792023      Links: 4
Access: (0750/drwxr-x---)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-10-04 11:59:18.525300619 +1100
Modify: 2022-10-04 11:59:19.685313875 +1100
Change: 2022-10-04 11:59:19.685313875 +1100
Birth: -

In turn, this causes problem for the --apache plugin as seen in the linked thread:

[Mon Oct 03 23:06:43.166813 2022] [core:error] [pid 1532] (13)Permission denied: [client 3.135.17.41:27500] AH00035: access to /.well-known/acme-challenge/EePksl7BbSJpmKH7jobt4u1aXc3n_wOPd8hXvV14Ux8 denied (filesystem path '/var/lib/letsencrypt/http_challenges') because search permissions are missing on a component of the path

certbot.compat.filesystem.makedirs is supposed to deal with umask to make sure the modes are right, but it appears to be ineffective in this case.

Thanks for writing all of this up.

Do you think looking into this is something we should place in the current milestone or just tag as "priority: significant" for us to (hopefully) look into in the future?

I think this would be good to fix sooner rather than later, let's milestone it: https://community.letsencrypt.org/search?q=%22because%20search%20permissions%20are%20missing%20on%20a%20component%20of%20the%20path%22

this appears to be due to how we calculate our temporary umask in makedirs:

certbot/certbot/certbot/compat/filesystem.py

Line 278 in e3448fa

umask(current_umask | 0o777 ^ mode)

this works fine for permissive umasks like 022, but for 027 like in CIS Ubuntu the more strict bits are or'd into our new umask.

i think the solution here is just to create a umask from our desired mode (e.g. 777 - mode) rather than basing it on the old umask, does that sound right?

Makes sense to me.

Looking at the code, changing this function globally would affect quite a few directories. I wonder if it impolite to ignore the user's requested umask? I am not sure what approach other UNIX software takes.

i couldn't find any clear guidelines on proper umask usage, although i did notice examples in git source code of setting it to w/e was desired. python's makdirs docs also suggest changing the umask if we want to affect intermediate dirs.

interestingly, we also do this umask dance in the apache http challenge setup, in code written to address this same bug a few years ago:

certbot/certbot-apache/certbot_apache/_internal/http_01.py

Lines 187 to 195 in e3448fa

    
           old_umask = filesystem.umask(0o022) 
        
           try: 
        
               filesystem.makedirs(self.challenge_dir, 0o755) 
        
           except OSError as exception: 
        
               if exception.errno not in (errno.EEXIST, errno.EISDIR): 
        
                   raise errors.PluginError( 
        
                       "Couldn't create root for http-01 challenge") 
        
           finally: 
        
               filesystem.umask(old_umask)

so i guess we either need to do the umask dance in all uses of makedirs, or change the function to do it/ensure the specified mode is correctly applied?

one alternative approach could be to set a known-working umask default like 022, if we don't want all intermediate dirs to have the same mode as our target dir

@wgreenberg, do you have a link to "examples in git source code of setting it to w/e was desired" handy?

If not, no worries, but I'd be interested in seeing it as I'm starting to think more about this problem.

@wgreenberg, do you have a link to "examples in git source code of setting it to w/e was desired" handy?

If not, no worries, but I'd be interested in seeing it as I'm starting to think more about this problem.

the main example i found was here, where git's setting up the permissions for a tar archive:

code: https://github.com/git/git/blob/36f8e7ed7d72d2ac73743c3c2226cceb29b32156/archive-tar.c#L414
docs for this option: https://git-scm.com/docs/git-archive#Documentation/git-archive.txt-tarumask

in other instances, i saw a lot of "create dir, then chmod", which we could also do

My thoughts here are that we shouldn't modify filesystem.makedirs. The filesystems module was written largely as a replacement for Python's os module but with better Windows support for Certbot's purposes. (We even toyed with the idea of monkey patching stdlib os with the filesystem module at runtime for a while.) In the spirit of this, I personally think filesystem.makedirs does the Right Thing™ in that it behaves like os.makedirs on UNIX systems (in the way that it did in Python < 3.7 as described here which we supported when this function was written).

I also think that in general we should try to respect the user's umask. Poking around myself a bit, I also struggled to find many suggestions about how to handle the umask, but most of what I saw either respected the user's umask or allowed the user to configure the effective umask through an application specific setting.

Personally what I think we should do is to ignore the user's umask when creating Certbot's "working directory" (/var/lib/letsencrypt by default), but otherwise continue to respect the user's umask. I'd probably just do this here with a try/finally block like is done in the Apache plugin. EDIT2: I'd probably also add a comment describing why we're doing this.

This umask problem may pop up for us yet again somewhere else with this conservative approach, but I'm not sure where that'd be. I thought about the possibility of /etc/letsencrypt where servers need to be able to access certs and private keys, but we create directories on all the relevant paths there with extremely restrictive permissions anyway.

What do people think?

EDIT: For what it's worth, the core idea here is essentially the same as Alex's idea at #9448 (comment).

@bmw sounds good to me. from what i can tell, util.make_or_verify_dir (https://github.com/certbot/certbot/blob/master/certbot/certbot/util.py#L189-L213) is what does the heavy lifting to create the working dir, though it's also used elsewhere (e.g. creating the hooks/config dirs). given the assurances it makes about creating a dir w/ the proper permissions, and the fact that it errors out if given a dir with incorrect ones, do you think it's fair to ignore the user's umask when its called?

I personally do not and think we should treat the working dir as a special case by handling potential umask problems outside of make_or_verify_dir because I think that function only checks permissions when its strict parameter is True and its usually not due to the default value of the parameter and relevant configuration option.

Reworked the PR to just change the umask for the working dir, lmk what you think!

	old_umask = filesystem.umask(0o022)
	try:
	filesystem.makedirs(self.challenge_dir, 0o755)
	except OSError as exception:
	if exception.errno not in (errno.EEXIST, errno.EISDIR):
	raise errors.PluginError(
	"Couldn't create root for http-01 challenge")
	finally:
	filesystem.umask(old_umask)

Creation of /var/lib/letsencrypt (and other core directories) is affected by umask

My operating system is (include version):

I installed Certbot with (snap, OS package manager, pip, certbot-auto, etc):

I ran this command and it produced this output:

Certbot's behavior differed from what I expected because: