FileNotFoundError during restore

[jakegatsby]

Summary: An unhandled FileNotFoundError occurs in wal-e during PostgreSQL startup following backup-fetch and writing recovery.conf.

Environment: Debian Stretch, PostgreSQL 9.6, Python 3.5, wal-e 1.1.0

Steps to Reproduce:

• install postgresql
• modify postgresql.conf as per instructions
• start postgres, perform multiple backup-push operations (filesystem based in this case)
• stop postgres, remove $PGDATA directory
• perform backup-fetch $PGDATA LATEST
• add recovery.conf to $PGDATA directory as per wal-e instructions
• start postgres

Results:

wal_e.operator.backup INFO     MSG: begin wal restore
        STRUCTURED: time=2018-06-23T12:30:16.762775-00 pid=672 action=wal-fetch key=file://localhost/backups/wal_005/000000010000000000000015.lzo prefix=backups/ seg=000000010000000000000015 state=begin
lzop: <stdin>: not a lzop file
wal_e.main   CRITICAL MSG: An unprocessed exception has avoided all error handling
        DETAIL: Traceback (most recent call last):
          File "/usr/local/lib/python3.5/dist-packages/wal_e/cmd.py", line 657, in main
            args.prefetch)
          File "/usr/local/lib/python3.5/dist-packages/wal_e/operator/backup.py", line 353, in wal_restore
            self.gpg_key_id is not None)
          File "/usr/local/lib/python3.5/dist-packages/wal_e/worker/worker_util.py", line 58, in do_lzop_get
            return blobstore.do_lzop_get(creds, url, path, decrypt, do_retry=do_retry)
          File "/usr/local/lib/python3.5/dist-packages/wal_e/blobstore/file/file_util.py", line 52, in do_lzop_get
            raise exc
          File "/usr/local/lib/python3.5/dist-packages/wal_e/blobstore/file/file_util.py", line 64, in write_and_return_error
            key.get_contents_to_file(stream)
          File "/usr/local/lib/python3.5/dist-packages/wal_e/blobstore/file/calling_format.py", line 53, in get_contents_to_file
            with open(self.path, 'rb') as f:
        FileNotFoundError: [Errno 2] No such file or directory: '/backups/wal_005/000000010000000000000015.lzo'

[James Howe]

This is sort of expected, as not finding a WAL segment is how postgres knows it's finished.
However, this should not log a CRITICAL uncaught exception.

Even worse is that the prefetch constantly spams these CRITICAL errors into syslog.

CRITICAL MSG: An unprocessed exception has avoided all error handling 
DETAIL: Traceback (most recent call last):
  File "/opt/wal-e/lib/python3.6/site-packages/wal_e/cmd.py", line 662, in main
    backup_cxt.wal_prefetch(args.BASE_DIRECTORY, args.SEGMENT)
  File "/opt/wal-e/lib/python3.6/site-packages/wal_e/operator/backup.py", line 382, in wal_prefetch
    self.gpg_key_id is not None, do_retry=False)
  File "/opt/wal-e/lib/python3.6/site-packages/wal_e/worker/worker_util.py", line 58, in
    do_lzop_get return blobstore.do_lzop_get(creds, url, path, decrypt, do_retry=do_retry)
  File "/opt/wal-e/lib/python3.6/site-packages/wal_e/blobstore/file/file_util.py", line 52, in
    do_lzop_get raise exc
  File "/opt/wal-e/lib/python3.6/site-packages/wal_e/blobstore/file/file_util.py", line 64, in write_and_return_error
    key.get_contents_to_file(stream)
  File "/opt/wal-e/lib/python3.6/site-packages/wal_e/blobstore/file/calling_format.py", line 53, in get_contents_to_file
    with open(self.path, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/pg_backup/wal_005/000000010000000F00000010.lzo'