vericast / conda-mirror

Mirror upstream conda channels

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Validation errors caused by incomplete tar file

dmkent opened this issue · comments

I recently hit an issue where an exception was unhandled when validating a file that
was, for unknown reasons, not a valid tar file. Traceback was:

    Traceback (most recent call last):
      File "/miniconda/envs/conda-mirror/lib/python3.5/site-packages/conda_mirror/conda_mirror.py", line 303, in _validate
        t.extractfile('info/index.json').read().decode('utf-8')
      File "/miniconda/envs/conda-mirror/lib/python3.5/tarfile.py", line 2066, in extractfile
        tarinfo = self.getmember(member)
      File "/miniconda/envs/conda-mirror/lib/python3.5/tarfile.py", line 1741, in getmember
        tarinfo = self._getmember(name)
      File "/miniconda/envs/conda-mirror/lib/python3.5/tarfile.py", line 2321, in _getmember
        members = self.getmembers()
      File "/miniconda/envs/conda-mirror/lib/python3.5/tarfile.py", line 1752, in getmembers
        self._load()        # all members, we first have to
      File "/miniconda/envs/conda-mirror/lib/python3.5/tarfile.py", line 2344, in _load
        tarinfo = self.next()
      File "/miniconda/envs/conda-mirror/lib/python3.5/tarfile.py", line 2275, in next
        self.fileobj.seek(self.offset - 1)
      File "/miniconda/envs/conda-mirror/lib/python3.5/bz2.py", line 277, in seek
        return self._buffer.seek(offset, whence)
      File "/miniconda/envs/conda-mirror/lib/python3.5/_compression.py", line 143, in seek
        data = self.read(min(io.DEFAULT_BUFFER_SIZE, offset))
      File "/miniconda/envs/conda-mirror/lib/python3.5/_compression.py", line 99, in read
        raise EOFError("Compressed file ended before the "
    EOFError: Compressed file ended before the end-of-stream marker was reached

There seems to have been something wrong with the file downloaded – have yet to reproduce what. Possibly our caching proxy helped make things worse…

However, the file should have failed validation, been removed and conda-mirror continued rather than giving up completely.

Will submit a small patch that resolves this by handling EOFError.