int size errors when uploading large file (8GB) on 32-bit machine (raspberry pi)

[Joseph Gleason]

Error:

[Exception] Python int too large to convert to C long
[Thread Failure] Python int too large to convert to C long

Command line:
s4cmd put -c 4 /mnt/xfer/largefile s3://bucketarray/largefile

Platform: Debian on armv7l (Raspbian)
Software:

s4cmd=2.1.0+ds-1

Hypothesis: Some code somewhere is attempting to convert a large python into into a C 'long' which can vary in size between platforms. Probably needs to be a long long or something.

[Kris Lou]

The problem is here:

s4cmd/s4cmd.py

Lines 1296 to 1307 in e74e0e7

 @log_calls 
 def read_file_chunk(self, source, pos, chunk): 
   '''Read local file chunk''' 
   if chunk==0: 
       return StringIO() 
   data = None 
   with open(source, 'rb') as f: 
     f.seek(pos) 
     data = f.read(chunk) 
   if not data: 
     raise Failure('Unable to read data from source: %s' % source) 
   return StringIO(data) 

In particular:

s4cmd/s4cmd.py

Line 1304 in e74e0e7

 data = f.read(chunk) 

When the chunk is larger than a 32-bit system can handle, then it crashes. This is probably the case for all files over 2 GB.

One workaround seems to be setting --max-singlepart-upload-size to something smaller than 2 GB (and it defaults to 4.5 gb).