sylabs / scs-library-client

The client supports parallel downloads using HTTP Range Requests. Currently, the client assumes ranged requests are supported if and only if the server performs a redirect via a http.StatusSeeOther. Otherwise, c.singleStreamDownload is utilized:

scs-library-client/client/pull.go

Lines 273 to 278 in 2e30d58

    
           if res.StatusCode == http.StatusOK { 
        
           	// Library endpoint does not provide HTTP redirection response, treat as single stream, direct download 
        
           	c.Logger.Logf("Library endpoint does not support concurrent downloads; reverting to single stream") 
        
           	return c.singleStreamDownload(ctx, dst, res, pb) 
        
           }

This negatively impacts download performance against servers that support ranged requests, but do not redirect via http.StatusSeeOther, and is the root cause of #129.

The current logic should be replaced with logic that detects HTTP range support in the standard fashion (ref).

This currently affects (only) Singularity Enterprise 2.0.x, which does not redirect on the /v1/imagefile endpoint.

	if res.StatusCode == http.StatusOK {
	// Library endpoint does not provide HTTP redirection response, treat as single stream, direct download
	c.Logger.Logf("Library endpoint does not support concurrent downloads; reverting to single stream")

	return c.singleStreamDownload(ctx, dst, res, pb)
	}

Improve detection of HTTP "Range" requests in ConcurrentDownloadImage