publishDir fails to publish files to s3 when >5gb

[Quin Scacheri]

When the result of a process is larger than 5gb, the upload to s3 fails with the error:

The specified copy source is larger than the maximum allowable size for a copy source: 5368709120 (Service: Amazon S3; Status Code: 400; Error Code: InvalidRequest

Minimal reproducible example:

nextflow.enable.dsl=2

process createFile {
    publishDir "s3://some-bucket", mode: 'copy', overwrite: true, failOnError: true
    output:
        path("*.gz")
    shell:
    '''
    truncate -s 10G test.tar.gz
    '''
}

workflow {
    createFile()
    createFile.out.view { "File: $it" }
}