Send files larger than 2 GB to AWS from IBM i

Follow

Issue

Zend Server for IBM i is a 32 bit program, which means it is limited to file sizes around 2 GB.  This example shows how to use the split command in PASE to break the file into smaller chunks and send them up to AWS.  The split command is a standard command in Linux, so this technique would also apply there.

Environment

Any version of Zend Server for IBM i.

Resolution

The way to accomplish this with 32 bit PHP is to 'chunk' the file. To 'chunk' the file means to break it up into smaller pieces, and transmit the pieces. PASE uses the 'split' command to break up the file. Then the AWS SDK provides for sending the pieces. You can read the AWS documentation here:

Upload a File in Multiple Parts Using the PHP SDK Low-Level API

Chunking the file is fairly straightforward.  Here is the format for the PASE command:

$ split -b 1000m /path/to/large/file /path/to/output/file/prefix

You end up with files having prefix_aa, prefix_ab, so you can then pass each chunk to uploadPart, and finally call completeUpload and you’re done.  Here is an example PASE session:

$ split -b 500m test testnew 
$ ls -la testnew*
-rw-r--r-- 1 maurice 0 524288000 Mar 10 14:38 testnewaa
-rw-r--r-- 1 maurice 0 524288000 Mar 10 14:39 testnewab
-rw-r--r-- 1 maurice 0 524288000 Mar 10 14:39 testnewac
-rw-r--r-- 1 maurice 0 524288000 Mar 10 14:40 testnewad
-rw-r--r-- 1 maurice 0 314572800 Mar 10 14:40 testnewae

So you call Aws\S3\S3Client::createMultipartUpload() which will initialize the upload and give you an upload id.

Then call Aws\S3\S3Client::uploadPart()with a chunk and a part number, iteratively or in parallel for all the chunks.

Finally you call Aws\S3\S3Client::completeMultipartUpload() to finalize the upload.

For code examples and more information, please refer to the AWS documentation in the above link.

Have more questions? Submit a request

Comments

  • Avatar
    Zanikweb Zanikweb

    I tried to upload 2GB PHP file on wordpress site but unable to upload and got error is their any other method to upload file on wordpress theme my site: http://www.zanikweb.com

Powered by Zendesk