Stream Files to Amazon S3
April 20, 2015 • programming
For any SaaS platform it is common to use a 3rd party hosting service for uploading files and serving them through a CDN. Amazon S3 is a common choice.
Usually, the file upload from the client side (say, AngluarJS) is sent to the server (node server running on GNU/Linux box) which is then forwarded to Amazon S3. This approach is inefficient because the file needs to be opened and “read” by the server and then forwarded to S3. The solution provided here:
- Extracts all the metadata (file name, size, mime type)
- Opens a File stream to the Amazon S3 bucket
- And writes the file directly to it.
Code:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| var express = require('express'); | |
| var router = express.Router(); | |
| var multer = require('multer'), //for handling multipart/form-data | |
| fs = require('fs'), | |
| S3FS = require('s3fs'), //abstraction over Amazon S3's SDK | |
| s3fsImpl = new S3FS('your-bucket-here', { | |
| accessKeyId: 'Your-IAM-Access', | |
| secretAccessKey: 'Your-IAM-Secret' | |
| }); | |
| // POST a new Path | |
| router.post('/', [ multer(), function(req, res) { | |
| var file = req.files.file; | |
| console.log(file); | |
| /* Output: | |
| { | |
| fieldname: 'file', | |
| originalname: 'ice-boxes.jpg', | |
| name: '2658a8f666e33ab1ec39dc8b7b20970b.jpg', | |
| encoding: '7bit', | |
| mimetype: 'image/jpeg', | |
| path: 'public/uploads/2658a8f666e33ab1ec39dc8b7b20970b.jpg', | |
| extension: 'jpg', | |
| size: 88076, | |
| truncated: false, | |
| buffer: null | |
| } | |
| */ | |
| //Create a file stream | |
| var stream = fs.createReadStream(file.path); | |
| //writeFile calls putObject behind the scenes | |
| s3fsImpl.writeFile(file.name, stream).then(function () { | |
| fs.unlink(file.path, function (err) { | |
| if (err) { | |
| console.error(err); | |
| } | |
| }); | |
| res.status(200).end(); | |
| }); | |
| }]); | |
| module.exports = router; |
I struggled with it for quite some time when I needed to implement this feature in one of my projects. I hope it will help the community in the future.