Automating File Uploads to Amazon S3 using Chokidar CLI

Project Overview

At our digital agency, we specialize in creating innovative and efficient solutions for businesses. Recently, we partnered with a skydiving facility to automate their media file distribution process. The facility captures high-resolution videos and images of each skydiver’s jump, but their traditional method of delivering these files to customers involved physical USB drives—an outdated, costly, and labor-intensive process. They needed a digital solution that would allow their customers to easily access, view, and download their media files online from any device, also allowing them to share their files directly with friends and family.

The Problem

Before our intervention, the skydiving facility’s post-production team would manually edit the media files, which included color correction and enhancements. Once completed, these files were stored on network storage devices and distributed to customers using USB drives. This approach, while functional, posed several challenges:

• High operational costs related to physical storage and USB distribution.
• Inefficiencies due to manual processes.
• Limited convenience for customers who had to both wait at the tunnel until the files had been edited.

The facility needed a streamlined, automated, and digital approach to distribute these media files to enhance customer experience while reducing operational overhead.

The Solution: Automated Cloud Uploads with Chokidar CLI

To solve this, we developed an automated system that handles the upload of media files directly to an Amazon S3 bucket once the editing process is completed. This is where the Chokidar CLI, a file-watching tool, came into play. By integrating Chokidar CLI, our solution monitors the local directory where edited media files are stored and automatically triggers an upload script whenever new or updated files are detected.

This automation eliminates the need for manual uploads and physical storage, ensuring that each file is securely stored in the cloud and ready for customer access.

Seamless Access through Link Generation

After the upload process, our system generates a pre-signed URL for each media file. This URL provides secure, temporary access, allowing customers to download their videos and images without needing physical media. The system then sends a personalized email to each customer containing their unique download link and instructions on how to access the files.

This automated distribution method drastically reduces the time and effort involved in delivering media while providing a much more user-friendly experience for the skydivers.

Benefits of Cloud Storage

• Cost Efficiency: By moving away from physical storage devices and manual distribution, the skydiving facility significantly reduces its operational costs, particularly in materials and labor.
• Enhanced Customer Convenience: With digital access to their videos and images, customers can now view and download their media from anywhere, at any time, without having to wait for a physical device
• Scalability: As the skydiving facility grows, the cloud storage solution can easily scale to accommodate increasing volumes of media files without needing additional physical infrastructure.

The Code: How It Works

Let’s dive into the code to see how we implemented this automated cloud upload system using Chokidar CLI and Amazon S3:

1. Create Watcher on Folder

Copy
const chokidar = require('chokidar');

const watcher = chokidar.watch(`${config.FOLDER}`, {
	persistent: true,
});

watcher
	.on('add', (path) => upload(path))
	.on('unlink', (path) => remove(path))
	.on('error', (error) => console.error(`Watcher error: ${error}`))
	.on('ready', () => console.log('Watcher started'));

2. Upload to Amazon S3

Copy
const AWS = require('aws-sdk');
const fs = require('fs');

const s3 = new AWS.S3({
	accessKeyId: config.AWS_ACCESS_KEY_ID,
	secretAccessKey: config.AWS_SECRET_ACCESS_KEY,
});

const upload = (path) => {
	const file = fs.readFileSync(path);

	s3.upload(
		{
			Bucket: config.BUCKET,
			Key: path.replace(config.FOLDER, ''),
			Body: file,
		},
		(error, data) => {
			if (error) {
				console.error(`Upload error: ${error}`);
			} else {
				console.log(`File uploaded: ${data.Location}`);
			}
		},
	);
};

3. Create S3 Key

Copy
const createKey = (path) => {
	// Based on your requirement, create the S3 key.
	// Maintain the same folder structure as the local directory.
	const key = path.replace(`${config.FOLDER}/`, '');
	return key;
};

4. Handle Uploading File

Copy
const upload = async (path) => {
	try {
		const key = createKey(path);
		const exists = await checkExists(key);

		if (exists) {
			console.log('File already exists, skipping upload.');
			return { status: false, message: 'File already exists' };
		}

		const fileContent = fs.readFileSync(path);
		await s3Client.send(
			new PutObjectCommand({
				Key: key,
				Bucket: config.S3_BUCKET_NAME,
				Body: fileContent,
			}),
		);

		// Get signed URL to access the file
		const s3SignedUrl = s3Client.getSignedUrl('getObject', {
			Bucket: config.S3_BUCKET_NAME,
			Key: key,
			Expires: 3600, // URL expiration time in seconds
		});

		return { status: true, url: s3SignedUrl };
	} catch (e) {
		return { status: false, error: e.toString() };
	}
};

5. Handle Removing File

Copy
const remove = async (path) => {
	try {
		const key = createKey(path);
		const exists = await checkExists(key);

		if (!exists) {
			console.log('File does not exist, skipping removal.');
			return { status: false, message: 'File does not exist' };
		}

		await s3Client.send(
			new DeleteObjectCommand({
				Key: key,
				Bucket: config.S3_BUCKET_NAME,
			}),
		);
		return { status: true };
	} catch (e) {
		return { status: false, error: e.toString() };
	}
};

6. Sending URLs to Customers

Ensure you integrate this logic into your existing email or SMS sending system to notify customers with the URL link.

7. Create bundle (dmg or exe)

Copy
npm install -g pkg

pkg . --targets node14-win-x64,node14-macos-x64,node14-linux-x64

Conclusion

Through this automated cloud-based solution, we have modernized the way the skydiving facility delivers media to its customers. By leveraging the power of Amazon S3 and the Chokidar CLI for automation, we not only reduced operational costs but also greatly improved customer satisfaction. This project highlights our commitment to providing tailored digital solutions that drive efficiency and enhance user experiences.

You can now copy this content directly into your Markdown file. Let me know if you need any further assistance!