Serverless Storage Solutions

# CHAPTER 10

Serverless Storage Solutions #

1. Introduction #

2. Learning Objectives #

• Define Object Storage (Amazon S3, Google Cloud Storage).

• Differentiate between Block Storage (VM Disks) and Object Storage.

• Understand the architecture of Pre-Signed URLs for secure uploads.

• Configure Cross-Origin Resource Sharing (CORS) for direct browser uploads.

• Utilize Cloud Storage to trigger Event-Driven serverless workflows.

3. Beginner-Friendly Explanation #

• Block Storage (Traditional Hard Drive): Like a filing cabinet. You have to format it, organize it into folders, and it has a strict physical limit. When it's full, you must buy a bigger cabinet.

• Object Storage (Amazon S3): A magic valet service. You hand the valet a box (a file). You don't care where they park it. The valet hands you a unique claim ticket (a URL). When you want the box back, you hand them the ticket, and they instantly retrieve it. The parking lot has infinite space, and you only pay for the exact size of the boxes you park.

4. Amazon S3 (Simple Storage Service) #

• Buckets: The top-level folder. Bucket names must be globally unique across all of AWS (e.g., my-app-profile-pics-1234).

• Objects: The actual files (images, videos, JSON documents). Each object has Data, Metadata, and a globally unique URL.

• Infinite Scale: You can store 1 byte or 5 Petabytes. S3 auto-scales invisibly.

5. The "Pass-Through" Anti-Pattern #

6. The Solution: Pre-Signed URLs #

1. 1. The Frontend (React app) asks the Lambda function: "I want to upload profile.jpg."

1. 2. The Lambda function uses IAM permissions to generate a temporary, cryptographically signed URL (A Pre-Signed URL) valid for exactly 5 minutes.

1. 3. The Lambda function returns this URL to the Frontend instantly.

1. 4. The Frontend takes the 10MB image and uploads it directly to S3 using the Pre-Signed URL, bypassing the API Gateway and Lambda entirely!

7. Mini Project: Conceptual Direct Upload #

Step-by-Step Overview:

1. 1. The Bucket: Create an S3 Bucket. Configure its CORS policy to allow PUT requests from your frontend domain.

1. 2. The Lambda Code: Write a function to generate the secure ticket:

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const s3 = new S3Client({ region: "us-east-1" });

export const handler = async (event) => {
    // Determine a random filename
    const filename = `uploads/${Date.now()}-image.jpg`;
    
    const command = new PutObjectCommand({
        Bucket: "my-app-profile-pics-1234",
        Key: filename,
        ContentType: "image/jpeg"
    });

    // Generate the magic URL valid for 300 seconds
    const uploadURL = await getSignedUrl(s3, command, { expiresIn: 300 });

    return {
        statusCode: 200,
        body: JSON.stringify({ uploadURL, filename })
    };
};

1. 3. The Result: Your React app hits this API, gets the uploadURL, and pushes the file directly into the massive, infinitely scalable S3 bucket. Zero compute bottleneck!

8. Real-World Scenarios #

9. Best Practices #

• Lifecycle Policies: Object storage costs add up. Configure an S3 Lifecycle Policy to automatically delete temporary files after 7 days, or automatically move old files (like 1-year-old invoices) to S3 Glacier (an ultra-cheap, deep-archive storage tier), automating cost optimization without writing any code.

10. Cost Optimization Tips #

• Bandwidth/Egress Costs: Storing data in S3 is incredibly cheap (pennies per GB). However, data *leaving* AWS to the public internet (Egress) is expensive. If you are serving massive video files to users, always place a CDN (like Amazon CloudFront) in front of your S3 bucket to cache the data locally and drastically reduce egress bandwidth costs.

11. Exercises #

1. 1. Explain the architectural "Pass-Through" anti-pattern regarding file uploads in a Serverless environment.

1. 2. Describe the mechanical flow of utilizing a Pre-Signed URL to facilitate a direct-to-storage file upload.

12. FAQs #

13. Interview Questions #

• Q: A client needs to allow users to upload 2GB video files to their serverless application. Detail the architecture required to facilitate this upload securely without hitting API Gateway payload limitations or incurring excessive Lambda execution costs.

• Q: Explain the integration between Amazon S3 and AWS Lambda in an Event-Driven Architecture. Provide an example of how an S3 ObjectCreated event can decouple an upload workflow from a processing workflow.

14. Summary #

15. Next Chapter Recommendation #