skills.rest
richardnroman
richardnromanCOMMUNITY
Explore all skills by this author →
GitHub

processing-s3-uploads-with-step-functions

Deploys an event-driven Step Functions workflow routing S3 uploads to Lambda or Fargate by file size.

Updated Jul 21, 2026
Search Tags:#aws#workflow orchestration#s3#eventbridge#lambda#fargate#step functions
One-click install
npx skills add https://github.com/richardnroman/Synthetic-Data-Exchange-Licensing-Platform --skill processing-s3-uploads-with-step-functions-richardnroman
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: processing-s3-uploads-with-step-functions
Source: https://github.com/richardnroman/Synthetic-Data-Exchange-Licensing-Platform/tree/main/.agents/skills/processing-s3-uploads-with-step-functions
Command: npx skills add https://github.com/richardnroman/Synthetic-Data-Exchange-Licensing-Platform --skill processing-s3-uploads-with-step-functions-richardnroman

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires boto3, and includes scripts (resource) and references (resource) components.

What problem does it solve? Processing S3 uploads often requires different compute depending on file size: small files fit Lambda's limits while large files need containerized Fargate tasks. Manually wiring EventBridge, Step Functions, Lambda, ECS, ECR, VPC networking, and scoped IAM roles into a working event-driven pipeline is error-prone and time-consuming. ## Core Features & Use Cases - Size-Based Routing: An EventBridge rule triggers a Step Functions state machine on S3 object creation, and a Choice state sends files up to 6 MB to Lambda and larger files to a Fargate task. - Complete Infrastructure Provisioning: Creates the S3 bucket with EventBridge notifications and KMS encryption, ECR repository, ECS cluster and task definition, VPC security group with scoped egress, and five least-privilege IAM roles via AWS CLI. - Built-In Observability: Configures a CloudWatch log group with KMS encryption, an SQS dead-letter queue for failed EventBridge invocations, and a CloudWatch alarm on Step Functions execution failures. - Use Case: A data engineering team receives mixed-size uploads from partners and needs small CSVs processed in seconds by Lambda while multi-gigabyte archives run on Fargate containers, all orchestrated automatically. ## Quick Start Ask the agent to deploy the S3 upload processing workflow by providing a bucket name, AWS region, ECR repository name, and Step Functions state machine name.

Frequently Asked Questions about processing-s3-uploads-with-step-functions

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I route S3 uploads to Lambda or Fargate based on file size?▼

Enable EventBridge notifications on the S3 bucket, create an EventBridge rule matching Object Created events, and target a Step Functions state machine. A Choice state checks $.detail.object.size and routes files over 6 MB to an ECS RunTask state and smaller files to a Lambda Invoke state.

How do I trigger a Step Functions state machine from S3 events?▼

Enable EventBridge notifications on the bucket with put-bucket-notification-configuration, then create an EventBridge rule with source aws.s3 and detail-type Object Created. Add the state machine as a target using an IAM role with states:StartExecution permission.

What is the file size threshold between Lambda and Fargate processing?▼

The state machine routes files larger than 6 MB (6291456 bytes) to the Fargate task and all other files to Lambda. The Lambda function also rejects files over 512 MB as a defensive validation measure.

Why is my EventBridge rule not triggering Step Functions on S3 uploads?▼

Verify EventBridge notifications are enabled on the bucket with get-bucket-notification-configuration, confirm the rule exists with describe-rule, and check the target has the correct state machine ARN and an IAM role allowing states:StartExecution.

Why is my Fargate task stuck in PROVISIONING state?▼

Fargate tasks stall when subnets lack auto-assign public IP or the route table has no internet gateway route for 0.0.0.0/0. Verify the subnets have internet access and the security group permits outbound HTTPS and DNS traffic.

When should I not use Step Functions for S3 file processing?▼

Skip this pattern when all files fit Lambda's limits, since a direct S3 to Lambda trigger is simpler. Also avoid it for real-time streaming workloads, where Kinesis is the appropriate service.

Related Skills

Explore More Software Engineering Skills→
selrai-companyselrai-company

aws-connector

Install and configure the AWS CLI with access-key or SSO authentication.

Official
Intermediate
cheetah-alocheetah-alo

data-analytics

Implement Python ETL pipelines with schema validation and idempotent execution.

Community
Intermediate
mreferremreferre

setting-up-cloudwatch-alarm-notifications

Configure encrypted SNS topics and subscriptions for CloudWatch alarm notifications.

Community
Intermediate
sitharaj88sitharaj88
1

aws-kinesis

Generate AWS Kinesis configurations for data streams, Firehose, and analytics.

Community
Advanced