SQS #

In modern distributed system architecture — especially those adopting microservices patterns, event-driven architecture, and serverless — direct synchronous communication between services (like direct HTTP REST API calls) often triggers system vulnerabilities. If one target service becomes overloaded or dies, the sending service also fails, creating a cascading failure domino effect.

To solve this problem, we need an intermediary component that can decouple service dependencies and act as a load leveler. Within the AWS ecosystem, AWS Simple Queue Service (SQS) is a fully managed, serverless message queue service that serves as the main foundation for building reliable, secure, and highly scalable asynchronous communication.

What is AWS SQS? #

AWS SQS is a managed message queue service that lets application components exchange data asynchronously without needing to know each other’s location or status directly.

The message delivery process conceptually involves three main roles:

  1. Producer: The application component (e.g., a web server or Lambda function) that creates and sends data messages into an SQS queue.
  2. Message Queue: The managed SQS queue that stores messages safely and durably across AWS’s distributed server cluster spanning multiple Availability Zones.
  3. Consumer: The application component (like a worker process on ECS or a Lambda function) that periodically pulls messages from the queue, processes the data, and deletes the message from the queue after processing completes.
flowchart LR
    Client["HTTP Client"] -->|Request| APIGW["API Gateway"]
    APIGW -->|Synchronous Trigger| LambdaProd["Lambda Producer"]
    LambdaProd -->|"Send Message (Async)"| SQS["AWS SQS Queue<br/>(Buffer / Load Leveling)"]
    SQS -->|Event Source Mapping| LambdaCons["Lambda Consumer"]
    LambdaCons -->|Save Data| DynamoDB["Amazon DynamoDB"]
    
    LambdaCons -.->|Failed > maxReceiveCount| DLQ["Dead Letter Queue (DLQ)<br/>(Error Message Isolation)"]

Queue Types in AWS SQS #

AWS SQS provides two queue types with different performance characteristics and cost limits. The queue type choice must match our application’s business logic needs.

1. Standard Queue #

This is the default queue type offering nearly unlimited horizontal scalability.

  • At-Least-Once Delivery: SQS guarantees every message will be delivered at least once to a consumer. However, due to SQS’s distributed nature, sometimes the same message copy gets delivered more than once.
  • Best-Effort Ordering: The order of messages leaving the queue is generally preserved, but AWS doesn’t guarantee it absolutely.
  • Unlimited Throughput: Supports transaction counts (send/receive messages) per second at nearly unlimited scale.
  • Ideal Use Cases: Application log processing, video conversion pipelines, mass notification email delivery, or any scenario insensitive to data duplication and ordering.

2. FIFO Queue (First-In-First-Out) #

FIFO queues are designed for scenarios requiring strict execution ordering and absolute duplicate message elimination.

  • Strict Ordering: Messages are guaranteed to leave the queue in the exact same order they entered.
  • Exactly-Once Processing: SQS automatically detects and removes duplicate messages sent within a 5-minute window using the MessageDeduplicationId parameter.
  • Limited Throughput: By default supports up to 300 transactions per second (or up to 3,000 transactions per second if the high throughput FIFO feature is enabled).
  • Ideal Use Cases: Financial/banking transactions, e-commerce checkout systems (to prevent double purchases), or sequential document approval flows.

Essential Configuration Options #

Understanding SQS configuration parameters is key to avoiding operational bugs like double message processing or data loss.

1. Visibility Timeout #

When a consumer retrieves a message from an SQS queue, the message isn’t immediately deleted. SQS temporarily hides the message so other consumers don’t pick it up too. This hiding window is called the Visibility Timeout.

  • How It Works: The consumer must process the message and send a delete request to SQS before the Visibility Timeout expires.
  • Potential Problem: If the consumer takes longer to process the message than the Visibility Timeout duration, the message reappears in the queue and gets picked up by another consumer, triggering double processing.
Double-Processing Prevention: Always configure the Visibility Timeout to at least 6 times your consumer Lambda function’s maximum execution duration. If your Lambda function has a 30-second timeout, configure your SQS queue’s Visibility Timeout to at least 180 seconds.

2. Long Polling vs. Short Polling #

Long polling helps us reduce SQS API call costs and save consumer CPU usage.

  • Short Polling (ReceiveMessageWaitTimeSeconds = 0): SQS immediately returns an empty response if the queue is quiet. This triggers a high number of unnecessary empty HTTP requests and increases our monthly bill.
  • Long Polling (ReceiveMessageWaitTimeSeconds = 1 to 20): SQS holds the HTTP connection for up to 20 seconds if the queue is empty. If a message arrives during that wait, SQS immediately sends it to the consumer.

Always use Long Polling with a 20-second value in production environments for request cost efficiency.

3. Dead Letter Queue (DLQ) #

A DLQ is a special SQS queue that holds messages repeatedly failing to be processed by consumers due to data corruption (poison pill) or application code bugs.

We configure the maxReceiveCount parameter (e.g., 3). If a message has been retrieved and processed 3 times but always fails (returning to the queue because the visibility timeout expired), SQS automatically moves that message to the DLQ. This prevents corrupted messages from clogging the main queue indefinitely.

Practical Implementation Example: Node.js #

Here’s an example of producer code for sending messages to SQS, plus an idempotent consumer processing messages from SQS with duplicate prevention logic using DynamoDB (idempotency tracking).

1. Producer (Sending Messages to SQS) #

import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";

const sqsClient = new SQSClient({ region: "ap-southeast-1" });
const QUEUE_URL = process.env.SQS_QUEUE_URL;

export const handler = async (event) => {
    try {
        const orderData = {
            orderId: "ORD-998273",
            amount: 150000,
            timestamp: new Date().toISOString()
        };

        const command = new SendMessageCommand({
            QueueUrl: QUEUE_URL,
            MessageBody: JSON.stringify(orderData),
            // ✓ Sending additional attributes as metadata
            MessageAttributes: {
                "EventType": {
                    DataType: "String",
                    StringValue: "OrderCreated"
                }
            }
        });

        const response = await sqsClient.send(command);
        console.log(`Message sent successfully to SQS. MessageId: ${response.MessageId}`);

        return {
            statusCode: 200,
            body: JSON.stringify({ success: true, messageId: response.MessageId })
        };
    } catch (error) {
        console.error("Failed to send message to SQS", error);
        return { statusCode: 500, body: JSON.stringify({ error: error.message }) };
    }
};

2. Idempotent Consumer (Processing SQS Events) #

// CORRECT: Preventing duplicate side effects by recording successfully processed MessageIds
import { DynamoDBClient, PutItemCommand } from "@aws-sdk/client-dynamodb";

const dbClient = new DynamoDBClient({ region: "ap-southeast-1" });
const IDEMPOTENCY_TABLE = process.env.IDEMPOTENCY_TABLE_NAME;

export const handler = async (event) => {
    // SQS trigger sends an array of message records (batch)
    for (const record of event.Records) {
        const messageId = record.messageId;
        const body = JSON.parse(record.body);

        try {
            // ✓ Step 1: Try saving the messageId to the DynamoDB idempotency table
            // If the MessageId already exists, the database rejects the request (Conditional Write)
            await dbClient.send(new PutItemCommand({
                TableName: IDEMPOTENCY_TABLE,
                Item: {
                    "MessageId": { S: messageId },
                    "ProcessedAt": { S: new Date().toISOString() }
                },
                ConditionExpression: "attribute_not_exists(MessageId)"
            }));

            // ✓ Step 2: Process the main business logic if the messageId was never processed
            console.log(`Processing order: ${body.orderId} with SQS unique ID: ${messageId}`);
            
            // Perform business calculations here...

        } catch (dbError) {
            if (dbError.name === "ConditionalCheckFailedException") {
                // ✓ CORRECT: Safely detect duplicate messages and skip reprocessing
                console.warn(`Message with ID ${messageId} was already processed before. Skipping.`);
                continue;
            }
            // Re-throw other unexpected errors so Lambda retries
            throw dbError;
        }
    }
};

Summary #

  • AWS SQS is a fully managed serverless message queue that decouples service dependencies asynchronously.
  • Standard Queue offers unlimited throughput with at-least-once delivery guarantees, but doesn’t guarantee absolute ordering.
  • FIFO Queue guarantees strict message ordering (first-in-first-out) and exactly-once processing, with certain maximum throughput limits.
  • Visibility Timeout must be configured larger (at least 6 times) than the consumer’s maximum execution duration to avoid double processing.
  • Enable Long Polling (20 seconds) to significantly save SQS API call costs by reducing empty receives.
  • Dead Letter Queue (DLQ) must be used to isolate corrupted messages (poison pills) so they don’t clog the system’s main queue.
← Previous: Fargate   Next: SNS →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact