Pub/Sub #

In modern distributed system architecture, asynchronous communication patterns are the main pillar guaranteeing reliability, scalability, and flexibility between loosely coupled services. When a monolithic application is split into dozens of microservices, synchronous chained communication via HTTP APIs triggers cumulative latency problems and single points of failure. If one downstream service is disrupted, all upstream transactions fail instantly.

Google Cloud Pub/Sub (Publish/Subscribe) addresses these challenges as a globally scaled messaging middleware service fully managed by Google. Designed with a purely serverless model, Pub/Sub lets development teams send and receive millions of messages per second without installing, clustering, or tuning traditional message brokers like RabbitMQ or Apache Kafka. Pub/Sub acts as the central nervous system connecting all our cloud infrastructure components reliably, asynchronously, and automatically.

Internal Architecture and Message Delivery Mechanism #

Understanding how Pub/Sub works internally makes it easier to design efficient, failure-tolerant message delivery patterns. Behind the scenes, Pub/Sub separates the message sender (publisher) role from the message receiver (subscriber) role using logical entities called Topic and Subscription.

flowchart TD
    Publisher["Publisher App (e.g. Cloud Run, GCF)"] -->|"Publish Message"| Topic["Pub/Sub Topic"]
    Topic -->|"Push Delivery"| PushSub["Push Subscription"]
    Topic -->|"Pull Delivery"| PullSub["Pull Subscription"]
    PushSub -->|"HTTP POST (CloudEvents)"| ServiceA["Cloud Run Target Service"]
    PullSub -->|"gRPC Pull Request"| Worker["Worker Instance (Pull Consumer)"]
    ServiceA -->|"Error/Max Retry Exceeded"| DLQ["Dead Letter Queue Topic"]

    style PushSub stroke:#0288d1,stroke-width:2px
    style PullSub stroke:#0288d1,stroke-width:2px

1. The Role of Topic and Subscription #

• Topic: A topic is a channel or logical destination where publishers send messages. Publishers never know who will read the message; their job is only to package the message into binary or JSON data format and send it to the Topic’s HTTP/gRPC endpoint.
• Subscription: A subscription is an entity representing interest in one specific Topic. One Topic can have many Subscriptions (fan-out pattern). Every message arriving at the Topic is automatically copied to all active Subscriptions underneath it. Through this model, one event (e.g., order_created) can be processed in parallel independently by the warehouse system, payment system, and email system.

2. At-Least-Once Delivery Mechanism #

Pub/Sub natively provides an At-Least-Once message delivery guarantee. This means Google Cloud guarantees every published message will be delivered at least once to each entitled subscriber. After delivering a message, Pub/Sub waits for an Acknowledgement (ACK) signal from the subscriber within a time limit (Ack Deadline).

If the subscriber processes the message successfully and sends an ACK, the message is removed from the Subscription queue. However, if network issues or an application crash occurs before the ACK is sent, Pub/Sub redelivers the same message after the deadline expires. Consequently, developers must design subscriber systems to be Idempotent (safe to process the same message repeatedly without corrupting data state).

3. Exactly-Once Delivery Capability #

To reduce the complexity of handling duplicate messages on the application side, GCP provides the Exactly-Once Delivery option at the Subscription level. When enabled, Pub/Sub uses an internal state reconciliation mechanism based on Google Spanner’s global database to track message ACK status in real-time. If a message is being processed or has been successfully ACKed by one consumer, Pub/Sub blocks redelivery of the same message to other consumers, even during regional network failures.

Subscription Types: Pull vs. Push vs. Direct Export #

When designing integrations, Pub/Sub offers three main methods for delivering messages to consumers. This choice greatly affects latency, compute costs, and our system’s network structure.

1. Pull Subscription (Reactive Consumption) #

In the Pull model, the subscriber application acts actively by periodically sending gRPC requests to the Pub/Sub API to request new messages (polling).

• How It Works: The subscriber sends a Pull call, receives a set of messages, processes them, then sends an Acknowledge call carrying the relevant message IDs.
• Best Scenarios: Perfect for large-scale batch processing worker systems, stateful applications running on Compute Engine or Google Kubernetes Engine (GKE), or when consumers want full control over message consumption speed (flow control) to avoid overloading their internal memory capacity.

2. Push Subscription (Automatic Serverless Consumption) #

In the Push model, the Pub/Sub service acts proactively as a webhook client sending messages directly to the subscriber’s HTTP/HTTPS endpoint.

• How It Works: Every time a new message arrives at the subscription, Pub/Sub sends an HTTP POST request carrying the message payload data to a predefined target URL (e.g., a Cloud Run endpoint).
• Best Scenarios: Ideal for serverless services like Cloud Run or Cloud Functions because we don’t need to write message polling loop code. The Cloud Run container just wakes up to receive ordinary HTTP requests and is auto-scaled based on incoming event volume. This model also saves costs because subscriber containers can scale down to zero when there are no events.

3. Direct Export Subscriptions (BigQuery & Cloud Storage) #

Pub/Sub also supports delivering data directly to data warehouses or object storage without writing intermediary glue code.

• BigQuery Subscription: Messages arriving at the Topic are written directly into the target BigQuery table in real-time. The service handles JSON schema parsing automatically and writes data in streaming format.
• Cloud Storage Subscription: Messages are accumulated and written into text or Avro files in a Google Cloud Storage bucket based on specific time intervals or file sizes. This option is very useful for building audit log pipelines or low-cost raw data backups.

Message Ordering and Schemas #

Building large-scale event-driven architecture requires tight control over data integrity. Pub/Sub provides two advanced features to address execution ordering and payload data structure problems.

Message Ordering (Key-Based Ordering) #

By default, Pub/Sub distributes messages in parallel across global partitions, so the arrival order at the consumer side doesn’t always match the publication order. However, for specific use cases like bank transaction history or database change logs (Change Data Capture), execution order is absolute.

• Ordering Key: We can attach a string marker called an ordering key to messages. Pub/Sub guarantees that all messages with the same ordering key are delivered to the subscriber sequentially (first-in-first-out per key).
• Trade-off: Enabling this feature limits parallel delivery throughput capacity, because Pub/Sub must hold the next message until the previous one is successfully processed and ACKed by the subscriber.

Schema Registry (Data Contract Validation) #

When a system is built by many different teams, payload message structures are prone to unexpected breaking changes that can break downstream applications. Pub/Sub addresses this with the Schema Registry feature.

• How It Works: We can register formal data schemas using Apache Avro or Protocol Buffers (Protobuf) formats at the Topic level. Before a message is accepted by the Topic, Pub/Sub validates the payload structure. If the message violates schema rules (e.g., a required field is missing or the data type is wrong), Pub/Sub rejects it with HTTP 400 error code.
• Benefit: Guarantees strong data contracts between services without needing manual validation code on the subscriber side.

Retry Policy, Dead Letter Queue (DLQ), and Error Handling #

Downstream system failures are a certainty in cloud computing. Pub/Sub has built-in error handling to ensure no critical messages are lost during disruptions.

1. Backoff Retry Policy #

When a subscriber fails to process a message and returns a non-2xx response (for Push) or calls Nack (for Pull), Pub/Sub re-queues the message for redelivery.

• Immediate Retry (Default): The message is redelivered as soon as possible without delay.
• Exponential Backoff: We can configure minimum and maximum delay limits (e.g., 10-second delay up to a 600-second maximum). The delivery delay doubles with each repeated failure, giving the downed backend database system time to recover before being bombarded with message requests again.

2. Dead Letter Queue (DLQ) #

There’s a special bug category called poison messages — messages with corrupted or malformed formats that will always fail processing no matter how creative our error handling is. Without special handling, these poison messages trigger infinite retry loops that waste application memory and compute costs.

• DLQ Mechanism: We can configure the dead-letter-topic property and a maximum delivery attempt limit (max delivery attempts, e.g., 5 times). If a message fails processing 5 consecutive times, Pub/Sub automatically moves it to a special Dead Letter Queue Topic for storage and manual analysis by the development team. The original message is then removed from the main Subscription so normal message flow isn’t blocked.

Implementation Code Example: Node.js Publisher & Consumer #

Let’s create a real implementation example using Node.js and the official @google-cloud/pubsub SDK to demonstrate safe message publishing and a robust pull consumer.

1. Installing the SDK Library #

Install the official Google Cloud Pub/Sub library in our Node.js project:

npm install @google-cloud/pubsub

2. Publisher Code (publisher.js) #

The following code shows how to publish messages to a Topic with custom metadata, ordering keys, and asynchronous failure handling.

// CORRECT: Using the official @google-cloud/pubsub SDK
import { PubSub } from '@google-cloud/pubsub';

const pubSubClient = new PubSub({
    projectId: 'my-gcp-project-id'
});

async function publishOrderEvent(orderId, orderData) {
    const topicName = 'order-events';
    const topic = pubSubClient.topic(topicName, {
        // Batching configuration optimization for high throughput
        batching: {
            maxMessages: 100,
            maxMilliseconds: 10,
        }
    });

    // Convert the application payload to a binary buffer
    const dataBuffer = Buffer.from(JSON.stringify(orderData));

    // ✓ CORRECT: Adding custom attributes for routing/filtering
    // and including an ordering key to guarantee per-user processing order
    const messageAttributes = {
        eventType: 'ORDER_CREATED',
        sourceSystem: 'ecommerce-checkout',
    };
    
    const orderingKey = `user-${orderData.userId}`;

    try {
        console.log(`Sending message for Order ID: ${orderId}...`);
        
        const messageId = await topic.publishMessage({
            data: dataBuffer,
            attributes: messageAttributes,
            orderingKey: orderingKey
        });

        console.log(`Message sent successfully. Message ID: ${messageId}`);
        return messageId;
    } catch (error) {
        // ✗ Don't ignore publication failures. Write to central logs for audit
        console.error(`Failed to send message to topic ${topicName}:`, error);
        throw error;
    }
}

// Simulated function execution
publishOrderEvent('ORD-99882', {
    orderId: 'ORD-99882',
    userId: '10029',
    totalAmount: 1250000,
    items: ['SSD 1TB', 'RAM DDR5 32GB']
});

3. Pull Subscriber Code (subscriber.js) #

The code below defines a Pull-type subscriber that persistently listens for new events, configures parallel message handling limits, and manages message acknowledgement (ACK/NACK) safely.

import { PubSub } from '@google-cloud/pubsub';

const pubSubClient = new PubSub({
    projectId: 'my-gcp-project-id'
});

function startSubscriptionListener() {
    const subscriptionName = 'inventory-order-sub';
    const subscription = pubSubClient.subscription(subscriptionName, {
        // Setting the concurrency limit for internal application memory consumption
        flowControl: {
            maxMessages: 20, // Maximum 20 messages processed in parallel at a time
        }
    });

    console.log(`Subscriber listening for messages from: ${subscriptionName}...`);

    // Handler when a message arrives
    subscription.on('message', async (message) => {
        console.log(`Received message ID: ${message.id}`);
        console.log(`Message Attributes:`, message.attributes);

        try {
            // Parse the raw payload
            const orderPayload = JSON.parse(message.data.toString());
            console.log(`Processing inventory update for Order ID: ${orderPayload.orderId}`);

            // Run the main business logic (e.g., update database stock)
            await updateInventoryStock(orderPayload.items);

            // ✓ CORRECT: Send the ACK signal if processing succeeds without errors
            message.ack();
            console.log(`Message ${message.id} processed and ACKed successfully.`);
        } catch (error) {
            // ✗ Don't let errors kill the worker loop.
            console.error(`Failed to process message ${message.id}:`, error);
            
            // Send the NACK signal so Pub/Sub immediately redistributes this message to another worker instance
            message.nack();
        }
    });

    // Handler for internal SDK connectivity errors
    subscription.on('error', (error) => {
        console.error(`An internal error occurred on the subscription listener:`, error);
    });
}

// Simulated async database function
async function updateInventoryStock(items) {
    return new Promise((resolve) => setTimeout(resolve, 500));
}

// Run the worker listener
startSubscriptionListener();

Comparison: GCP Pub/Sub vs. Apache Kafka vs. AWS SQS/SNS #

To help system architects choose the message routing solution best matching their technical needs, the table below compares Pub/Sub with other leading industry solutions.

Best Practices for Scalability and Security #

Building a secure, efficient Pub/Sub architecture requires applying the following design principles:

1. Handle Consumer Idempotency Consistently #

Because of the At-Least-Once delivery guarantee, duplicate message delivery from connection failures at the end of a transaction is common. Our consumers must be designed to reject duplicate requests.

• DO: Use a unique business transaction ID embedded in the payload (e.g., orderId or a transaction UUID) as a unique key in the target database. If the database detects the same key was already written, immediately ignore the message without triggering an error.

2. Use the Payload Reference Pattern for Large Data (> 10MB) #

The maximum message size limit in Google Cloud Pub/Sub is 10 MB per message. Sending payloads near this limit is a major anti-pattern because it increases network transmission latency and incurs expensive bandwidth costs.

• DO: Use the Claim Check Pattern. Upload large data files (like videos or images) to Google Cloud Storage first. Then, send a small Pub/Sub message containing only metadata and the file reference URL in Cloud Storage. Subscribers then download the file directly from the GCS bucket when processing begins.

3. Monitor Backlog Metrics Proactively #

The health of our event-driven system is reflected in the consumer’s ability to process message queues on time.

• DO: Create monitoring dashboards in Cloud Monitoring using the pubsub.googleapis.com/subscription/num_undelivered_messages metric (backlog message count) and pubsub.googleapis.com/subscription/oldest_unacked_message_age metric (age of the oldest un-ACKed message). Set automatic alerts to Slack or pager if undelivered message age exceeds a 15-minute limit, indicating a bug or stuck worker application.

Summary #

• Google Cloud Pub/Sub is a globally scaled serverless message broker that asynchronously decouples publishers and subscribers.
• The At-Least-Once guarantee requires developers to design idempotent consumers to handle potential message duplication.
• Push Subscriptions are perfect for Cloud Run because they natively trigger subscriber instance auto-scaling based on event traffic.
• Schema Registry supports Avro and Protobuf to guarantee payload message contract consistency without writing manual validation code.
• Use Dead Letter Queues (DLQ) to automatically handle poison messages and prevent infinite retry cycles.
• Use the claim check pattern by moving large payload data (> 10MB) to Cloud Storage and sending only the URL reference.