Eventarc #

In the modern digital transformation era, event-driven architecture (EDA) has become the main foundation for building responsive, high-performance, loosely coupled distributed systems. In traditional microservice systems, inter-service integration is often forced to use rigid synchronous HTTP calls. When a file is uploaded to object storage, we must write special backend code to detect the upload, or have the frontend trigger webhooks to other services sequentially. This approach increases the risk of cascading failures and adds code complexity.

Google Cloud Eventarc addresses this complexity as a fully managed serverless event routing service. Eventarc acts as a universal intermediary system capable of capturing events from various sources — internal Google Cloud services, audit logs, our own custom applications, or third-party SaaS platforms — filtering them granularly, translating them into industry standards, and channeling them directly to serverless compute targets like Cloud Run, Cloud Functions, or Google Cloud Workflows. With Eventarc, we can build large-scale reactive systems without writing glue code or managing a single event broker infrastructure.

Internal Architecture and Technology Behind Eventarc #

To understand Eventarc’s reliability in capturing and delivering millions of events in real-time, we need to see how it’s built on top of GCP infrastructure foundations.

1. Pub/Sub as the Background Transport Engine #

Under the hood, Eventarc leverages the reliability of Google Cloud Pub/Sub infrastructure for message transmission. Every time we create an Eventarc Trigger, Google automatically provisions a hidden Pub/Sub topic acting as the event transport path. Users don’t pay a cent for this hidden topic; all costs are consolidated into Eventarc usage rates.

2. CNCF CloudEvents v1.0 Format Standard #

One of the biggest challenges in event-driven systems is the variety of payload data formats from different sources. Eventarc solves this by adopting the CloudEvents v1.0 standard managed by the Cloud Native Computing Foundation (CNCF). All events passing through Eventarc are packaged into a uniform standard JSON structure containing mandatory metadata attributes like:

• specversion: The CloudEvents specification version used (always 1.0).
• type: The event type that occurred (e.g., google.cloud.storage.object.v1.finalized for file uploads).
• source: The URI identifying the event’s origin (e.g., the GCS bucket name).
• id: The unique event ID for easier duplicate message detection on the consumer side.
• time: The timestamp of when the event first occurred in the origin system.
• datacontenttype: The media format of the data payload (usually application/json).
• subject: Specific details about the affected resource (e.g., the file path /photos/avatar.png).
• data: The specific payload of the original event containing detailed object metadata.

This standardization ensures target receivers (like Cloud Run services) can process events from any source using the same universal CloudEvents parser library.

3. Cloud Audit Logs as a Universal Event Source #

What if we want to trigger a function when a Compute Engine VM is stopped, or when a BigQuery dataset is created? Most GCP services don’t have built-in event triggers. Eventarc overcomes this by integrating with Cloud Audit Logs.

Every time modification activity occurs in any GCP service, activity logs are written to Cloud Audit Logs. Eventarc monitors these audit logs in real-time. If it detects an activity pattern matching our trigger filter, Eventarc converts that log entry into CloudEvent format and sends it to the target destination. Through this clever mechanism, more than 130 GCP services automatically become potential event sources for our applications.

Event Source Types #

Eventarc groups event sources into several main categories to simplify authentication and routing configuration:

1. Google Cloud Services (Direct Events) #

Events triggered directly from supporting GCP services without going through audit logs. Classic examples are Google Cloud Storage (e.g., object creation events object.v1.finalized or object deletion object.v1.deleted) and real-time Firestore queries. These direct event options offer very low delivery latency because they don’t wait for audit logs to finish writing to disk.

• GCS Trigger Evolution Gen 1 vs Gen 2: In old Cloud Functions Gen 1, storage triggers used direct Pub/Sub-based notifications hidden inside the bucket. In Gen 2, Eventarc acts as the single routing system. This simplifies network topology because we no longer need to provision or maintain Pub/Sub correlations manually at the bucket level; everything is managed declaratively through one Eventarc API.

2. Custom Application Events (Custom Channels) #

We can use Eventarc to deliver our own application-created events across GCP projects or to other developer teams. We publish custom events to a dedicated Eventarc Channel using the standard CloudEvents format. Eventarc then routes them to various interested targets based on configured filters.

3. SaaS Third-Party Partners #

Eventarc supports native integration with well-known external SaaS providers like Auth0, Datadog, PagerDuty, and MongoDB Atlas. Using the Partner Channels feature, we can capture security events from Auth0 (e.g., user_signup or failed_login events) and route them directly to our Cloud Run backend on GCP to trigger investigation workflows or automatic welcome email delivery without exposing our public API endpoints to the internet.

• Partner Channel Working Mechanism: Integration begins when the SaaS partner platform publishes events to the Google Partner Source endpoint authorized by our GCP project ID. In the Google Cloud console, we then approve the creation of an association channel (partner channel). Once the channel is active, we can create Eventarc triggers with specific filter criteria to route that SaaS payload directly to our private internal services without writing intermediary scripts.

Event Transmission Diagram (Eventarc Flowchart) #

Here’s a visualization of how events flow from various source categories, are filtered by the Eventarc routing engine, and distributed to serverless targets with failure handling.

flowchart TD
    subgraph Event Sources
        GCS["Cloud Storage Bucket"]
        CAL["Cloud Audit Logs"]
        CustomApp["Custom Application (Pub/Sub)"]
        SaaS["SaaS Partner (e.g. Auth0)"]
    end
    
    subgraph Eventarc Routing Engine
        Trigger["Eventarc Trigger"]
        Router["Filter & Router (CloudEvents v1.0)"]
        Trigger --> Router
    end
    
    subgraph Target Destinations
        CR["Cloud Run Service"]
        GCF["Cloud Functions Gen 2"]
        Workflows["GCP Workflows"]
    end
    
    GCS --> Trigger
    CAL --> Trigger
    CustomApp --> Trigger
    SaaS --> Trigger
    
    Router -->|"HTTP POST"| CR
    Router -->|"HTTP POST"| GCF
    Router -->|"HTTP POST"| Workflows
    Router -->|"Failure after Max Retry"| DLQ["Pub/Sub Dead Letter Queue"]

    style Router stroke:#0288d1,stroke-width:2px

Delivery Policy, Retry, and Failure Handling (DLQ) #

Guaranteeing reliable event delivery in a dynamic cloud environment is crucial. Eventarc applies strict delivery guarantee systems to ensure no events are lost along the way.

1. At-Least-Once Delivery Guarantee #

Eventarc guarantees every event is delivered at least once to the target. Because targets (like Cloud Run or Cloud Functions) receive events through ordinary HTTP POST requests, Eventarc considers delivery successful if the target responds with a successful HTTP status code (2xx). If the target responds with an error code (non-2xx) or the connection times out, Eventarc schedules redelivery. Consequently, our target application code must be designed Idempotent to avoid side effects from duplicate event processing.

2. Exponential Retry Policy #

To handle temporary overload issues (temporary rate limits) on targets, Eventarc applies an automatic retry policy with exponentially increasing delays (exponential backoff). The system keeps periodically attempting to deliver the event for up to a maximum of 24 hours since the event was first created.

3. Dead Letter Queue (DLQ) Integration #

If after 24 hours of retries the target still fails to respond successfully (e.g., due to a permanent bug in the target application code), Eventarc moves the problematic event to a Dead Letter Queue (DLQ) based on a Pub/Sub topic we configured beforehand. By moving corrupted events to the DLQ, we prevent other event queues from piling up and can safely analyze error causes without losing the original event data.

4. Load Management and Throttling #

Eventarc protects our downstream apps from request flooding during event storms. If tens of thousands of files are uploaded to GCS at once, Eventarc monitors our target Cloud Run’s response speed. If Cloud Run starts returning HTTP 429 status or experiencing latency increases, Eventarc temporarily holds event delivery in its Pub/Sub queue (backpressure handling), flowing events according to the target’s scalability capacity without collapsing our backend servers.

IaC Configuration: Eventarc Trigger via Terraform #

Managing Eventarc configuration at production level is highly recommended using Infrastructure as Code (IaC) tools like Terraform to guarantee cross-environment reproducibility. Below is an example of Terraform code creating an Eventarc Trigger that captures new file creation events in Cloud Storage and sends them to a Cloud Run service.

# 1. Defining a dedicated Service Account for the Eventarc Trigger
resource "google_service_account" "eventarc_sa" {
  account_id   = "eventarc-trigger-sa"
  display_name = "Service Account khusus untuk Eventarc routing"
}

# 2. Granting permission for Eventarc to invoke Cloud Run
resource "google_cloud_run_service_iam_member" "run_invoker" {
  service  = google_cloud_run_service.backend_service.name
  location = google_cloud_run_service.backend_service.location
  role     = "roles/run.invoker"
  member   = "serviceAccount:${google_service_account.eventarc_sa.email}"
}

# 3. Granting permission for Eventarc to read logs from Cloud Storage
resource "google_project_iam_member" "eventarc_receiver" {
  project = var.project_id
  role    = "roles/eventarc.eventReceiver"
  member  = "serviceAccount:${google_service_account.eventarc_sa.email}"
}

# 4. Defining an Eventarc Trigger to detect files in Cloud Storage
resource "google_eventarc_trigger" "gcs_trigger" {
  name     = "gcs-upload-trigger"
  location = "asia-southeast2" # Jakarta region

  # Determining the compute target receiving events
  destination {
    cloud_run_service {
      service = google_cloud_run_service.backend_service.name
      region  = google_cloud_run_service.backend_service.location
      path    = "/events/gcs-upload" # Target HTTP endpoint
    }
  }

  # Determining the event filter being searched
  matching_criteria {
    attribute = "type"
    value     = "google.cloud.storage.object.v1.finalized"
  }

  matching_criteria {
    attribute = "bucket"
    value     = google_storage_bucket.media_uploads.name
  }

  # Connecting the sender service account
  service_account = google_service_account.eventarc_sa.email

  # Explicit dependencies to guarantee resource creation order
  depends_on = [
    google_cloud_run_service_iam_member.run_invoker,
    google_project_iam_member.eventarc_receiver
  ]
}

Comparison: Eventarc vs. Pub/Sub (When to Choose Which?) #

Many developers are confused distinguishing Eventarc’s role from Pub/Sub because both are used for asynchronous message delivery. The table below clarifies the role boundaries of each technology.

Monitoring, Diagnostics, and Distributed Tracing #

Managing event-based systems requires high visibility so we can identify where messages get stuck when problems occur.

• Cloud Logging Integration: Every time an Eventarc Trigger receives an event from a source and delivers it to a target, it writes delivery audit logs to Cloud Logging. We can use detailed log queries in the Cloud Logging Console to diagnose event delivery:

  resource.type="eventarc_trigger"
  severity>=WARNING
  

  This query displays all transmission failures, including the HTTP 4xx/5xx status codes returned by the target Cloud Run, making request rejection debugging easier.
• Distributed Tracing with Cloud Trace: Eventarc natively forwards the W3C distributed tracing header (traceparent) from the event publisher to the HTTP target. By enabling the Cloud Trace SDK in our target Cloud Run application, we can see detailed graphical timeline span visualizations showing one event’s journey from when the file was uploaded to GCS, processed by Eventarc, until finished processing by our server code — all in one unified dashboard.
• Alerting on DLQ Failures: To guarantee production data safety, we’re recommended to set up a monitoring dashboard (Cloud Monitoring alerting policy) watching the pubsub.googleapis.com/topic/send_message_operation_count metric on the DLQ topic. If messages enter that topic, the operational team receives automatic notifications via Slack or PagerDuty to investigate immediately and prevent loss of user transaction data.

Event-Driven Design Best Practices with Eventarc #

Apply the following architecture principles to guarantee the reliability of our event-based systems:

1. Must Implement Idempotency on the Target Side #

Because of the At-Least-Once delivery guarantee, redelivery of the same event due to momentary network issues at the end of a transaction is normal.

• DO: Read the id attribute in the CloudEvents metadata. Use this unique ID as a deduplication unique key in database storage or Redis caching. If that event ID was already successfully processed, immediately return HTTP 200 status without re-executing business logic.

2. Configure Least Privilege Service Accounts #

Avoid using default service accounts with broad access to deploy Eventarc triggers.

• DO: Create an isolated service account for each trigger. Only grant the roles/run.invoker role specifically on the target receiver Cloud Run container, plus the roles/eventarc.eventReceiver role to receive related event data.

3. Choose the Same Region for Event Sources and Targets #

Deploying a Cloud Storage bucket in the us-central1 region, an Eventarc trigger in europe-west3, and a target Cloud Run in asia-southeast2 causes high inter-continental network latency and bloated outbound data transfer costs (egress bandwidth).

• DO: Locate the data bucket, Eventarc trigger, and target Cloud Run compute in one same geographic region (e.g., all in asia-southeast2 Jakarta) for the best data transmission performance.

Summary #

• Google Cloud Eventarc is a serverless event router that simplifies building event-driven architecture on GCP.
• Adopts the CloudEvents v1.0 standard from CNCF to unify payload data formats from all event source types.
• Leverages Cloud Audit Logs to turn more than 130 GCP services into automatic reactive event sources.
• Provides SaaS Partner Channels integration to capture external events from Auth0, Datadog, or PagerDuty to GCP targets.
• Guarantees At-Least-Once delivery with automatic exponential retries for 24 hours and Dead Letter Queue (DLQ) support.
• Use Terraform to define triggers declaratively to ensure consistency across dev/staging/prod environments.