From source to destination

Follow a sample event through ingestion, transformation and delivery.

StreamSource guidesUpdated 12 October 2026

Map the event path

Our example reads vehicle telemetry, normalizes it and delivers an analytics-ready record. Each step has one responsibility.

MQTT source→Normalize event→PostgreSQL sink

This is an illustrative pipeline contract. It is not an importable deployment manifest or a claim that a live workspace has been created.

Configure the source

Identify the broker address, topic filter and authentication method. Use a dedicated test topic while validating your pipeline.

Setting	Example	Review
Topic filter	vehicles/demo/telemetry	Make sure it includes only the intended devices.
Transport	TLS connection	Validate the server certificate and hostname.
Authentication	Secret reference	Store credentials in a secret manager, not in event payloads.
Output topic	fleet.raw.telemetry	Keep the original event for debugging and replay planning.

See the source catalogue for other ingestion options.

Normalize and validate

Define the field mapping before writing custom code. Missing or invalid required fields should have a visible outcome, rather than quietly disappearing.

Input	Output	Rule
event_id	eventId	Required string; preserve across retries.
device_id	deviceId	Required string; retain the source identifier.
speed	speedKph	Convert to a finite, non-negative number.
recorded_at	recordedAt	Validate a timestamp and preserve its timezone.

Expected output for the sample event:

JSON example

{
  "eventId": "evt-demo-001",
  "deviceId": "vehicle-101",
  "speedKph": 42.5,
  "recordedAt": "2026-10-12T08:00:00Z"
}

Use built-in transforms where possible. Add custom code when you need domain-specific validation or enrichment.

Choose delivery behavior

For a database destination, decide how events map to rows and how repeated deliveries are handled.

Use eventId as a unique key for this example.
Decide whether a repeated key is ignored, replaced or treated as an error.
Map timestamp and numeric fields to appropriate destination types.
Review retry and batching behavior for the connector you choose.

Delivery behavior depends on the connector and destination. Do not assume exactly-once delivery without testing retries and checking the integration’s guarantees.

Test the edge cases

Test	Expected result
A valid sample event	One correctly mapped destination record.
The same event delivered twice	The configured duplicate-handling policy is applied.
A non-numeric speed	A visible validation error; no invalid numeric record.
A missing device identifier	A rejected or quarantined event with an explanation.
Destination temporarily unavailable	Retries and failures remain visible; recovery is verified.

Write down expected outcomes before running each test. Keep a copy of the input and output so changes can be compared.

Follow the pipeline in operation

Watch more than message counts. Compare the source rate, processing errors, destination writes and consumer lag to identify which step needs attention.

Growing lag: review processing capacity and destination latency.
Repeated validation failures: check upstream field changes.
Write failures: review destination access and schema compatibility.

Continue with the cloud deployment guide when your sample pipeline behaves as expected.