StreamSource docs

Introduction

StreamSource is a Kafka-based streaming platform that you deploy into your own AWS, GCP or Azure account. It bundles managed Kafka, a connector catalog, no-code and custom transforms, a schema registry, autoscaling and monitoring, and lets you control all of it from one dashboard or the ssctl CLI.

Your data stays in your account. Our control tower only sends deployment instructions and receives health metrics. Message payloads never pass through StreamSource servers.

Quickstart

This guide builds a small pipeline in the free sandbox: sample GPS devices flow through a geofence, and alerts go to a webhook. No cloud account or credit card is needed.

Install the CLI

You can do everything below from the dashboard. The CLI is shown because it's easier to copy.

$ curl -fsSL https://get.streamsource.dev/ssctl | sh
$ ssctl login
$ ssctl workspace use sandbox

Build the pipeline

1. Add a sourceThe sandbox comes with a demo-gps source that writes to sandbox.raw.gps. Check it is running:
   bashCopy

   $ ssctl component get demo-gps
   NAME      KIND    STATUS   OUT
   demo-gps  source  Running  sandbox.raw.gps

2. Add a geofence transformIt reads positions and writes an event whenever a vehicle enters or leaves the polygon.
   bashCopy

   $ ssctl component create depot-fence \
       --type transform/geofence \
       --in sandbox.raw.gps --out sandbox.evt.fence \
       --set zones=@depot.geojson

3. Send alerts to a webhookUse any URL that accepts a POST, such as a test endpoint.
   bashCopy

   $ ssctl component create alerts-hook \
       --type sink/webhook --in sandbox.evt.fence \
       --set url=https://example.com/hooks/fleet

4. Watch it runOpen Pipelines in the dashboard to see the canvas, or tail the topic:
   bashCopy

   $ ssctl topic tail sandbox.evt.fence --limit 5

Core concepts

StreamSource has a small model. Once these five ideas are clear, everything else follows.

Workspace

An isolated space for one team or project. Each workspace gets its own topics, Kafka users and permissions, so components in one workspace cannot read another workspace's data.

Components

Everything you deploy is a component, and there are three kinds:

Topics connect everything

Components never call each other. They only read and write Kafka topics. The pipeline is not stored anywhere. If component A writes topic T and component B reads topic T, the canvas draws A → B. There is no workflow engine or scheduler in the data path, so adding or replacing a component never needs a redeploy of the others.

Environment

A Kubernetes cluster running StreamSource in a specific cloud account and region, such as prod-india on AWS ap-south-1. One environment can hold many workspaces.

Sources

A source brings data from outside into a topic. Pick one from the connector catalog, fill in its settings form, and choose the topic it writes to.

Device gateways

Telematics devices such as Teltonika and iTriangle connect over TCP to a native gateway. The gateway decodes nothing on the hot path: it writes each frame as-is to a raw topic, keyed by device ID, and a decoder transform turns it into typed events.

name: teltonika-tcp
type: source/teltonika
replicas: 2
config:
  port: 5027
  codecs: [8, 8E, 16]
outputs:
  - fleet.raw.teltonika

Database change capture

PostgreSQL, MySQL, MongoDB and SQL Server sources run on Kafka Connect. They read the database log, so they add no query load to your primary. Each change's log position becomes its event ID.

Transforms

Transforms read from topics, change the data and write to topics. Most needs are covered by no-code transforms that you configure in the dashboard.

How commits work

A transform only commits its input offset after every output record, or its dead-letter record, has been stored by Kafka. If a transform emits several records from one input, each gets the ID <input id>/<n>.

Custom code

When no-code transforms aren't enough, write your own in Python, Go, Node or Java. Connect a Git repository or use the browser editor. StreamSource builds an image, versions it and deploys it like any other component.

from streamsource import transform

@transform(src="fleet.evt.position", dst="fleet.evt.alerts")
def harsh_braking(event, state):
    prev = state.get(event.key)
    state[event.key] = event.value
    if prev and prev["spd"] - event.value["spd"] > 30:
        yield {"type": "harsh_braking", **event.value}

Bring a container

Any container that reads or writes Kafka can become a component. Add a connector.yaml with a JSON Schema for its settings, and StreamSource generates the form, stores secrets in your cloud's secret manager and scales it on consumer lag.

Sinks

A sink reads topics and writes to an external system: a database, warehouse, object store, queue or HTTP endpoint.

Sinks only commit their offset after the external write succeeds. Writes are idempotent, keyed on (workspace, ce_id), so a retry after a crash updates the same row instead of adding a second one.

$ ssctl component create pg-positions \
    --type sink/postgres --in fleet.evt.position \
    --set table=positions --set batch.size=500 \
    --secret dsn=pg-prod-dsn

Topics and naming

Every topic follows one pattern: <workspace>.<stage>.<name>. The stage tells you what the topic holds and how long it keeps data.

Partitions and ordering

The record key is the ordering key, such as a device ID. Order is guaranteed only per key. Adding partitions to a keyed topic changes which partition a key lands on, so StreamSource refuses to increase partitions on a keyed topic unless you pass --allow-key-remap.

Schemas

Every evt topic has a registered schema. StreamSource ships a schema registry with a Confluent-compatible API, so standard Kafka clients work unchanged.

• Formats: Protobuf for platform schemas. JSON Schema for your own schemas, because it's easier to start with.
• Compatibility: BACKWARD by default. A change that would break existing consumers is rejected when you register it.
• Domain packs: packs such as Mobility ship their schemas, for example mobility.position.v1.

Deploy to your cloud

In production, StreamSource runs in your own cloud account. After a one-time setup, every supported operation runs from the dashboard. You only open your cloud console for billing, account settings or emergency recovery.

Setup on AWS

1. SubscribeFind StreamSource in AWS Marketplace, or start from Settings → Environments → Connect cloud.
2. Grant accessLaunch the CloudFormation stack. It creates one cross-account IAM role, limited to resources tagged streamsource:managed.
3. ProvisionPick a region. StreamSource creates an EKS cluster, Kafka, the schema registry and the agent. This takes about 25 minutes.
4. BuildThe environment appears as Healthy in the dashboard. Deploy your first source.

GCP uses a Terraform module with a service account, and Azure uses an ARM template with a managed identity. The steps are otherwise the same.

How the agent connects

The agent inside your cluster enrolls once with a single-use token, then uses short-lived certificates that rotate automatically. It opens an outbound mTLS connection to our control tower, so you don't expose any inbound port. It can only deploy workloads from an approved list.

Delivery guarantees

StreamSource delivers at least once, with stable event IDs so that downstream systems can remove duplicates. A crash or network failure can cause a repeat, never a silent gap.

Event headers

Every record carries these headers, following the CloudEvents Kafka binding. The payload stays untouched in the message value.

How event IDs are made

• Devices: source:device:sequence when the protocol has a sequence number, otherwise a hash of the frame plus the record index.
• HTTP and MQTT: the client's idempotency key when present.
• Database CDC: the log position (LSN or binlog offset).

A timestamp alone is never used as an event ID.

Monitoring and dead letters

Each component reports throughput, consumer lag, error rate and dead-letter count. Grafana dashboards and Prometheus metrics come preconfigured in every environment, and components scale out automatically when their lag grows.

Replaying dead letters

When a record can't be processed, it goes to the component's dlq topic with the error attached. Fix the cause, then replay:

$ ssctl dlq list fleet.dlq.decoder --since 1h
$ ssctl dlq replay fleet.dlq.decoder --to fleet.raw.teltonika --since 1h

Roles and access

Roles are set per workspace. Sign in with Google, Microsoft or email. Every change is recorded in an audit log that you can export.

ssctl CLI

Everything in the dashboard is available from ssctl. Commands act on the current workspace unless you pass --workspace.

REST API

The REST API is what the dashboard and ssctl use. Authenticate with a workspace API token in the Authorization: Bearer header.

$ curl https://api.streamsource.dev/v1/workspaces/fleet/components \
    -H "Authorization: Bearer $SS_TOKEN"

Limits and defaults

Troubleshooting

A component's lag keeps growing

Check its error rate first. If errors are low, it needs more replicas or the topic needs more partitions. Autoscaling can't add replicas beyond the topic's partition count.

I see duplicate rows in my database

Duplicates are expected under at-least-once delivery. Make sure your sink writes upserts keyed on ce_id, and that custom code keeps the ID when it passes events through.

My schema change was rejected

Under BACKWARD compatibility you can add optional fields but not remove or rename required ones. Create a new versioned type, such as .v2, for breaking changes.

Get support

Sandbox users can ask in the community forum. Team and Enterprise customers can raise a ticket from Help → Support in the dashboard, or reach us through the contact page.