Stream Lineage

Overview

Stream Lineage renders an interactive, left-to-right graph of every producer, topic, processor (Managed Connect, Flink SQL, Tableflow), lake table, and consumer group running in your (business_unit, stage) context. It answers the question "who produces to and consumes from this topic, and what runs in between?" without leaving the self-service portal.

The graph is reconstructed on demand by querying Confluent Cloud directly — Managed Connect, Flink SQL v1, Tableflow, the Telemetry (Metrics) API, the Schema Registry, and the Kafka REST endpoint — and mapping the results to the open OpenLineage object model. Nothing is stored: every page load rebuilds the graph from authoritative sources (with a short cache window — see Caching).

Two surfaces share the same backend:

• The full-lineage page at /ui/lineage — cluster-wide graph for the current context.
• The Stream Lineage card on the topic page (/ui/topics/{bu}/{stage}/{topic}) — a 1-hop subgraph centred on the focused topic. It calls the same endpoint with an extra ?topic=<name> query param that pushes the topic filter server-side into the Metrics queries, so the response is small and fast.

Access

Stream Lineage is writer-gated: developers, BU admins, super-admins, and platform admins can view it. Viewers and unauthenticated users are denied. The gate applies to both the page (/ui/lineage) and the backing endpoint (GET /{business_unit}/{stage}/lineage); the sidebar link is also hidden for viewers so they normally never reach the page.

How It Works

On every uncached page load the backend issues parallel async calls to seven Confluent sources and merges the results into a single OpenLineage event stream. The frontend then bootstraps a Cytoscape.js graph with a dagre layout.

When the request carries ?topic=<name> (the topic-page card always does), the dataflow source AND-filters every Metrics query on metric.topic so the API returns only rows involving that topic. Cluster-wide callers (/ui/lineage) omit the param and get the full graph.

Each source runs independently. If one fails (auth, transport, 5xx), the others still contribute — see Partial Responses.

Reading the Graph

Data flows left to right. Two node families render distinctly:

Toolbar & Interaction

Caching

Lineage responses are cached server-side for 60 seconds via aiocache, keyed by (business_unit, stage, window.start, window.end, topic). Repeated loads within the window share a single build — important because a full graph build can fan out a dozen or more API calls against Confluent. Cluster-wide (topic=None) and topic-scoped responses occupy independent cache slots so they don't poison each other. Picking a different period in the toolbar produces a new cache entry; the default key rolls forward every minute as the window's end is snapped to the current minute.

The browser also gets Cache-Control: private, max-age=60. Newly created resources (topics, connectors, consumer groups) show up on the next build after the TTL expires.

Partial Responses

Source failures never return a 5xx. The endpoint always answers 200 OK with the events it could build, and adds two response headers when one or more sources failed:

A yellow banner above the graph enumerates the failed sources so you can tell at a glance whether you are looking at the full picture or a degraded view. The most common causes:

Limitations