01
Visual + Code
Drag-and-drop pipeline builder for analysts, native SQL/Python/Scala for engineers — the same execution graph runs both.
GeneFlow · Data engineeringGENEDATA / 01
Build the flow.Keep the context.
Connect your sources, shape your data, and orchestrate your pipelines in one workspace. Every transformation keeps its lineage.
Shared context · Lineage · Governance
Connected
Your sources
Database CDC
Streaming events
Files & APIs
Connected intelligenceGeneFlow pipeline
Governance
Business impactTrusted data products
Shared contextLineageGovernance
02
Streaming + Batch
Unified compute for real-time CDC, micro-batch, and overnight backfills. One pipeline, one set of metrics, one bill.
03
Built-in Observability
Per-task lineage, data-quality assertions, and SLA tracking — alerts fire before downstream dashboards go stale.
The GeneFlow Family
Four members, and the two products they hand off to.
GeneFlow is a family rather than a single tool. Each member is independently adoptable and shares the same catalog, control plane, and lineage — and the tables it produces are governed by GeneCatalog and consumed by Cortex SQL and Cortex AI without an export step in between.
GeneFlow Connect
Managed ingestion from 33 live connectors — databases via log-based CDC, streaming topics, object stores, and SaaS applications — with schema inferred and catalogued on first read.
Learn moreGeneFlow Pipelines
Declarative batch and streaming transforms with incremental materialisation, so a one-row upstream change rebuilds one partition rather than the whole table.
Learn moreGeneFlow Jobs
Dependency-aware orchestration across every pipeline, with self-healing retries from the last known-good checkpoint and per-task SLA tracking from the first run.
Learn moreGeneFlow Designer
Visual authoring for analysts. Drag sources and transforms onto a canvas; the result compiles to the same execution graph an engineer would write in SQL.
Learn moreGeneCatalog
The governance product GeneFlow writes into. Classification and column-level lineage are emitted as transforms execute, so every pipeline is covered without separate instrumentation.
Learn moreCortex AI
The intelligence product GeneFlow feeds. Experiments, feature definitions, and managed endpoints read the governed tables your pipelines produce — no export, no second copy.
Learn moreSee It Working
Five workloads, one surface.
Build visually or in SQL, orchestrate, govern, and monitor — without leaving the platform or switching vendors. Move between the tabs to see the surface each workload actually uses.
Build
Pipelines / revenue_dailyValidateDeploy
Sources
Postgres CDC
Kafka topic
S3 / GCS
Salesforce
REST endpoint
Transforms
Join
Aggregate
Dedupe
Assert
join_sessions
Type
Stream join
Window
5 minutes
Watermark
30s
Output
Append
Assertions
3 active
Compose a pipeline visually — or write it in SQL
GeneFlow Designer lets analysts drag sources and transforms onto a canvas; engineers express the same pipeline in SQL, Python, or Scala. Both compile to one execution graph, so a visual pipeline and hand-written code sit in the same DAG with the same lineage.
Explore the pipeline builder- GeneFlow Connect supplies 33 live connectors from a catalogue of 84, catalogued on first read
- Windowed stream joins alongside batch transforms in one graph
- Assertions attach to any node and halt publication on failure
Develop
Develop / revenue_daily.sqlFormatRun
-- Incremental: only changed partitions recompute
CREATE MATERIALIZED VIEW gold.revenue_daily
PARTITIONED BY (order_date)
WITH (freshness = '5 minutes') AS
SELECT
o.order_date,
o.region,
SUM(o.amount) AS gross_revenue,
COUNT(DISTINCT o.customer_id) AS buyers
FROM silver.orders o
WHERE o.status = 'settled'
GROUP BY 1, 2;
-- Contract: halts publication on failure
ASSERT gross_revenue >= 0 ON VIOLATION FAIL;
Succeeded1,284 rows · 340ms · 2 partitions rebuilt
| order_date | region | gross_revenue |
|---|---|---|
| 2026-08-01 | EMEA | 1,284,410 |
| 2026-08-01 | AMER | 2,910,338 |
| 2026-08-01 | APAC | 884,102 |
| 2026-07-31 | EMEA | 1,190,776 |
| 2026-07-31 | AMER | 2,744,015 |
Declarative SQL with incremental materialisation
Declare the result you want and a freshness target; the engine works out what to recompute. A one-row upstream change rebuilds one partition, not the whole table.
Read the transformation API- Only changed partitions recompute — no full-table rebuilds
- Data contracts expressed inline as assertions
- Results preview against live data before deploy
Orchestrate
Orchestration / Run historyFilterNew job
| Run | Pipeline | Status | Duration | Rows | Timeline |
|---|---|---|---|---|---|
| run_8f21 | revenue_daily | Succeeded | 4m 12s | 18.2M | |
| run_8f20 | customer_360 | Running | 2m 03s | 6.1M | |
| run_8f19 | inventory_sync | Failed | 0m 48s | — | |
| run_8f18 | clickstream_raw | Succeeded | 11m 30s | 241M | |
| run_8f17 | fx_rates | Queued | — | — |
Dependency-aware scheduling with self-healing retries
GeneFlow Jobs runs pipelines on cron, event triggers, or continuously. A late-arriving source delays its dependants rather than letting them publish stale output, and transient failures retry from the last known-good checkpoint.
See orchestration options- Cross-pipeline dependencies resolved in one scheduler
- Retries resume from checkpoint rather than restarting the DAG
- Per-task SLA tracking from the first run
Govern
Catalog / silver.ordersCertifyRequest access
| Column | Type | Classification | Null % |
|---|---|---|---|
| order_id | bigint | — | 0.00 |
| customer_email | string | PII | 0.01 |
| card_last4 | string | PCI | 2.40 |
| amount | decimal(12,2) | — | 0.00 |
| region | string | — | 0.00 |
Lineage
↑postgres.orders
↑kafka.clicks
•silver.orders
↓gold.revenue_daily
↓exec_dashboard
Owner
data-platform@
Catalogued and classified on write
Every dataset is registered as it lands, with column contents profiled for PII and PCI before anyone queries them. Lineage is emitted by the engine as transforms execute, so impact analysis is a query rather than an archaeology project.
Explore governance- Column-level lineage, upstream and downstream
- Content-based classification, not column-name guessing
- Certification workflow with a named owner per asset
Monitor
Observability / orders_factAlertsLineage
Freshness
4m ago
Volume
−61%
Null rate
0.02%
Schema
Stable
Volume anomaly · orders_fact
Row count 61% below learned baseline. 3 downstream assets affected — revenue_daily, exec_dashboard, churn_model.
Freshness and volume alerts before a dashboard goes stale
Signals are emitted for every table without manual instrumentation, and compared against a learned baseline rather than a static threshold. Alerts name every downstream asset affected, so triage starts with known impact.
Explore observability- Learned baselines per signal, including seasonality
- Lineage-aware alerts naming affected downstream assets
- Full-fidelity replay of any past window
Execution Graph
One DAG, from change capture to serving layer.
Every pipeline compiles to a single execution graph. Streaming and batch tasks sit in the same DAG, share the same lineage, and are scheduled by the same dependency resolver — so a late-arriving source delays its dependants instead of silently publishing stale output.
How It Works
Connect, model, validate, ship.
The same four steps whether you are moving one table or rebuilding a warehouse — and each step is versioned, so a pipeline can be reviewed and rolled back like any other code change.
Connect the source
Pick from 33 live connectors, or point at any JDBC, REST, or object-store endpoint. Schema is inferred on first read and registered in the catalog.
Model the transform
Compose in the visual builder or write SQL, Python, or Scala. Both produce the same execution graph, so teams can mix approaches in one pipeline.
Assert the contract
Attach freshness, volume, uniqueness, and referential assertions. A failing gate halts publication rather than propagating bad data downstream.
Schedule and observe
Deploy on cron, event triggers, or continuous streaming. Per-task metrics, lineage, and SLA tracking are on from the first run.
Capabilities
What the engine handles for you.
Pipeline work is mostly the unglamorous parts — backfills, schema drift, retries, and late data. These are handled in the runtime rather than left as an exercise for each team.
Change data capture
Log-based CDC for Postgres, MySQL, Oracle, and SQL Server with transactional ordering preserved end to end.
Incremental transforms
Only changed partitions recompute. A one-row upstream change does not trigger a full-table rebuild.
Schema-drift handling
New columns propagate automatically; breaking changes raise a review rather than failing at 3am.
Backfill and replay
Re-run any window against current logic with full idempotency, without hand-written catch-up jobs.
Self-healing retries
Transient failures retry with backoff against the last known-good checkpoint instead of restarting the DAG.
Column-level lineage
Every field traces to its sources through each transform, so impact analysis is a query rather than an archaeology project.
In Practice
Who this is for.
The same engine serves very different jobs — the difference is which surface each role works in.
Data Engineer
Retire the orchestration zoo
Replace separate ingestion, transformation, and scheduling tools with one graph. Dependencies resolve across what used to be three systems and three on-call rotations.
One runtime, one alerting surface, one bill.
Data Analyst
Ship a pipeline without filing a ticket
Build in the visual editor against governed sources. Output lands in the catalog with lineage attached, so it is reviewable rather than a shadow dataset.
Self-service that governance can actually approve.
Platform / SRE
Know before the business does
Freshness and volume assertions fire on the pipeline, not on the dashboard. Incidents surface at the failing task with lineage showing exactly what is downstream.
Detection ahead of the first user complaint.
What Changes
What changes when the pipeline stops leaving the platform.
Assembled stacks lose context at every hop between tools. A single execution graph keeps it.
| Dimension | Before Genedata | With Genedata |
|---|---|---|
| Tooling | Separate ingestion, transformation, and orchestration vendors | One engine covering the full lifecycle |
| Lineage | Breaks at each tool boundary; impact analysis is manual | Column-level and continuous across every task |
| Backfills | Bespoke catch-up scripts, frequently non-idempotent | Replay any window against current logic |
| Bad data | Detected downstream, in a dashboard, by a stakeholder | Quality gate halts publication at the failing task |
| Schema drift | Overnight failure, then a manual reconciliation | Additive changes propagate; breaking ones raise a review |
33 live · 84 cataloguedConnectors
CDC + micro-batchStream Mode
UnlimitedPipeline DAGs
Per-taskSLA Tracking
The next step
From source to production, on one engine.
Stop paying three vendors for ingestion, transformation, and orchestration. GeneFlow covers the full lifecycle — and because it's the same engine running your warehouse and AI, your pipelines never leave the platform.
Interactive walkthrough
From handoff to contract
Six scenes on GeneFlow: watch a pipeline become a contract between steps, break it on purpose, heal it with a review, price a backfill before it runs, and move production by changing a pointer.
Open the walkthrough7 min · 6 scenes
Also related
FAQ
GeneFlow, answered.
The questions that come up most often in evaluations. If yours isn't here, the architecture guide goes deeper.
What is GeneFlow?
GeneFlow is the data engineering product line on the Genedata platform. It covers ingestion, transformation, and orchestration as one product rather than three tools — GeneFlow Connect brings data in, GeneFlow Pipelines transforms it, and GeneFlow Jobs schedules the whole graph. Because they share a catalog and control plane, lineage and governance are continuous rather than reassembled at each boundary.
How is GeneFlow Connect different from a standard connector library?
Most connector libraries move bytes and stop there. GeneFlow Connect registers each dataset in the catalog on first read, infers and versions its schema, and emits lineage from the moment data lands — so a newly connected source is governed before anyone queries it, rather than after someone notices it isn't.
Do I have to choose between the visual builder and code?
No. GeneFlow Designer and hand-written SQL, Python, or Scala compile to the same execution graph, so one pipeline can contain both. An analyst's visual transform and an engineer's code sit in the same DAG, share the same lineage, and are scheduled by the same resolver.
What happens when a pipeline fails partway through?
Transient failures retry with backoff from the last known-good checkpoint rather than restarting the DAG. If a data-quality assertion fails, publication halts at that task instead of propagating bad data downstream, and the alert names every downstream asset that would have been affected.
How does GeneFlow handle schema drift?
Additive changes — a new column upstream — propagate automatically. Breaking changes raise a review rather than failing overnight, and because every schema version is retained you can answer what a table looked like on the date a given report was produced.
Can I run GeneFlow in my own cloud account?
Yes. GeneFlow deploys in the Genedata enterprise cloud, inside your own AWS, Azure, or GCP account, or into a jurisdiction-bound sovereign region. The APIs, governance model, and security posture are identical across all three; only the perimeter changes.
Related content
Architecture
GeneFlow Reference Architecture
Execution model, connector catalog, and the transformation API in full.
RunbookProduction Pipeline Runbook
Operational patterns for backfills, replay, and incident response.
CatalogGeneFlow Connect Sources
The full list of managed sources with CDC and streaming support.
StatusPlatform Status
Live health and incident history across every Genedata region.
Take the next step
Start building on GeneFlow.
Walk through a live pipeline with an engineer, or go straight to the architecture guide and connector catalog.