Talk to sales
Workbook

Data Platform Engineer

You own: the lower-level platform plumbing GeneFlow rides on — Postgres, S3, K8s clusters, image registry, CDC into the warehouse.

What's new for you

WasNow in GeneFlow
Every team's ML uses different storageAll artifacts under s3://bucket/<tenant_id>/...
K8s manifests created by handjobs-service deployer reconciles gf_endpoints automatically
Postgres schema sprawlAll ML state under gf_* prefix, 5 migrations
pgvector was opt-inRequired for prompt similarity search

Day-1 setup

# Postgres
psql -c "CREATE EXTENSION IF NOT EXISTS vector;"
psql -f infrastructure/postgres/migrations/0014_geneflow.sql
psql -f infrastructure/postgres/migrations/0015_geneflow_collab.sql
psql -f infrastructure/postgres/migrations/0016_geneflow_serving.sql

# Helm
helm upgrade --install genedata ./infrastructure/k8s/helm/genedata \
  --set global.agentRuntime.enabled=true \
  --set global.modelServing.enabled=true \
  --set observability.grafanaDashboards.enabled=true

Workflow 1 — Wire CDC to your warehouse

Use Debezium / Fivetran / Airbyte to replicate gf_* tables. Recommended exclusions: gf_inference_logs (high volume — replicate hourly rollup instead).

Minimum CDC config:

tables:
  - gf_experiments, gf_runs, gf_metrics, gf_params, gf_tags
  - gf_artifacts
  - gf_registered_models, gf_model_versions, gf_stage_transitions
  - gf_prompts, gf_prompt_versions
  - gf_endpoints, gf_endpoint_revisions
  - gf_drift_baselines, gf_drift_alerts
  - genedata_audit_log
exclude:
  - gf_inference_logs       # use rollup instead
  - gf_prompt_collab_sessions

Workflow 2 — Storage layout

s3://genedata-geneflow-{region}/
  {tenant_id}/
    experiments/
      {experiment_id}/
        {run_id}/artifacts/
          model/model.pkl
          logs/...
    prompt-snapshots/             # collab editor snapshots
    inference-logs/                # cold archive

Lifecycle: 90 days warm → glacier; 7 year retention. Versioning ON. KMS-encrypted.

Workflow 3 — Postgres indexing / scaling

Every gf_ table has (tenant_id, ...) indexes. Watch out for:

  • gf_metrics — hot table. Per-step rows; > 100M is normal. Use BRIN if you scan time ranges.
  • gf_inference_logs — hottest. Consider partitioning by tenant (DECLARATIVE) once > 50M rows.
  • gf_prompt_versions.embedding — pgvector ivfflat index, recreate after big bulk loads.
-- Quick health check
SELECT relname, n_live_tup, pg_size_pretty(pg_relation_size(relid))
FROM pg_stat_user_tables WHERE relname LIKE 'gf_%'
ORDER BY n_live_tup DESC;

Workflow 4 — Container registry

ImageBuilt fromUsed by
registry.genedata.io/geneflow-runner:<sha>services/jobs-service/Dockerfile.runnerK8s Job pods
registry.genedata.io/geneflow/<model>:v<n>per-model image (built by your CI)Serving pods
registry.genedata.io/geneflow-service:<sha>services/geneflow-service/DockerfileExtracted service
registry.genedata.io/websocket-gateway:<sha>services/websocket-gateway/DockerfileCollab gateway

Customer image-pull secrets are provisioned in the Helm values (one Secret reference per tenant if you support per-tenant registries).

Workflow 5 — Backup + DR

  • Postgres: WAL-G to S3 every 15 min, full snapshot daily, 30-day retention
  • Artifact bucket: cross-region replication
  • Audit log: ship to write-once S3 bucket (Object Lock = compliance mode)
  • DR test: at least quarterly, restore staging from prod backups

See docs/runbooks/01-disaster-recovery.md.

Workflow 6 — Cost guardrails on the platform itself

Run this nightly:

SELECT 'gf_inference_logs' AS tbl, pg_size_pretty(pg_total_relation_size('gf_inference_logs')) AS size
UNION ALL SELECT 'gf_metrics', pg_size_pretty(pg_total_relation_size('gf_metrics'))
ORDER BY 1;

Spikes mean a tenant's serving pod isn't sampling correctly — flag MLOps.

Where to go next