Talk to sales
Workbook

DevOps / DataOps Engineer

You own: CI/CD, Helm, GitOps, secret rotation, image signing. You make GeneFlow ship safely + fast.

What's new for you

WasNow in GeneFlow
Bespoke deploy pipelines per serviceStandard Helm chart infrastructure/k8s/helm/genedata/ covers everything
Manual K8s manifests for MLAuto-reconciled by jobs-service deployer
Secrets in CI varsExternal Secrets Operator → AWS/GCP Secrets Manager
Grafana dashboards JSON in S3ConfigMap + sidecar auto-loads

Day-1 setup

# Helm install
helm upgrade --install genedata ./infrastructure/k8s/helm/genedata \
  --set image.tag=$IMAGE_SHA \
  --set global.agentRuntime.enabled=true \
  --set global.modelServing.enabled=true \
  --set observability.grafanaDashboards.enabled=true \
  --set observability.prometheusRules.enabled=true

# ArgoCD if you use it
kubectl apply -f infrastructure/argocd/apps/genedata.yaml

Workflow 1 — Ship a release

  1. Merge to main → Bitbucket Pipelines builds + tests + scans
  2. CI tags <service>:<sha> and pushes to registry.genedata.io
  3. ArgoCD sees the new Helm values and reconciles
  4. PromoteCriteria gates dictate when staging → prod (see PROMOTION_CRITERIA.md)

Smoke check post-deploy:

gfctl experiments list                                        # tracking works
gfctl endpoints list                                           # serving works
gfctl prompts list                                             # registry works
kubectl get pods -l app.kubernetes.io/component=model-serving  # serving pods healthy

Workflow 2 — Rotate the system token

# 1. Issue new token in AWS Secrets Manager (or your secrets backend)
aws secretsmanager update-secret --secret-id genedata/jobs-system-token \
  --secret-string "$(genedata auth issue-system-token --scope internal:ingest)"

# 2. ESO syncs into K8s Secret geneflow-runner-secrets automatically
# 3. Rolling restart the deployments that envFrom it
kubectl rollout restart deploy/jobs-service deploy/geneflow-service -n genedata

Workflow 3 — Add a Helm value with safe defaults

The pattern: every new feature ships disabled by default under a top-level toggle. Example:

global:
  modelServing:
    enabled: false              # opt-in
    deployerServiceAccount: jobs-service

Then guard the templates: {{- if .Values.global.modelServing.enabled }}.

Workflow 4 — Sign serving images (cosign)

# In CI
cosign sign --key cosign.key registry.genedata.io/geneflow/fraud-detector:v3

# Helm enforces verification via policy controller
helm upgrade ... \
  --set serving.imagePolicy=enforce \
  --set serving.imagePolicy.publicKey=$(cat cosign.pub)

Workflow 5 — Roll a new Grafana dashboard out

  1. Drop JSON into infrastructure/observability/grafana/dashboards/
  2. Reference it in templates/observability/grafana-dashboards.yaml:

``yaml data: my-new-dashboard.json: |- {{ .Files.Get "../../observability/grafana/dashboards/my-new-dashboard.json" | indent 4 }} ``

  1. helm upgrade — sidecar picks it up, appears in Grafana folder

Workflow 6 — Backups + observability are part of CI

# bitbucket-pipelines.yml step
- step:
    name: Verify backups exist
    script:
      - wal-g backup-list | grep "$(date -u +%Y-%m-%d)" || exit 1
    services: [postgres]

Common gotchas

  • Never skip pre-commit hooks (--no-verify) — they enforce migrations are forward-only, Helm renders cleanly, etc.
  • Don't hard-code namespace — use {{ .Release.Namespace }} everywhere.
  • The migration 0015_geneflow_collab.sql must run BEFORE you enable the collab editor toggle — order matters.
  • Pre-quantum SSH warning to Bitbucket is informational only; not a real failure.

Where to go next