TL;DR
A buildable spec: one warehouse, one heartbeat scheduler, one touchpoint table.
- Numbered, tool-agnostic requirements for each of the three components, with acceptance criteria a build can be tested against.
- Our stack — Postgres, FastAPI, Redis, Cloud Scheduler + a tiny workflow, IAP — appears as a reference implementation; equivalents work fine.
- Success criterion: after setup, adding an automation is a one-file change with no infrastructure work.
1. Summary
Build an internal platform that lets a GTM team run scheduled automations (scoring, CRM sync, Slack alerts, meeting prep, transcript processing) as small units of code on top of persistent data the company owns. Success criterion: after initial setup, adding an automation is a one-file change by one person (or an AI agent) with no infrastructure work, and each new automation is cheaper to build than the last.
Non-goals: a DAG orchestrator (Airflow-class), a customer-facing product, real-time (<1 min) event processing, replacing the CRM or sequencer.
2. System overview

3. Component 1 — persistent data warehouse
R1.1 All computed and purchased GTM data (scores, rankings, signals, notes, payment history) persists in a queryable store the team owns.
R1.2 The warehouse is the source of truth; SaaS tools are destinations. Data pushed to the CRM (e.g. score fields reps sort by) is a copy — the warehouse holds the asset.
R1.3 External signals are pulled into the warehouse on a schedule, keyed by company domain, once — never re-enriched per campaign or per list.
R1.4 Scheduled jobs refresh the data continuously (scores nightly, rankings daily), so every downstream question reflects current reality rather than a decaying snapshot.
Warehouse choice. Snowflake, Databricks, or BigQuery all qualify; for GTM-sized data plain Postgres is usually sufficient. Using the CRM itself as the warehouse (Salesforce/HubSpot custom objects) is clunky and not ideal but acceptable if unavoidable. Prefer wherever other key metrics — especially product usage — already live, so scores can join against them directly.
4. Component 2 — automation layer
4.1 Authoring
R2.1 An automation is a small unit of code that declares its own cadence (e.g. five-minutely / hourly / daily / weekly) alongside its logic. No per-job entries in any external scheduler; cadence is reviewed and deployed with the code.
R2.2 Automations run inside one small web service with shared connections to the warehouse, CRM, Slack, sequencer, and billing.
4.2 Scheduling semantics
R2.3 A single periodic trigger (every ~5 minutes) asks the service "which jobs are due?" and invokes exactly those. The trigger knows nothing about cadences.
R2.4 Mutual exclusion: a job can never run twice concurrently — including a manual trigger colliding with the schedule.
R2.5 Self-pacing: after a run, a job is not due again until its interval elapses, measured from the run's start (long runs must not drift the schedule).
R2.6 Crash recovery: if the service dies mid-run, the job becomes due again within a bounded window (≤30 minutes) regardless of its cadence.
R2.7 Missed triggers self-heal: the next trigger picks up everything due.
R2.8 One job's failure must not affect any other job, and failed runs must be alertable.
R2.9 The claim/lock state lives in a transactional store: atomic acquire, millisecond reads, no cost per query. A Postgres warehouse can host it as a job_claims(path, status, expires_at) table (zero extra infrastructure). Analytical warehouses (Snowflake/BigQuery/Databricks) must not host it — wrong locking semantics, seconds of latency, compute billed per poll; pair them with a small Redis or Postgres.
4.3 Operation
R2.10 Anyone on the team can trigger a job manually ("Run now"). Manual runs bypass the cooldown but never the running-lock (R2.4).
R2.11 Every run is recorded — start/end, elapsed, status, error, and the job's returned stats. Every job returns a small stats summary (e.g. accounts_scored: 41210) so "succeeded but did nothing" is visible.
R2.12 One dashboard lists every job, its cadence, next run, and recent history. Jobs feeding business reporting also write to a durable run-log table in the warehouse.
R2.13 The platform sits behind the company SSO perimeter (e.g. IAP, Cloudflare Access); the trigger authenticates with a machine identity. No in-app auth code.
5. Component 3 — touchpoint log
Not new infrastructure: one more warehouse table plus a convention.
R3.1 One table records every outbound touch across all channels — sequence emails, alert-driven outreach, event invites, logged human emails: touchpoints(email, domain, channel, source, summary, occurred_at, sent_by).
R3.2 Every automation that sends anything writes a row on send. Sequencer webhooks and a periodic CRM-activity pull land human/tool touches in the same table.
R3.3 A single pre-send guard — ok_to_contact(email, policy) — is called by every sending automation and returns false if the person was touched within the policy window, across all channels. Cadence policy is one piece of code, not settings scattered across tools.
R3.4 Rollout order: writers and backfill first, guard second — enforcing the guard on incomplete history vetoes valid sends.
R3.5 Because the log lives in the warehouse, measurement is a join: touches → replies → meetings.
6. Conventions for every automation
- R4.1 Idempotent refreshes: recompute-style jobs replace their output wholesale in one transaction; running twice is harmless.
- R4.2 Incremental jobs resume from a high-water mark in the destination table; missed runs catch up automatically.
- R4.3 Event-driven alerts use a marker pattern: the event source (webhook) stamps
ready_to_send; a five-minute job sends the enriched alert and flips the marker. At-least-once by design — a duplicate alert beats a silently lost one. - R4.4 Time windows (weekday mornings, business hours) are enforced inside the job, not by the scheduler.
- R4.5 Ordering between jobs is soft: register in dependency order, tolerate reading data one cycle old; daily idempotent refreshes self-heal.
7. Acceptance criteria
| Scenario | Required behavior |
|---|---|
| Job's next window opens mid-run | No second run starts; schedule doesn't drift |
| Service killed mid-run | Job re-runs within ≤30 min |
| One job throws | Recorded + alertable; all other jobs unaffected |
| Claim store wiped | Every job re-runs once; idempotency makes it harmless |
| Trigger missed | Next trigger runs everything due; no data gap |
| Manual + scheduled collide | Exactly one executes |
| Job succeeds with zero output | Visible on dashboard via stats summary |
8. Rollout plan
- Warehouse + first scores — schema plus two or three scoring jobs pulling from your data provider (ours: Sumble) — component 1.
- Automation layer — claim wrapper, due-discovery, driver, SSO perimeter, dashboard (component 2; ~a day in the reference implementation).
- Seed automations — establish the R4 conventions; later automations (by humans or AI agents) pattern-match the seeds.
- Touchpoint log — table and writers, then the guard (component 3, no new infra).
Appendix — reference implementation sketches
An automation (complete):
@router.post("/run-account-scoring", schedule=Schedule.DAILY)
async def run_account_scoring() -> dict[str, int]:
df = await score_accounts(warehouse, sumble_query_fn)
return {"accounts_scored": len(df)}The two-phase claim (Redis flavor; a Postgres job_claims table works identically):
acquired = await redis.set(claim_key, "running", nx=True, ex=30 * 60)
if not acquired:
return {"status": "skipped"}
started = time.monotonic()
try:
return await endpoint(*args, **kwargs)
finally:
ttl = max(1, int(schedule.seconds_until_next_run() - (time.monotonic() - started)))
await redis.set(claim_key, "done", ex=ttl)Due discovery — a job is due exactly when its claim is absent:
values = await client.mget([_claim_key(p) for p in paths])
return {"paths": [p for p, v in zip(paths, values) if v is None]}The driver (GCP Workflow, fired by Cloud Scheduler */5 * * * *; any cron + curl works):
main:
steps:
- getManifest:
call: http.get
args:
url: https://gtm.example.com/api/scheduled/due
auth: {type: OIDC, audience: "<IAP OAuth client ID>"}
result: manifest
- callRoutes:
parallel:
for:
value: path
in: ${manifest.body.paths}
steps:
- post:
call: http.post
args:
url: ${"https://gtm.example.com" + path}
auth: {type: OIDC, audience: "<IAP OAuth client ID>"}Production hardening: try/except each POST so one failure never blocks the rest; raise at the end so monitoring alerts. Durable run log: gtm_run_log(system, run_date, completed_at, row_count, elapsed_ms, status, detail).
