~atlan

Workflow Signals Platform

Problem. ~40% of all workflow failures categorized as UNKNOWN. Classification ran off a single static LLM call — no memory, no feedback loop, no confidence scoring. Accuracy: ~30%. Teams had no reliable signal for why a workflow failed.

Built. LangGraph-based agentic system with semantic memory, dynamic confidence scoring, human-in-the-loop feedback, in-flight request deduplication, and sidecar log fetching for container-level failures. Full UI: failure-category views, ownership mapping, drill-down links, and a dedicated debug and pattern investigation experience.

Impact

• UNKNOWN failures: ~40% → <1%
• Classification accuracy: ~30% → ~90%
• LLM cost: -50% via in-flight deduplication
• Alert noise: ~48/day → only actionable signals
• Warehouse, pipelines, enterprise teams onboarded with zero code changes
• Now the canonical reliability signal for the org

Quality Control Agent

Problem. 5.2% regression rate. Regressions caught post-production. Root cause: no systematic enforcement of defensive coding — missing null checks, type guards, error handling. Code reviews were fully manual.

Built. Fully agentic, AI-native PR reviewer live across multiple repositories. Evaluates every PR across five paradigms — Defensive Coding, Test Quality, Critical Bugs, Fault Tolerance, Error Handling — using a rubric-based decision matrix. Flags new regressions and old hotspots touched in the diff.

Impact

• 200+ PRs reviewed across repos
• Critical and past-RCA bugs caught in 35+ PRs during early build cycle
• Mean time to resolve a regression: 43 hours → 12 hours
• Prevents regressions pre-merge — directly targets the #1 regression pattern

Intelligent Alerting & Outage Detection

Problem. Teams learned about production failures from a next-day Mixpanel report — a 24-hour lag. Existing alerts: ~48/day, mostly noise, causing fatigue. No outage-level detection.

Built. Near-realtime, signal-driven alerting with intelligent outage detection, per-tenant deduplication, and pattern-based ownership routing. Self-serve config — any team onboards with zero code changes.

Impact

• Failure awareness: 24 hours → near-realtime
• Achieved TTR of 2 hours on a production incident — a stated team goal
• Noise eliminated — only actionable signals, covering all teams
• Zero-touch onboarding across warehouse, pipelines, enterprise

Temporal & App SDK Observability

Problem. Activity failure data trapped inside Temporal UI with short retention. Stack traces and exception context silently dropped during failures. No durable path to query what failed and why after retention expires.

Built. Designed the observability architecture, implemented activity failure logging fixes, coordinated the release. Standardized CUE/native metrics via SDK-based patterns: publish metrics, Redshift metrics, get_outputs() adoption paths. Reusable standard app teams adopt without per-app reinvention.

Impact

• Activity failures now queryable in structured logs beyond Temporal UI retention
• SDK metrics standard live — app teams adopt via SDK, no reinvention per app
• Observability recordings and release communication shipped to stakeholders

Error Code System & Failure Attribution

Problem. Failures had no structured ownership. Customer-fixable failures routed to the wrong queue. Config and non-config failures conflated. On-call signal quality poor — failure ownership buried in logs.

Built. End-to-end error code taxonomy with an inheritance-based class hierarchy, clear responsibility semantics, and ownership mapping. Failure attribution system that surfaces owner, error code, and customer-safe message directly on workflows — live across production tenants, validated on real failure scenarios.

Impact

• Failure attribution live in production
• Customer-fixable failures correctly routed — on-call noise reduced
• Structured error hierarchy: inheritance-based classes, not brittle string matching
• Clean split between config failures, platform failures, and customer action items

AE Workflow Run Observability

Problem. No centralized visibility into workflow run health, failure trends, or per-tenant reliability for AE workflows.

Built. AE Workflow Run Observability Grafana dashboard: run health, success/failure splits, failure attribution, tenant health, trend views, duration distributions, node-level breakdowns. Bumped SDK support so workflow_run attributes pass through to ClickHouse. Fixed publish circuit-breaker failures being miscounted in SLA calculations.

Impact

• Full dashboard live in production
• workflow_run attributes flowing to ClickHouse — data pipeline enabled end-to-end
• SLA calculations corrected — circuit-breaker miscounting removed

Knowledge Folders & Files

The problem behind the problem. Atlan governs structured data — tables, schemas, pipelines. But the actual meaning behind that data lives somewhere else: in PDFs, policy docs, SOPs, KPI glossaries, compliance rules. What "active customer" really means. How a metric is actually validated. This context sat in Drive, Confluence, SharePoint — outside Atlan, invisible to agents. AI answers were incomplete because the business logic layer was unreachable.

Knowledge Folders and Files was Atlan's answer: make unstructured business context a first-class governed asset — discoverable, versioned, traceable — so agents can use it the same way they use tables and schemas. Your SOPs become queryable knowledge, not a PDF your agent ignores.

Where I came in — shipping it at scale. The feature had been built but couldn't scale. Every new tenant hit a wall: deploying the knowledge service required 11 manual fixes per tenant. I designed and shipped the fix that made it work reliably on every tenant, eliminating the manual steps entirely. Knowledge Folders and Files launched to production on May 27, 2026.

Then the next gap — making agents actually use it. The knowledge layer existed, but agents inside Atlan were still blind to it. I defined the architecture for Knowledge File MCP Tools and shipped the full stack: the backend APIs and 6 new MCP tools that exposed the entire knowledge layer to every agent in the platform — list files, read content, create folders, upload new knowledge.

Then lifecycle, because create and read are the easy half. The end-to-end delete design had to span orchestration, Heracles-mediated object-store access, Atlas state and the frontend at once, and the ordering was the whole point: soft-delete in Atlas first so nobody ever sees a broken reference, delete the bytes through a service-signed URL, hard-delete only once the bytes are actually gone, and retry object-store failures asynchronously rather than failing the user’s request. That is the difference between a feature and an asset lifecycle you can trust.

Impact

• Knowledge Folders and Files live in production — May 27, 2026
• Tenant deployment blockers: 11 → 0
• Agents can now read, discover, and write to the knowledge layer via MCP
• Metric Glossary Agent and enrichment agents unblocked — customer SOPs and KPI definitions now flow into AI enrichment
• MCP toolkit expanded: 32 → 38 tools

Knowledge Synthesis Agent

Problem. Knowledge Folders and Files made unstructured content a governed asset. But somebody still had to author every glossary term, folder and data product by hand. The content lived in GitHub, Confluence, Glean and Slack; the governed asset lived in Atlan; the gap between them was a person copying things across.

Built. An agent that turns connected sources into governed Atlan outcomes rather than just showing documents to a model. The design decision that mattered was putting two very different paths behind one product surface:

• infer=false — deterministically parse structured YAML or Markdown and map it straight onto Atlan fields. No LLM call, no token cost, no chance of invention.
• infer=true — synthesise structured knowledge out of prose, with schema-constrained output.

Field mapping became configuration instead of code: a source key like term_name maps to GlossaryTerm.name, so one integration serves any customer’s document format instead of being hardcoded to the first one we saw.

I took it past the demo. Registered it against the App Platform marketplace contract, defined the generated connector and workflow configuration, wired the Automation Engine execution manifest, and shipped the frontend that goes with it — source connection flow, GitHub support, credential validation, and the infer control.

And the bridge to actually using it. Stored knowledge is not usable knowledge. The pipeline parses asynchronously, chunks, embeds and indexes, so agents retrieve the relevant passage instead of reading a whole document — with chunk-level metadata and version and content hashes so filtering, context expansion and stale-content invalidation all work at real document scale. Agents get indexed-status awareness too, so they can make an informed search-versus-read decision rather than guessing.

The evidence gate. A generated glossary term is a claim about the business, and a claim needs a source. Generated outcomes carry their source evidence and the gate is enforced, so citations are the basis for trusting a proposed knowledge change rather than decoration underneath it. This is the part I would defend hardest: an agent that writes governed assets without evidence is not a feature, it is a liability with good UX.

Impact

• Local demo → installable on any tenant and runnable from the UI, in about six weeks
• Faithful ingestion costs zero LLM tokens — inference is opt-in, not the default
• A new source format is a config change, not a code change
• One agent produces KnowledgeFiles, KnowledgeFolders and Glossary Terms

Reclassifying the Problem Before Scaling It

Problem. The first implementation was a LangGraph-style, handler-only agent. It demoed well. It was also structurally wrong for the job, and the failure modes were already visible: handler applications never created Atlas run history, blocking requests could exceed the effective Cloudflare timeout, and nothing in the execution model could support long-running, multi-tenant, incremental crawls that survive partial failure.

The call. I argued the product’s real job was not conversational reasoning. It was a production data connector whose extraction stage happens to contain an LLM — structurally the same shape as the Snowflake or Tableau connectors we already ran. That reframing settles the orchestration question: Temporal belongs on the outside as the durable workflow engine, and LangGraph belongs on the inside, contained within a synthesis activity as a local reasoning primitive.

Built. The thesis became concrete engineering requirements: explicit preflight, extraction, synthesis, transformation and publish activities; per-activity retries, heartbeats, batching and durable progress; incremental markers and content hashes that only advance after a successful publish; connector-agnostic JSONL routed through the existing Publish App so writes inherit diffing, circuit-breaking and ordering instead of hitting Atlas directly; standard credential flows and tenant-isolated task queues rather than a bespoke demo-only model. I also shipped fire-and-poll as an immediate UX bridge, so the timeout stopped hurting users while the full worker path was built.

Impact

• A demo-shaped agent became a platform that can run scheduled, resumable, multi-tenant jobs
• Run history, retries and recovery came free from Temporal instead of being reinvented per agent
• Writes inherited the Publish App’s diffing and circuit-breaking rather than going straight at Atlas
• The shape generalised — it now applies to any context agent, not just this one

The useful skill here was not the architecture. It was noticing that the question had been miscategorised before anyone had scaled the wrong answer.

Automation Engine Harness & Agent Runtime

Problem. The synthesis agent kept exposing gaps in the shared agent harness that had nothing to do with the synthesis agent. Each application was about to solve them separately — and one of the candidate solutions was going to be “read the secret over HTTP”.

Built. I fixed them in the harness, as reusable platform capability, using the agent as a proving ground rather than adding application-specific workarounds.

• Credential boundary. Added credential_id to MCPServerConfig and resolved it in-process when the harness opens an outbound MCP connection. Application code stores and passes only a reference; the harness materialises the secret for the live connection and forwards the auth headers. Per-tenant authenticated MCP servers, with no path that reads a secret over HTTP.
• Transport. Corrected the assumption that legacy SSE would do. It hung against real remote services. The harness uses Streamable HTTP, which let vendor-native MCP services and Atlan-hosted adapters share one connection contract.
• First-party auth. The tenant’s own Atlan MCP does not need a user-created connection — the engine can use its tenant platform identity for search, lineage, glossary and knowledge.
• Session durability. Reconstructed persisted transcripts from Postgres checkpoints, so a run recovers the history it needs instead of trusting in-memory state.
• Knowledge-aware agents. Added knowledge-file tools to the deep-agent harness, so any workload can read business documentation as part of its reasoning loop instead of each workflow building its own document access.
• Controlled reasoning. Added a reasoning-effort and thinking-budget switch with provider-combination validation — off by default, so no existing agent changes behaviour.
• Observability. Added run-attribute passthrough so workflow context is captured at dispatch and emitted as first-class OpenTelemetry attributes.

Impact

• Secrets stay behind the credential boundary — no plaintext hop between services
• Platform capability, not per-application workarounds — every agent inherits it
• External MCP interop verified against real remote services, not assumed
• Agents can opt into deeper reasoning without changing the default for anyone else

Turning an Agent into Governed Configuration

Problem. An agent configured by a prompt box cannot be governed. There is no way to state which sources it may touch, no way to narrow that per install, and no way to stop its behaviour changing underneath the people relying on it.

Built. Three governance axes, deliberately kept separate:

• Template — the Atlan-authored outcome and behaviour contract.
• Grant — which source types the agent may access. Widening it mints a new version.
• Scope — the subset of those sources it may actually read. Editable, and enforced at runtime rather than displayed in a UI.

Alongside it, the operational honesty work: dry-run as a real capability that executes the data plane and stops before publish, rather than a UI-only pretence; the ability to stop a run midway; preflight that reports the truth, including the case where a missing credential was being skipped while the run carried on against the remaining servers; and run semantics that distinguish success, degradation and failure instead of collapsing into “something went wrong”.

On the source side I built an adapter contract where a new hosted source self-registers and is exposed as a search/read tool pair, so adding a source changes nothing else in the system — with one credential and connection lifecycle shared by vendor-native and Atlan-hosted sources alike. Connect once, reuse across agents, with verification before storage, in-place rotation and deletion.

Impact

• Agent behaviour became versioned configuration, not an unversioned prompt
• Scope is enforced at runtime — governance the system applies, not governance the UI describes
• A malformed token now fails loudly at connect time. It used to be stored as “connected”, silently omitted by the harness, and the agent would report success having run with fewer tools than it was asked for
• Resume was treated as an end-to-end correctness problem: without a re-entrant write tail, replaying an interrupted run duplicates published assets

The Custom Agent Marketplace

Problem. Both ends of the obvious design space are wrong. A marketplace of pre-built agents does not scale, because every agent in it is something an engineer had to build first — the catalogue grows at the speed of the team, not the customer. And a prompt box does not work either: a prompt cannot be bound to sources, cannot be scoped, cannot be versioned, and cannot be trusted to write governed assets into a catalogue people depend on.

Built. The piece I led from scratch: let the user state the outcome in their own language, and have the backend assemble the agent. Somebody says what they want to achieve — turn our Confluence runbooks into governed glossary terms, keep this data product’s documentation current — and the resolution logic works out everything underneath it:

• Which skills to bind — what this agent needs to know how to do
• Which sources to connect — and the grant and scope that come with them
• Which outcome to generate — the asset type it is permitted to write

Skills are what the agent knows. MCP tools are its hands — how it actually reaches a source, reads a passage, writes an asset. The Automation Engine harness is the runtime that executes it durably, with the credential boundary, the transport contract and session recovery sitting underneath. Everything in the entries above is load-bearing here: the connector-shaped Temporal workflow, the evidence gate, the template/grant/scope axes, the dry run that stops before publish.

The Knowledge Synthesis Agent was the proof that one outcome could work end to end. This is what it became: the outcome is a parameter, not the product.

Impact

• An agent stopped being something an engineer ships and became something a user describes
• The marketplace is generative rather than a catalogue — a new outcome does not need a new codebase
• Natural-language intent lands as governed configuration: bound skills, granted sources, enforced scope, a versioned template
• Every agent it produces inherits the same durable execution, evidence gate and publish path — governance is structural, not per-agent discipline

The interesting part was never the agent. It was making the outcome the thing you configure, and letting everything else fall out of that.

Production Reliability Fixes

As of May 2026, across wisdom, heracles, Horizon, and analytics.

• Beta KEDA bypass — wisdom-bound routes unblocked in beta
• Serialized folder creation — concurrent-create race conditions removed
• Horizon AI analysis — wired analyze flows through horizon-api, fixed workflow UUID handling
• OTel resource attributes — workflow logs routing correctly to ClickHouse-backed combined views
• Knowledge router fix — preview endpoint mounted at both paths so heracles and MCP callers both resolve
• Atlas fileType gap — fixed 502s on agent-saved .md uploads