~atlan

Workflow Signals Platform

Problem. A large share of workflow failures landed in an UNKNOWN bucket. Classification was a single static LLM call — no memory, no feedback loop, no notion of confidence. Most labels were wrong, so nobody trusted any of them, and teams had no reliable signal for why a workflow failed.

Built. A LangGraph-based agentic classifier with semantic memory, dynamic confidence scoring, human-in-the-loop feedback, in-flight request deduplication, and sidecar log fetching for container-level failures. A full UI on top: failure-category views, ownership mapping, drill-down links, and a dedicated debug and pattern-investigation experience.

Challenge. Failures arrive in bursts — one root cause, many identical failures at the same moment. A naive classifier pays for the same answer over and over. Deduplicating requests while they are still in flight turned each burst into one call.

Impact

• The UNKNOWN bucket went from a large share of failures → almost none
• Labels went from mostly wrong → mostly right
• LLM spend cut substantially through in-flight deduplication
• Dozens of noisy alerts a day → only actionable signals
• Teams onboard self-serve, with zero code changes
• Became the canonical reliability signal for the org

Quality Control Agent

Problem. Regressions were reaching production and being caught there. The dominant pattern was not exotic: missing null checks, type guards and error handling. Review was entirely manual, so whether any of it got caught depended on who was reviewing that day.

Built. A fully agentic, AI-native PR reviewer, live across multiple repositories. It evaluates every PR across five paradigms — Defensive Coding, Test Quality, Critical Bugs, Fault Tolerance, Error Handling — with a rubric-based decision matrix, and flags both new regressions and old hotspots the diff happens to touch.

Impact

• Hundreds of PRs reviewed across repos
• Critical bugs — including ones a past RCA had already named — caught before merge
• Time to resolve a regression: days → hours
• Aimed squarely at the most common regression pattern, not at style

Intelligent Alerting & Outage Detection

Problem. Teams learned about production failures from a next-day report. The alerts that did exist fired dozens of times a day, were mostly noise, and had trained people to ignore them. Nothing detected an outage as an outage.

Built. Near-realtime, signal-driven alerting with outage detection, per-tenant deduplication, and pattern-based ownership routing. Self-serve configuration — any team onboards with zero code changes.

Impact

• Failure awareness: next day → near-realtime
• Hit the team’s time-to-resolve goal on a real production incident
• Noise removed — only actionable signals, covering every team
• Zero-touch onboarding

Temporal & App SDK Observability

Problem. Activity failure data was trapped inside the Temporal UI with short retention. Stack traces and exception context were silently dropped during failures. Once retention expired there was no durable way to ask what failed, or why.

Built. Designed the observability architecture, implemented the activity failure-logging fixes and coordinated the release. Standardised how apps publish metrics through SDK-level patterns, so an app team adopts one way of doing it instead of inventing its own. Then rolled it across the connector apps and packaged the how-to as a reusable AI skill, so the next app adds metrics by asking for them.

Impact

• Activity failures queryable in structured logs long after the workflow UI has forgotten them
• One SDK metrics standard — adopted, not reinvented, per app
• Recordings and release communication shipped to stakeholders

Error Code System & Failure Attribution

Problem. Failures had no structured ownership. Customer-fixable failures were routed to the wrong queue. Configuration and platform failures were conflated, and who owned a failure was buried in logs — so on-call signal quality was poor.

Built. An end-to-end error code taxonomy with an inheritance-based class hierarchy, clear responsibility semantics and ownership mapping. On top of it, a failure attribution system that surfaces the owner, the error code and a customer-safe message directly on the workflow — validated against real failure scenarios.

Impact

• Failure attribution live in production
• Customer-fixable failures routed correctly — on-call noise down
• Inheritance-based error classes, not brittle string matching
• A clean split between configuration failures, platform failures and customer action items

Workflow Run Observability

Problem. There was no single place to see workflow run health, failure trends, or per-tenant reliability.

Built. A Grafana dashboard for run health, success/failure splits, failure attribution, tenant health, trends, duration distributions and node-level breakdowns. Extended the SDK so run attributes flow through to ClickHouse, and fixed circuit-breaker trips being miscounted as failures in SLA calculations.

Impact

• Dashboard live in production
• Run attributes flowing to ClickHouse — the data path works end to end
• SLA numbers corrected — circuit-breaker miscounting removed

What act i taught me: most of this was not a tooling problem. It was an ownership problem. A failure nobody owns is noise, however well it is labelled.

Knowledge Folders & Files

The problem behind the problem. Atlan governs structured data — tables, schemas, pipelines. But the meaning behind that data lives somewhere else: in PDFs, policy docs, SOPs, KPI glossaries, compliance rules. What "active customer" really means. How a metric is actually validated. That context sat in wikis and shared drives, outside the catalog and invisible to agents, so AI answers were incomplete because the business-logic layer was unreachable.

Knowledge Folders and Files makes unstructured business context a first-class governed asset — discoverable, versioned, traceable — so agents can use it the same way they use tables and schemas. Your SOPs become queryable knowledge, not a PDF your agent ignores.

Where I came in — shipping it at scale. The feature had been built but could not scale. Deploying the knowledge service to a new tenant meant working through a runbook of manual fixes — provisioning failures, missing credentials, routing breakdowns. A GA blocker dressed as a runbook. I designed and shipped the change that made it deploy cleanly on every tenant, with no manual steps, and the feature went GA.

Then the next gap — making agents actually use it. The knowledge layer existed, but agents were still blind to it. I defined the architecture for knowledge-file MCP tools and shipped the full stack: the backend APIs and the MCP tools that expose the knowledge layer to every agent on the platform — list files, read content, create folders, upload new knowledge — followed by semantic indexing so agents retrieve passages rather than whole documents.

Then lifecycle, because create and read are the easy half. The end-to-end delete design had to span orchestration, object-store access, the metadata store and the frontend at once, and the ordering was the whole point: soft-delete in the catalog first so nobody ever sees a broken reference, delete the bytes through a service-signed URL, hard-delete only once the bytes are actually gone, and retry object-store failures asynchronously rather than failing the user’s request. That is the difference between a feature and an asset lifecycle you can trust.

Impact

• Knowledge Folders and Files generally available in production
• Per-tenant manual deployment fixes: a runbook → zero
• Agents can read, discover and write to the knowledge layer via MCP
• Glossary and enrichment agents unblocked — customer SOPs and KPI definitions now flow into AI enrichment

Knowledge Synthesis Agent

Problem. Knowledge Folders and Files made unstructured content a governed asset. But somebody still had to author every glossary term, folder and data product by hand. The content lived in code hosts, wikis, search tools and chat; the governed asset lived in the catalog; the gap between them was a person copying things across.

The call. An agent that turns connected sources into governed outcomes, not one that shows documents to a model — with two very different paths behind one product surface:

• Faithful mode — deterministically parse structured YAML or Markdown and map it straight onto catalog fields. No LLM call, no token cost, no chance of invention.
• Inference mode — synthesise structured knowledge out of prose, with schema-constrained output.

Built. Field mapping became configuration instead of code: a source key such as term_name maps onto a glossary term’s name, so one integration serves any customer’s document format instead of being hardcoded to the first one we saw. I took it past the demo — registered it against the platform’s app contract, defined the generated connector and workflow configuration, wired the agent runtime’s execution manifest, and shipped the frontend with it: source connection flow, credential validation and the inference control.

The bridge to actually using it. Stored knowledge is not usable knowledge. The pipeline parses asynchronously, chunks, embeds and indexes, so agents retrieve the relevant passage instead of reading a whole document — with chunk-level metadata, version and content hashes so filtering, context expansion and stale-content invalidation all work at real document scale.

The evidence gate. A generated glossary term is a claim about the business, and a claim needs a source. Generated outcomes carry their source evidence and the gate is enforced, so citations are the basis for trusting a proposed change rather than decoration underneath it. This is the part I would defend hardest: an agent that writes governed assets without evidence is not a feature, it is a liability with good UX.

Impact

• Local demo → installable on any tenant and runnable from the UI, in about six weeks
• Faithful ingestion costs zero LLM tokens — inference is opt-in, not the default
• A new source format is a config change, not a code change
• One agent produces knowledge files, knowledge folders and glossary terms

Reclassifying the Problem Before Scaling It

Problem. The first implementation was a LangGraph-style, handler-only agent. It demoed well. It was also structurally wrong for the job, and the failure modes were already visible: handler apps never created platform run history, blocking requests could outlive the edge proxy’s timeout, and nothing in the execution model could support long-running, multi-tenant, incremental crawls that survive partial failure.

The call. I argued the product’s real job was not conversational reasoning. It was a production data connector whose extraction stage happens to contain an LLM — the same shape as the warehouse and BI connectors that were already running. That reframing settles the orchestration question: Temporal belongs on the outside as the durable workflow engine, and LangGraph belongs on the inside, contained in a synthesis activity as a local reasoning primitive.

Built. The thesis became engineering requirements: explicit preflight, extraction, synthesis, transformation and publish activities; per-activity retries, heartbeats, batching and durable progress; incremental markers and content hashes that only advance after a successful publish; connector-agnostic JSONL routed through the existing publish pipeline so writes inherit diffing, circuit-breaking and ordering instead of hitting the metadata store directly; standard credential flows and tenant-isolated task queues. I also shipped fire-and-poll as an immediate UX bridge, so the timeout stopped hurting users while the full worker path was built.

Learned. The hard part looked like the agent. It was the category. Once the job was named a connector, most of the architecture had already been decided — by systems that had been running for years.

Impact

• A demo-shaped agent became a platform that runs scheduled, resumable, multi-tenant jobs
• Run history, retries and recovery came free from Temporal instead of being reinvented per agent
• Writes inherited the publish pipeline’s diffing and circuit-breaking
• The shape generalised — it now applies to any context agent, not just this one

The useful skill here was not the architecture. It was noticing that the question had been miscategorised before anyone had scaled the wrong answer.

Agent Harness & Runtime

Problem. The synthesis agent kept exposing gaps in the shared agent harness that had nothing to do with the synthesis agent. Each application was about to solve them separately — and one of the candidate solutions was going to be “read the secret over HTTP”.

Built. I fixed them in the harness, as reusable platform capability, using the agent as a proving ground rather than adding application-specific workarounds.

• Credential boundary. MCP server configuration carries a credential reference, resolved in-process when the harness opens an outbound MCP connection. Application code stores and passes only the reference; the harness materialises the secret for the live connection. Per-tenant authenticated MCP servers, with no path that reads a secret over HTTP.
• Transport. Moved to Streamable HTTP, which let vendor-native MCP services and platform-hosted adapters share one connection contract.
• First-party auth. The platform’s own MCP server does not need a user-created connection — the runtime uses the tenant’s platform identity for search, lineage, glossary and knowledge.
• Session durability. Persisted transcripts are reconstructed from Postgres checkpoints, so a run recovers the history it needs instead of trusting in-memory state.
• Knowledge-aware agents. Knowledge-file tools in the harness itself, so any workload can read business documentation in its reasoning loop instead of each workflow building its own document access.
• Controlled reasoning. A reasoning-effort and thinking-budget switch with provider-combination validation — off by default, so no existing agent changes behaviour.
• Observability. Run attributes captured at dispatch and emitted as first-class OpenTelemetry attributes.

Learned. The working assumption was that legacy SSE would do. Against real remote services it hung. Interop gets verified against the real thing, never assumed from the spec.

Impact

• Secrets stay behind the credential boundary — no plaintext hop between services
• Platform capability, not per-application workarounds — every agent inherits it
• External MCP interop verified against real remote services
• Agents can opt into deeper reasoning without changing the default for anyone else

Turning an Agent into Governed Configuration

Problem. An agent configured by a prompt box cannot be governed. There is no way to state which sources it may touch, no way to narrow that per install, and no way to stop its behaviour changing underneath the people relying on it.

Built. Three governance axes, deliberately kept separate:

• Template — the platform-authored outcome and behaviour contract.
• Grant — which source types the agent may access. Widening it mints a new version.
• Scope — the subset of those sources it may actually read. Editable, and enforced at runtime rather than displayed in a UI.

Alongside it, the operational-honesty work: dry-run as a real capability that executes the data plane and stops before publish, not a UI-only pretence; stopping a run midway; a preflight that names the source that failed instead of returning a bare gateway error; scheduled runs that show their real outcome and cost; and run semantics that distinguish success, degradation and failure instead of collapsing into “something went wrong”.

On the source side, an adapter contract where a new hosted source self-registers and is exposed as a search/read tool pair, so adding a source changes nothing else in the system — with one credential and connection lifecycle shared by vendor-native and platform-hosted sources. Connect once, reuse across agents, with verification before storage, in-place rotation and deletion.

The design not yet built — resume. Resuming a failed run from its checkpoint, with a compare-and-set state change as the lock, the run pinned to its original agent version, a cap on resumes, and the checkpoint position exposed so a caller can judge whether replay is safe. The reason it is careful: without a re-entrant write tail, replaying an interrupted run publishes the same assets twice.

Learned. A missing credential did not fail loudly. It was being skipped, and the run carried on against the remaining servers and reported success. It was the first of many green runs that had quietly done less than they claimed — and hunting those became most of act iii.

Impact

• Agent behaviour became versioned configuration, not an unversioned prompt
• Scope is enforced at runtime — governance the system applies, not governance the UI describes
• A malformed token now fails loudly at connect time instead of being stored as “connected” and silently producing a run with fewer tools than asked for
• Resume treated as an end-to-end correctness problem, not a button

The Custom Agent Marketplace

Problem. Both ends of the obvious design space are wrong. A marketplace of pre-built agents does not scale, because every agent in it is something an engineer had to build first — the catalogue grows at the speed of the team, not the customer. And a prompt box does not work either: a prompt cannot be bound to sources, cannot be scoped, cannot be versioned, and cannot be trusted to write governed assets into a catalog people depend on.

Built. The piece I led from scratch: let the user state the outcome in their own language, and have the backend assemble the agent. Somebody says what they want — turn our wiki runbooks into governed glossary terms, keep this data product’s documentation current — and the resolution logic works out everything underneath it:

• Which skills to bind — what this agent needs to know how to do
• Which sources to connect — and the grant and scope that come with them
• Which outcome to generate — the asset type it is permitted to write

Skills are what the agent knows. MCP tools are its hands — how it actually reaches a source, reads a passage, writes an asset. The harness is the runtime that executes it durably, with the credential boundary, the transport contract and session recovery underneath. Everything in the entries above is load-bearing here: the connector-shaped Temporal workflow, the evidence gate, the template/grant/scope axes, the dry run that stops before publish.

The synthesis agent was the proof that one outcome could work end to end. This is what it became: the outcome is a parameter, not the product.

Impact

• An agent stopped being something an engineer ships and became something a user describes
• The marketplace is generative rather than a catalogue — a new outcome does not need a new codebase
• Natural-language intent lands as governed configuration: bound skills, granted sources, enforced scope, a versioned template
• Every agent it produces inherits the same durable execution, evidence gate and publish path — governance is structural, not per-agent discipline

The interesting part was never the agent. It was making the outcome the thing you configure, and letting everything else fall out of that.

Into Customers’ Hands

Problem. The marketplace worked for the people who built it. Customers are a different test: their tenants, their identity setup, their documents, and their pods restarting at the worst possible moment.

The call. Ran the final checks before the first private-preview customers went live, and deliberately held one abstraction — custom MCP servers as agent sources — out of the first release rather than ship it half-proven. Wrote down the agent-experience limitations so the product side could set expectations honestly instead of discovering them in a demo.

Built. Took ownership of the agent app’s releases — approving them, and deploying to each new preview and early-access tenant — and of triaging agent-quality issues. Widened the cohort in steps through September and October, each step gated on the round of fixes before it landing.

Challenge. A customer demo run died midway when the cluster restarted the pod underneath it — exactly the failure the resume design was written for, and the reason it went back on the list. A shared dev environment meant someone else’s deploy could silently revert mine; we fixed that by stacking changes on one branch instead of racing each other.

Learned. This was the first time I shipped something I had built from scratch to customers. Under pressure I reached for the cheap workaround; the better move, almost every time, was to step back and look at what the workaround was hiding. And once a merge reaches customers within minutes, an approval stops being a courtesy — it is a deploy decision with my name on it.

Impact

• First customers live in private preview, then a widening early-access cohort
• First organic customer runs — not demos — within weeks
• Field teams started asking to enable it for more prospects

Making the Agent’s Output Correct

Problem. Early runs shipped missing READMEs, duplicate terms, empty categories, and counts that overstated what a run had done. The natural reading was “the model isn’t good enough yet.”

What it actually was. Not the model. A README check queried a column that did not exist, so it could never fail. A misspelled tool name silently degraded every definition the agent wrote. Re-runs re-derived everything instead of skipping what already existed. And the UI showed the glossary’s total size as if it were the run’s output.

Built. Gave README decisions their own table so the check can actually fail; fixed the tool name; made re-runs skip existing terms; made the UI report what the run created. Moved runs to structured reports — the agent declares its outcome, instead of the system guessing it from the last chat message.

Then the folder leak: an instruction naming one file made the agent read the whole folder. The agent was following its rules — and the rules treated seeing a file as permission to read it. A listing is not a licence. Whatever the user names is now binding, and every run records the documents it read — deliberately as an observation, not a failing check, until real runs show what the matching has to handle.

Learned. I expected a prompt problem. Every cause was plumbing. Before blaming the model, check whether the check can fail.

Impact

• Category READMEs: none → complete
• Verification re-runs produced zero duplicates
• The rollout cohort widened right after
• Fixes from others now get proven on a live run before merge, not only reviewed

Uploaded Documents as a Source — and Reading Them Properly

Problem. Agents could only read connected systems, so all the value sat behind connector setup — while the customer’s own uploaded SOPs and policies were right there, unreachable. And under that problem was a second one: diagram-heavy PDFs have almost no text layer. On a typical slide deck most pages yield almost nothing, and agents were answering from a fraction of the document without knowing it.

The call, part one. The suggestion was to reuse a generic read tool. I argued against it: no semantic search, hard limits on how much it could retrieve, and it reached the metadata store directly with a service token — skipping the knowledge service, and with it the credential boundary from act ii. Shipped the parsed-text path the same day, and set up experiments on synthetic documents comparing parsed text against handing the model the original file. That data is what led to part two.

Built, part two — models that read the PDF. Benchmarked frontier models and picked the most reliable. Parsing happens at upload, not at read: a long PDF takes minutes and the runtime kills long tool calls, so reads became cache hits. The parsed copy is keyed by the source hash plus the prompt and the model — editing the document and changing the prompt are the same cache miss, so there is never a migration job. Reads report which pipeline produced the content and fall back to text extraction, and the tool docs tell agents to check which one they got. The parsed copy is deleted with its source.

Calls in review. Kept the model in the cache key, so changing the default model re-parses everything automatically rather than serving stale reads. Kept rejecting malformed-encoding files — a loud rejection beats indexing garbage. Pushed back on formats the models do not read natively; they were deferred.

Challenge — found by running it, not reading it.

• Browser uploads never triggered a parse
• Folder delete always returned 404, on a parameter mismatch
• Queued parses failed once the uploader’s short-lived token expired
• A derived copy could outlive its source document — a data-lifecycle gap, closed before release

Known gaps, stated in the release rather than discovered later: parses lost to a pod restart are not recovered yet, and search still uses the older index, so diagram content is readable but not yet searchable.

Learned. None of those bugs were visible from reading the code. Every one of them showed up the first time it ran end to end. “Found by running it” is now the section of a release note I trust most.

Impact

• Zero setup — a glossary built from documents the customer had already uploaded
• Image-heavy PDFs yield several times more content; terms that only exist inside diagrams are recoverable
• A batch of simultaneous uploads went from all failing → all succeeding first try
• One connector’s outage no longer blanks the source list for every other agent
• Released to every tenant; the release note was forwarded company-wide

Green Runs With Nothing in Them

Problem. Enterprise customers using service accounts — the credential type security teams require — could not connect a major wiki source at all. The vendor’s older MCP endpoint was also on its way to retirement.

Built. Moved to the vendor’s newer remote MCP endpoint with bearer auth. Made connection checks make a real call instead of just listing tools. Passed the site identifier into the run. Found that agents were being loaded from a list endpoint that carried no MCP config, so every source had silently lost its required-tools check — fixed, with its own test.

Challenge. The first merge was reverted before rollout and re-landed unchanged. The customer tickets looked like a cluster of separate bugs; they were symptoms, and two root fixes resolved all of them.

Learned. Each surprise was a run that goes green with nothing in it. Basic auth lists the tools and then fails every real call. Semantic search returned nothing where a structured query found the pages. Page reads default to a summary, and return an empty body as a clean 200.

Impact

• Three separate paths to a successful run with zero output — closed
• Security-mandated service accounts work
• Ahead of the vendor’s forced cutover, not behind it
• Release note forwarded company-wide

Citations on Agent-Written Terms

Problem. Full runs collected evidence for every term — and then threw it away at publish. A steward certifying an AI-written definition had no way to trace it back to where it came from, at exactly the moment they needed to.

Built. Terms now carry their sources, as native catalog links or linked knowledge files. Run results return citations per term, and the UI shows them on the run and on the term. Only http(s) URLs render as links. Citation work can never fail a run. Got a product decision on the field design up front, and made it a global change so scheduled runs get it too. Designed next: per-source metadata — version, last modified, owner — keeping only the latest values, and storing people as display names, never account IDs.

Learned. The per-citation version is the same watermark incremental reads and refresh triggers need. Record it once, and three features get it for free.

Impact

• One click from an AI-written definition to its source
• Costs seconds per run, and can never fail one
• Released to the early-access cohort, and became an input to the re-run design

Re-runs That Remember

Problem. A re-run did not know what it had published last time. It redid the full research loop, renamed concepts into near-duplicates, never updated terms whose source had changed — and spent a lot of tokens to add a handful of terms.

The call. Earlier output is an input at every step that makes a decision — and you can check whether each step used it.

• Before a run, snapshot the agent’s own glossary from the catalog — no row cap, keyed by a hash of its content.
• Every comparison is a SQL join in a data plane. A model never reads two lists to compare them.
• Reader subagents return evidence, not verdicts. A separate per-row judge decides same or changed.
• Only new or changed rows are rewritten. Nothing is deleted, and a term is marked stale only after a complete read.

Rejected — my own first draft. A manifest file written at the end of each run. On the real run path its key collapsed to one key shared by every agent, the read tool refused its location, and a run that died after publishing would have orphaned its terms.

Challenge. Once real users are recorded as authors, every term looks human-edited. Older terms have no citations to compare against. And IDs got corrupted when models retyped them from step to step — so the design passes short handles instead.

Learned. Deriving state from what is already in the tenant beat keeping a parallel copy of it. The copy was the bug.

Status

• Design agreed; code in testing. Not shipped, so the impact is not measured — and is not claimed here.

Debugging Across Layers

Problem. Agent runs failed with symptoms that pointed at the wrong layer — the UI, the gateway, the catalog, the credential service, the cluster.

Root causes I traced.

• Agent creation denied on one tenant: not version drift — the images were identical. It was access configuration, not a bug, so I pushed for documented access prerequisites for customers.
• Source connection failing: traced the full call path, end to end, down to a single missing configuration entry.
• Runs that never finished: the autoscaler was scaling the worker to zero partway through long runs. Built the timeline.
• A lost pod: a spot-node removal that took many pods with it — and said clearly what could not be seen from inside the virtual cluster.
• An LLM-gateway blocker for a new agent template that turned out to be several stacked blockers — a missing route, a rolled-back upgrade, an unconfirmed fix, a provider mismatch. Held to the org’s release-age cooldown instead of reaching for the newest version, and the infra owners committed to the upgrade path.

A review that changed a design. A PR had the model confirm a long list of tool names before it would start. I argued the gate was a liveness check, so one name proves the binding; that the code already refuses a run deterministically when a server offers no tools; and that MCP is a discovery protocol — a dynamic tool list can be observed, not statically checked. Proposed refusing in code, putting only the facts the model cannot discover into the agent’s contract, letting the skill own discovery, and recording which tools each server actually served. Said plainly that the gate is right for static, built-in sources. The PR was reworked and merged.

Smaller fixes along the way.

• Serialised concurrent folder creation — a race condition removed
• OpenTelemetry resource attributes, so workflow logs land in the right combined views
• A file-type gap that returned 502s on agent-saved Markdown uploads
• An endpoint mounted at both paths, so internal and MCP callers both resolve
• Skill files mounted under their storage URI, so agents could not follow nested references — side-by-side evidence, then the owner’s fix

Learned. Most of these “bugs” were identity, configuration or infrastructure — not code. The useful output of an RCA is not the fix. It is a symptom turned into a ticket with the right owner.

Impact

• Customers unblocked; every symptom landed with an owner
• A merged design changed by argument, not by authority