00Case study · Jul 2025 – Aug 2026

AI assistant for a security SaaS

Security teams ask about their own data in plain language; the assistant streams answers and acts inside the app behind approval cards.

StaffShipped

My roleCreator and owner, from hackathon prototype to production: architecture, agent backend, chat UI, AWS infrastructure and operations.

200 customers
2 AWS regions
12 in-app actions
7 approval-card types
55 backend test files

01Overview

What it is, and why

I started the product’s AI assistant as a hackathon prototype and built it into a production feature used by 200 customers. Security teams ask questions about their own data in plain language; the assistant answers with live streamed replies and can act inside the app, while a person stays in control of every change.

Security teams work across a lot of product data: exposures, assets, applications and alerts. The assistant had to answer questions about that data and do the clicking for them, from navigating and filtering to preparing tickets, policies and automations.

Two constraints shaped the design. The product is multi-tenant, so every answer must stay inside the asking user’s company and role. And it runs on serverless, where a long streamed answer can be cut off by the platform mid-stream.

At a glance

How it works, in short

Agent backend: Python on AWS Lambda, Claude through AWS Bedrock with the Strands agent framework, streamed to the browser as AG-UI events.
Reaches product data through the MCP server, loading only the tools it needs via BM25 keyword search.
Acting safely: it can navigate, set filters and prepare tickets, policies and automations, and every change appears as an approval card a person confirms.
Cost and abuse controls: per-user and per-company rate limits, daily token and spending budgets and a concurrent-run cap, tuned from the back office without a redeploy.

Python
Strands
Bedrock
Lambda
AG-UI
React
CopilotKit

02Timeline

How it came together

Jul 2025 to Aug 2026.

Jul 2025
Hackathon prototype: an AI chat inside the product.
Dec 2025
Agent backend started; product data moves behind the new MCP server.
Jan 2026
Production chat merged into the web app.
Feb – Mar 2026
BM25 tool discovery, streaming and thinking events, session store.
Jul 2026
Second AWS region with region-aware Bedrock inference; per-company switch.
Aug 2026
Streaming hardening, usage controls, back-office config and approval-card create tools.

03Architecture

How the parts fit

One question, end to end. The agent is the hub: it checks limits, picks tools, calls the model and streams events back to the chat.

assistant · request path

Browser
- Chat panelStreamed replies, tool cards, reasoning panel
- Frontend actionsNavigate, set filters, approval cards
Agent service · AWS Lambda
- API GatewayResponse streaming, real incremental SSE
- AgentStrands on FastAPI, persistent per thread
- Usage controlsRate limits, budgets, run cap
Model, tools and state
- Claude on BedrockBedrock Guardrails on every call
- MCP server72 typed, permission-scoped tools
- Session storeS3 transcripts, DynamoDB metadata
- TracingOpenTelemetry to Datadog LLM Observability

Chat panel to API GatewayA question, with the page’s context
API Gateway to AgentOne streamed run per question
Agent to Usage controlsRate limits, budgets and the run cap are checked first
Agent to MCP serverBM25 picks the tools to load; each call runs as the signed-in user
Agent to Claude on BedrockClaude with the loaded tools; guardrails filter both ways
Agent to Chat panelAG-UI events: text, thinking, tool calls, finished
Agent to Frontend actionsA frontend tool halts the run until the UI answers
Agent to Session storeTranscript, metadata, tokens and cost per user and company
Agent to TracingA span per call, with tokens and cost

Point at a box or a flow to trace it. The numbers match the list.

04Inside the system

The parts that matter

01
Streaming on serverless
- FastAPI on AWS Lambda behind the Lambda Web Adapter; API Gateway response streaming delivers real incremental Server-Sent Events.
- Conversation state lives outside the function (S3 transcripts, DynamoDB metadata), so the in-memory per-thread agent is only a cache.
- An ECS Fargate target exists as an opt-in alternative behind the same API.
02
Many tools, a small context
- Tools come from the MCP server, plus frontend tools synced from the page the user is on; frontend tools override backend stubs.
- A BM25 index over the tool schemas gives the agent search_tools and list_loaded_tools, so it loads only what a question needs.
03
A person stays in control
- The agent proposes applications, SLA and prioritization policies and automations as approval cards; nothing is created until a person confirms.
- Direct-write MCP tools are excluded from the chat agent.
- 12 actions inside the app and 7 approval-card types.
04
Safety and tenancy
- Amazon Bedrock Guardrails (content filters, prompt-attack protection, denied topics) with a test harness.
- Page context enriches each message and is redacted from stored history.
- Tenant-isolated S3 prefixes and DynamoDB keys, and a per-company switch enforced in the agent, the MCP server and the UI.
05
Cost you can tune
- Per-user and per-company rate limits, daily token and spending budgets, and a cap on concurrent runs.
- An internal config API lets admins change any limit from the back office without a redeploy.
- Tokens and cost are stored per session, user and company.

assistant · illustrative replayfinished

userWhich critical exposures have no owner yet?
plandiscover tools · BM25 over 72 schemas → 4 loaded
toolsearch_exposures(severity="critical", owner=null)
result18 results · scoped to your role
toolpropose_automation(assign_by="asset_owner")
hitlapproval card · waiting for a person to confirm
doneRUN_FINISHED · 2.1k tokens · within budget

An illustrative replay of the assistant’s event stream, not production data.

05Deep dive

Hard problems

What stood in the way, what I did, and what came of it. Where nothing was measured, it says so.

Hard problem 01

The chat that never stopped loading

Problem: Some runs left an endless loader, or a tool indicator that never cleared. The causes crossed the agent backend, the streaming transport and the React chat state.
What I did: Traced the failure paths end to end, backend to UI, instead of patching the symptom.
A connect fast path: reopening a chat no longer builds a full agent or waits on an MCP handshake.
Deadlines composed below the platform’s kill limit, so the service always times out first and can say so.
A finished event on every code path, an orphan guard, single-flight agent construction, and model-stream errors sanitized instead of left hanging.
In the UI, a run-phase state machine and a bounded connect phase with retry replaced implicit loading flags.
Result: Closed by coordinated backend and UI changes in Aug 2026, with two postmortems. No before-and-after incident rate was measured.

Hard problem 02

A concurrency cap that cannot leak

Problem: A multi-tenant LLM feature needs a true cap on concurrent runs per user. In-process counters do not survive Lambda, and an “active in the last few minutes” window is imprecise.
What I did: One short-lived record per in-flight run, with a time-to-live of the run’s deadline plus a grace margin.
Created when the run starts and deleted in a finally block, so a crashed run stops counting on its own.
Per-user and per-company budgets layered on top, read from a config record admins edit.
Result: Merged in Aug 2026; admins tune limits per company without a deploy. There is no cost-savings figure.

Hard problem 03

Tools a model can find

Problem: With 72 MCP tools plus the frontend tools, sending every schema on every turn would crowd the model’s context, and the catalogue keeps growing.
What I did: An in-memory BM25 index over tool names, descriptions and schemas.
Two agent tools, search_tools and list_loaded_tools, so the model asks for what it needs.
Result: The model works with the tools it searched for, not the whole catalogue.

06Decisions

Trade-offs I made

What I chose, what I passed on, and why.

Decision · Runtime
Stay on AWS Lambda with API Gateway response streaming
OverECS Fargate, or a managed agent runtime
Streaming works, state is external and the agent cache is only an optimization. Fargate stays the fallback if turns grow past about 100 seconds. Recorded in a decision record.
Decision · Writes
The agent proposes, a person confirms
OverDirect-write tools in the chat agent
An automation or a policy changes how the product behaves for a whole team. A card shows exactly what will be created before it is.
Decision · Tool loading
Keyword search (BM25) over tool schemas
OverEvery tool in every prompt
Keeps the prompt small as the tool catalogue grows, with no embedding service to run.

07Outcomes

What came of it

Measured or owner-stated

Results

Used by 200 customers (owner-stated).
Runs in 2 AWS regions, EU and US.
12 in-app actions and 7 approval-card types.
55 backend test files: unit, integration, end to end and the guardrail harness.

Said plainly

Limits

No traffic, latency or incident-rate figures are quoted here.
The guardrails have a test harness, but no pass rate is quoted.