# AI assistant for a security SaaS

Jul 2025 – Aug 2026 · Staff · Shipped

I started the product’s AI assistant as a hackathon prototype and built it into a production feature used by 200 customers. Security teams ask questions about their own data in plain language; the assistant answers with live streamed replies and can act inside the app, while a person stays in control of every change.

**Scale:** 200 customers · 2 AWS regions · 12 in-app actions · 7 approval-card types · 55 backend test files

- Agent backend: Python on AWS Lambda, Claude through AWS Bedrock with the Strands agent framework, streamed to the browser as AG-UI events.
- Reaches product data through the MCP server, loading only the tools it needs via BM25 keyword search.
- Acting safely: it can navigate, set filters and prepare tickets, policies and automations, and every change appears as an approval card a person confirms.
- Cost and abuse controls: per-user and per-company rate limits, daily token and spending budgets and a concurrent-run cap, tuned from the back office without a redeploy.
- Reliability on serverless: deadlines set below the platform’s kill limit, a guaranteed finished signal on every path, and UI state machines with bounded waits; the runtime choice is recorded in a decision record.
- Safety and observability: content guardrails with a test harness, redaction of page context from stored history, a per-company switch, and per-call token and cost tracing.

**Stack:** Python, Strands, Bedrock, Lambda, AG-UI, React, CopilotKit

**My role:** Creator and owner, from hackathon prototype to production: architecture, agent backend, chat UI, AWS infrastructure and operations.

## Context

Security teams work across a lot of product data: exposures, assets, applications and alerts. The assistant had to answer questions about that data and do the clicking for them, from navigating and filtering to preparing tickets, policies and automations.

Two constraints shaped the design. The product is multi-tenant, so every answer must stay inside the asking user’s company and role. And it runs on serverless, where a long streamed answer can be cut off by the platform mid-stream.

## Timeline

- **Jul 2025:** Hackathon prototype: an AI chat inside the product.
- **Dec 2025:** Agent backend started; product data moves behind the new MCP server.
- **Jan 2026:** Production chat merged into the web app.
- **Feb – Mar 2026:** BM25 tool discovery, streaming and thinking events, session store.
- **Jul 2026:** Second AWS region with region-aware Bedrock inference; per-company switch.
- **Aug 2026:** Streaming hardening, usage controls, back-office config and approval-card create tools.

## Architecture

One question, end to end. The agent is the hub: it checks limits, picks tools, calls the model and streams events back to the chat.

- **Browser:** Chat panel (Streamed replies, tool cards, reasoning panel); Frontend actions (Navigate, set filters, approval cards)
- **Agent service · AWS Lambda:** API Gateway (Response streaming, real incremental SSE); Agent (Strands on FastAPI, persistent per thread); Usage controls (Rate limits, budgets, run cap)
- **Model, tools and state:** Claude on Bedrock (Bedrock Guardrails on every call); MCP server (72 typed, permission-scoped tools); Session store (S3 transcripts, DynamoDB metadata); Tracing (OpenTelemetry to Datadog LLM Observability)

Flows:

1. Chat panel → API Gateway: A question, with the page’s context
2. API Gateway → Agent: One streamed run per question
3. Agent → Usage controls: Rate limits, budgets and the run cap are checked first
4. Agent → MCP server: BM25 picks the tools to load; each call runs as the signed-in user
5. Agent → Claude on Bedrock: Claude with the loaded tools; guardrails filter both ways
6. Agent → Chat panel: AG-UI events: text, thinking, tool calls, finished
7. Agent → Frontend actions: A frontend tool halts the run until the UI answers
8. Agent → Session store: Transcript, metadata, tokens and cost per user and company
9. Agent → Tracing: A span per call, with tokens and cost

## Inside the system

### Streaming on serverless

- FastAPI on AWS Lambda behind the Lambda Web Adapter; API Gateway response streaming delivers real incremental Server-Sent Events.
- Conversation state lives outside the function (S3 transcripts, DynamoDB metadata), so the in-memory per-thread agent is only a cache.
- An ECS Fargate target exists as an opt-in alternative behind the same API.

### Many tools, a small context

- Tools come from the MCP server, plus frontend tools synced from the page the user is on; frontend tools override backend stubs.
- A BM25 index over the tool schemas gives the agent search_tools and list_loaded_tools, so it loads only what a question needs.

### A person stays in control

- The agent proposes applications, SLA and prioritization policies and automations as approval cards; nothing is created until a person confirms.
- Direct-write MCP tools are excluded from the chat agent.
- 12 actions inside the app and 7 approval-card types.

### Safety and tenancy

- Amazon Bedrock Guardrails (content filters, prompt-attack protection, denied topics) with a test harness.
- Page context enriches each message and is redacted from stored history.
- Tenant-isolated S3 prefixes and DynamoDB keys, and a per-company switch enforced in the agent, the MCP server and the UI.

### Cost you can tune

- Per-user and per-company rate limits, daily token and spending budgets, and a cap on concurrent runs.
- An internal config API lets admins change any limit from the back office without a redeploy.
- Tokens and cost are stored per session, user and company.

## Hard problems

### The chat that never stopped loading

**Problem.** Some runs left an endless loader, or a tool indicator that never cleared. The causes crossed the agent backend, the streaming transport and the React chat state.

**What I did.**

- Traced the failure paths end to end, backend to UI, instead of patching the symptom.
- A connect fast path: reopening a chat no longer builds a full agent or waits on an MCP handshake.
- Deadlines composed below the platform’s kill limit, so the service always times out first and can say so.
- A finished event on every code path, an orphan guard, single-flight agent construction, and model-stream errors sanitized instead of left hanging.
- In the UI, a run-phase state machine and a bounded connect phase with retry replaced implicit loading flags.

**Result.** Closed by coordinated backend and UI changes in Aug 2026, with two postmortems. No before-and-after incident rate was measured.

### A concurrency cap that cannot leak

**Problem.** A multi-tenant LLM feature needs a true cap on concurrent runs per user. In-process counters do not survive Lambda, and an “active in the last few minutes” window is imprecise.

**What I did.**

- One short-lived record per in-flight run, with a time-to-live of the run’s deadline plus a grace margin.
- Created when the run starts and deleted in a finally block, so a crashed run stops counting on its own.
- Per-user and per-company budgets layered on top, read from a config record admins edit.

**Result.** Merged in Aug 2026; admins tune limits per company without a deploy. There is no cost-savings figure.

### Tools a model can find

**Problem.** With 72 MCP tools plus the frontend tools, sending every schema on every turn would crowd the model’s context, and the catalogue keeps growing.

**What I did.**

- An in-memory BM25 index over tool names, descriptions and schemas.
- Two agent tools, search_tools and list_loaded_tools, so the model asks for what it needs.

**Result.** The model works with the tools it searched for, not the whole catalogue.

## Decisions

- **Runtime:** chose stay on AWS Lambda with API Gateway response streaming, over eCS Fargate, or a managed agent runtime. Streaming works, state is external and the agent cache is only an optimization. Fargate stays the fallback if turns grow past about 100 seconds. Recorded in a decision record.
- **Writes:** chose the agent proposes, a person confirms, over direct-write tools in the chat agent. An automation or a policy changes how the product behaves for a whole team. A card shows exactly what will be created before it is.
- **Tool loading:** chose keyword search (BM25) over tool schemas, over every tool in every prompt. Keeps the prompt small as the tool catalogue grows, with no embedding service to run.

## Outcomes

- Used by 200 customers (owner-stated).
- Runs in 2 AWS regions, EU and US.
- 12 in-app actions and 7 approval-card types.
- 55 backend test files: unit, integration, end to end and the guardrail harness.

## Limits

- No traffic, latency or incident-rate figures are quoted here.
- The guardrails have a test harness, but no pass rate is quoted.