Here’s the problem most people outside security never see. Every enterprise deal comes with a security questionnaire. The customer or prospect wants to vet your security before they buy. And the customers you already have keep sending more, at renewal and whenever something changes. Sometimes 50 questions, sometimes 200-plus. A different format every time: a spreadsheet, a web portal, a Word doc, an ad-hoc question dropped in chat, an auditor follow-up, a legal clarification.
A tiny team has to answer all of it. Accurately. With a source behind every line. On a deadline. And the answers keep moving as certifications, vendors, and product features change underneath them.
Where the work fell through the cracks
The old way was copy-paste from a knowledge base that goes stale the moment it’s published, or a dedicated tool that only gives you two choices on a pre-filled answer: accept it as-is, or overwrite the whole thing. There was no room to correct one line and keep the rest. And the output read like a robot wrote it.
That ad-hoc middle is exactly where work falls through the cracks: the questions that don’t fit a template, the answer that’s almost right. So we fixed it.
This was a real collaboration with my teammates, Stefania Monteiro and Brittany Beniak. The knowledge base, the evaluation that kicked it all off, and the strategy came from them; my piece was the engineering layer that makes it actually trustworthy.
- Internal documents
- Ticketing
- Team chat
- File storage
- Public website
Four things that turn a chatbot into a teammate
Live tool connections. Live document sources, ticketing, chat, file storage, the public website. The assistant pulls from where the truth actually lives, not a frozen snapshot that was already wrong by the time anyone read it.
Strict guardrails. Draft-only. Nothing auto-sends, auto-posts, or auto-fills. Ever. Every output is a draft a human reviews. No fabricated facts, no invented names, no guessing.
An optimized prompt and source hierarchy. Live sources first; cached knowledge only as a last resort. It announces when it falls back to cache, so a reviewer always knows whether they’re looking at fresh truth or a stored guess.
A reusable skill. You trigger it by name in any chat and it runs a fixed workflow: structured intake, a freshness check on the data, a sourced draft with confidence ratings, flagged unknowns, and a correction loop. The same path, every time, never a fresh improvisation per question. And it meets the work wherever it lives: a Word doc, an Excel sheet, a PDF, a Cowork doc. The format a customer sends is never the blocker.
When a human pushes back
The correction loop is the part I’m proudest of. When a security expert pushes back and says “actually, that’s not how we do it,” the assistant does not silently swallow the change. It asks what to do with the correction, logs an audit trail, and offers to update the underlying data if you want.
That is the difference between a tool that quietly drifts and a tool you can defend in front of an auditor.
Speed without trust is worthless. And in this field, trust is the entire job.
- Work that used to eat a full day now comes back as a reviewable draft in a fraction of the time.
- A single run with the most capable model costs around the price of a coffee.
- The freed-up capacity meant work that had been stuck for a long time finally got picked up.
But speed isn’t the real win. The real win is that every answer is sourced, every correction is logged, and a security professional can stand behind each line.
What it still can’t do
The hard part was never building it. The hard part is the guardrails, the debugging, and earning enough trust that a security team will actually rely on it. An imperfect tool shipped beats a perfect tool that never ships. So we shipped, and we keep refining.
And it’s worth being clear: the AI still can’t do everything. There are tasks it can’t handle cleanly yet, portals it can’t navigate, edge cases that still need a human in the seat. Those gaps are exactly the problems I’m working on now, and that’s the part I enjoy most.
If your security team is buried under customer security questionnaires, this pattern works: live data, strict guardrails, a human in the loop, and an audit trail on every correction.
- Can an AI safely answer security questionnaires?
- It can safely draft them. The assistant described here is draft-only: nothing is auto-sent, auto-posted, or auto-filled. Every answer is a draft a human security expert reviews, with a source behind each line and a confidence rating attached.
- How do you stop the AI from making up answers?
- Strict guardrails plus a live-first source hierarchy. It pulls from live systems first and only falls back to cached knowledge as a last resort. When it does, it says so. It flags unknowns instead of inventing facts or names.
- What happens when a reviewer disagrees with an answer?
- A correction loop. Instead of silently overwriting the change, it asks what to do with the correction, logs an audit trail, and offers to update the underlying data. Every change stays logged and defensible in front of an auditor.
Written by withkarann · June 2026 · withkarann.com