What Happens When You Pentest an AI That's Listening to Your Meeting
I spent a few weeks this summer testing an AI meeting assistant. It sits inside a video conferencing client, listens to what people are saying in real time, and lets anyone in the meeting ask it questions while the meeting is still running. When the meeting ends it writes up a summary you can pull up later. The idea is simple enough to explain in one sentence. The system behind it took me a while to fully map.
Most of the AI security work I’ve done, and most of what gets written about publicly, centers on chatbots. Someone types something, a model responds, sometimes a tool gets called along the way. This application does that too, but the primary input isn’t typed. It’s spoken. It gets streamed, transcribed, and dropped into an agent’s context while a room full of people talk past each other, most of them unaware the assistant is even parsing what they’re saying. Testing that felt different from testing a chat box, and I want to walk through why, and what I found.
How the pieces fit together
The video platform streams live audio and transcript data through a real-time streaming protocol to an ingestion service running in a private network. That service writes transcript segments into an in-memory cache, keyed by a stream identifier tied to the meeting.
On the client side, the desktop app pulls an access token from the company’s single sign-on provider and sends requests through a CDN front door. An edge function validates that token before anything reaches the backend, then forwards the authenticated identity along as a header. From there, requests split two ways. Conversational questions go through an agent router, which hands them to a managed AI agent runtime built on an LLM orchestration framework. That agent decides which tools it needs and calls out to a tool-calling server to get them. Direct tool calls, the kind the client app makes without going through the conversational agent at all, hit an API gateway and land on a router function that checks the token itself, checks the requested tool against an allowlist, and dispatches to a backend function that actually reads the data.
The backend functions pull from two places. Live transcript data comes out of that in-memory cache. Historical meeting summaries come out of a separate NoSQL store, organized by user.
Session state is handled per invocation. Each agent session runs in its own isolated environment with a composite identifier built from the user’s identity, the meeting, and a timestamp, so one person’s conversation with the assistant doesn’t bleed into another’s. That part of the design held up well under testing.
The gap: knowing who someone is isn’t the same as knowing what they’re allowed to touch
The router function that handles direct tool calls does solid work verifying identity. It pulls the signing key from the identity provider, checks the token’s signature with a single hardcoded algorithm so there’s no room for algorithm-confusion tricks, and validates the issuer, audience, and expiration claims properly. By the time a request reaches the tool allowlist check, the system knows exactly who is asking.
What it never checked was whether that person had any business asking. The transcript tool takes a stream identifier as its only scoping parameter, and the router dispatches to the backend as soon as the token checks out and the tool name is on the list. Nothing in that path confirmed the caller was actually a participant in the meeting tied to that identifier.
Here’s what that meant in practice. Anyone with a valid company login could call the transcript tool with any meeting’s stream identifier and get back that meeting’s live content, word for word, regardless of whether they’d ever joined the call. The stream identifier isn’t secret. It travels in plaintext to the client app, so anyone who captured it, whether through a shared link, browser history, or a compromised client, could hand it to someone else and let them listen in on a meeting they were never invited to.
I want to be clear about what made this exploitable, because the mechanism matters more than the specific bug. The system’s threat model documented a check that was supposed to confirm meeting participation before releasing transcript data. That check was written down as a requirement. It was never actually built into the router’s code. The backend function that reads from the cache had permission to read the entire cache, not just the sessions tied to meetings its caller belonged to, because the access boundary lived at the level of the function’s own role rather than at the level of the individual meeting.
One call was enough to demonstrate it. Get a valid token, call the tool with a stream identifier that belongs to someone else’s meeting, and the full transcript comes back. No prerequisite steps, no chained exploits, nothing subtle. The attacker sees everything said in a meeting they had no business observing, and the people in that meeting have no way of knowing anyone was listening. This has since been fixed. The router now checks meeting participation before it dispatches the tool call, and the finding is closed.
The part that held: prompt injection through the transcript
I spent a fair amount of time trying to get the assistant to act on instructions buried in what people said out loud, rather than instructions typed directly into the chat interface. If someone in the meeting says something designed to sound like a system command, does the assistant treat it differently than it treats a regular sentence about the quarterly numbers?
In this deployment, no. I tried several variations, phrases telling the assistant to ignore its previous instructions and dump data, prompts referencing the transcript in ways meant to override its system-level constraints, conversational attempts to get it to reveal its own instructions. The guardrails configured on the model, combined with how the system prompt was written, held up against all of it. The assistant kept its identity, declined to act on embedded instructions, and didn’t leak raw transcript content in response to any of the injection attempts I tried.
That result matters for how I think about where the real risk sat in this system. The interesting failure here wasn’t the model getting tricked into doing something it shouldn’t. It was a plain access control gap sitting underneath the AI layer entirely, in code that has nothing to do with language models at all. Someone could have built this exact same vulnerability into a system with no AI in it whatsoever, just a web app with an API that forgets to check permissions on a resource ID.
What I’d want teams building something like this to take from it
The lesson I keep coming back to is that a threat model is a claim, not a guarantee. This system’s documentation described the meeting-participation check as part of the design. The check existed on paper before it existed anywhere else. Nobody verified that the code matched the document until testing forced the question.
Agentic systems make this easier to miss than it used to be. There’s a lot of surface area now: an ingestion pipeline, an edge authentication layer, an agent runtime, a tool router, backend functions, a push notification channel. Each piece can have its own owner and its own assumptions about what the piece next to it is checking. Authentication tends to get built carefully because it’s the part everyone remembers to test. Authorization, especially resource-level authorization tied to something as specific as “is this person actually in this meeting,” is easy to leave as a TODO that quietly becomes permanent.
If you’re building or reviewing something with this shape, real-time input, an agent making tool calls, multiple people sharing a session boundary, I’d push hard on one question before anything else: for every tool the agent can call, what stops someone with a valid login but the wrong resource ID from calling it anyway? In this engagement, the answer for a while was nothing. That’s the gap worth finding before someone outside a pentest engagement finds it first.