Hi everyone, in this article, I’ll walk through a recent penetration test I conducted against a custom-built AI chatbot. As usual, we’ll cover:

• The application overview
• The high-level architecture
• The vulnerability
• The exploit

This assessment was conducted as a black-box test, meaning no source code access, no architectural documentation, and no internal visibility — only what an external attacker would see.

Application Overview

Let’s call the company A.Corp.

Instead of integrating a third-party chatbot, A.Corp built their own AI assistant from scratch:

• The model was trained in-house.
• The user interface was custom-built.
• The guardrails and safety controls were implemented internally.

Overview

Riding on the AI wave, this company, had come up with an AI chatbot. Unlike the ones in the market, the had created one for themselves, which means —

• It was an in-house trained model.
• They had created the UI themselves
• The guardrails were also created by their team