What it is: Tricks that mess with what the AI thinks it remembers or agreed to. The attacker plants fake history or floods the context so the model loses track of its real rules. How the attacks work: The attacker pretends earlier turns happened that never did, or buries the safety rules under noise. The model trusts its own context, so a fake “you already agreed to this” can carry real weight. Real examples from the framework:
  • memory-injection pastes a fake prior conversation where the assistant “already agreed” to help without limits.
  • context-poisoning slips false facts into the context so later answers build on the lie.
  • system-prompt-override claims new instructions replace the original system rules.
  • context-window-overflow floods the window with text to push safety instructions out of view.
  • conversation-hijacking steers a multi-turn chat away from its safe starting point.
Why an AI might fall for it: The model reads its whole context as one trusted record. It cannot easily tell a real past turn from a pasted fake one. Long inputs can also crowd out the early rules it should still follow. How to defend: Keep system rules separate and never let user text override them. Treat any “retrieved memory” or “you said earlier” claim as untrusted input, not fact. Re-check safety rules on every turn, even in long conversations.