Miguel de Icaza on Twitter: “This is so beautiful – SQL Injection attacks but for GPT-3 and other AI text models.” / Twitter
Posted by jpluimers on 2025/03/06
2.5 years after Miguel summarised the state of AI text models, and given SQL Injection (because of mixing control and data channels) still is a thing in the 2020’s, I wonder both how much improvement there has been on the AI side of things and how much it is used in pen testing.
So I archived the below tweets to be able to read back and figure out on the current state.
[Wayback/Archive] Miguel de Icaza on Twitter: “This is so beautiful – SQL Injection attacks but for GPT-3 and other AI text models.”:
GPT-3 / OpenAI Codex
- [Wayback/Archive] Brendan Dolan-Gavitt on Twitter: “This is why you don’t mix your control and data channels!”
- [Wayback/Archive] Riley Goodside on Twitter: “Exploiting GPT-3 prompts with malicious inputs that order the model to ignore its previous directions.”
Be sure to read the follow-up messages too, as they have some great information on how to prevent this.
- [Wayback/Archive] Riley Goodside on Twitter: “Exploiting GPT-3 prompts with malicious inputs that order the model to ignore its previous directions.”
- [Wayback/Archive] Samuel Teuber on Twitter: “@moyix Funnily enough, in my feed your tweet was exactly above the tweet on using “GPT-3 armed with Python” for question answering: Achieving arbitrary remote code execution seems to be easy enough…”
- [Wayback/Archive] Sergey Karayev on Twitter: “Here’s a brief glimpse of our INCREDIBLE near future. GPT-3 armed with a Python interpreter can · do exact math · make API requests · answer in unprecedented ways Thanks to @goodside and @amasad for the idea and repl! Play with it:
replit.com/@SergeyKarayev/gptpy“
Here too: read the follow-up messages as there is good info there. A few important ons:
- [Wayback/Archive] gpt.py – Python Repl – Replit
- [Wayback/Archive] Alex Bugeja on Twitter: “@sergeykarayev @goodside @amasad For those getting an error trying to run this, after forking the Repl, you need to insert a secret with
keyOPENAI_API_KEYandvalue= your Open AI key. See screenshot. Brings to mind the Arthur C Clarke quote about sufficiently advanced tech feeling like magic”
- [Wayback/Archive] Julian Bilcke on Twitter: “Combined with code generation GPT-3 is incredibly powerful”
which is part of a cool thread in itself as well.
- [Wayback/Archive] Brendan Dolan-Gavitt on Twitter: “@teuber_dev Ooh, fun! :) I have also in the past given Codex access to a Linux command line, with amusing results”
- [Wayback/Archive] Sergey Karayev on Twitter: “Here’s a brief glimpse of our INCREDIBLE near future. GPT-3 armed with a Python interpreter can · do exact math · make API requests · answer in unprecedented ways Thanks to @goodside and @amasad for the idea and repl! Play with it:
- [Wayback/Archive] a7111a.eth / a7111a.lens on Twitter: “@teuber_dev @moyix if this is for real lol at least we don’t have to fear AI singularity for another 10 years…”
- [Wayback/Archive] Julian Bilcke on Twitter: “@sergeykarayev @goodside @amasad I’ve given GPT-3 access to my terminal to let it run arbitrary shell command (injecting dats in files etc). I wouldn’t recommend it in term of security, but it can produce all kind of interesting things: “
- [Wayback/Archive] Sharif Shameem on Twitter: “@sergeykarayev @goodside @amasad GPT-3 can also be used to control a browser as an “agent” with the correct prompting:”
- [Wayback/Archive] Sharif Shameem on Twitter: “I gave GPT-3 access to Chrome with the objective “please buy me Airpods”. Pretty interesting if you ask me 🤔” (Video plus another interesting thread on where the AI got stuck, how this scales and how to go further)
- [Wayback/Archive] Joe Heitzeberg on Twitter: “@sharifshameem @sergeykarayev @goodside @amasad I build a prototype of this with headless chrome in the cloud, a simple DSL and some logic with prompt chaining… feels like magic”
[Wayback/Archive] ACT-1: Transformer for Actions
When writing this, it works as a debug plugin for Chrome, and I wonder how much it has advanced by now.
- {Wayback/Archive] Anton Bacaj on Twitter: “@sharifshameem And it only took little over one year:”
- [Wayback/Archive] Adept on Twitter: “1/7 We built a new model! It’s called Action Transformer (ACT-1) and we taught it to use a bunch of software tools. In this first video, the user simply types a high-level request and ACT-1 does the rest. Read on to see more examples ⬇️”
- [Wayback/Archive] Adept on Twitter: “2/7 This can be especially powerful for manual tasks and complex tools — in this example, what might ordinarily take 10+ clicks in Salesforce can be now done with just a sentence.”
- [Wayback/Archive] Adept on Twitter: “3/7 Working in-depth in tools like spreadsheets, ACT-1 demonstrates real-world knowledge, infers what we mean from context, and can help us do things we may not even know how to do. “
- [Wayback/Archive] Adept on Twitter: “4/7 The model can also complete tasks that require composing multiple tools together; most things we do on a computer span multiple programs. In the future, we expect ACT-1 to be even more helpful by asking for clarifications about what we want.”
Example with searching through craigslist and following up through gmail.
- [Wayback/Archive] Adept on Twitter: “5/7 The internet contains a lot of knowledge about the world! When the model doesn’t know something, it knows how to just look up the information online (seen here in voice input mode).”
Automatic multi-level searching through Wikipedia.
- [Wayback/Archive] Adept on Twitter: “6/7 ACT-1 doesn’t know how to do everything, but it’s highly coachable. With 1 piece of human feedback, it can correct mistakes, becoming more useful with each interaction.”
Another spreadsheet example, but now with teaching Adept how to fix the formula it generated and then it automatically applying the fix everywhere it had generated the wrong version.
- [Wayback/Archive] Adept on Twitter: “7/7 Read more at
adept.ai/act. We’re only scratching the surface — if you’re as excited about useful general intelligence as we are, apply atadept.ai/careers, or visitadept.ai/alphato join the waitlist for the alpha release of our upcoming product.”funni
- [Wayback/Archive] Adept on Twitter: “1/7 We built a new model! It’s called Action Transformer (ACT-1) and we taught it to use a bunch of software tools. In this first video, the user simply types a high-level request and ACT-1 does the rest. Read on to see more examples ⬇️”
Next decade?
Every generation has the right to their own in-band vulnerabilities (:
- [Wayback/Archive] Kyle Huey on Twitter: “@moyix Every generation needs a SQL-injection-like attack.” / Twitter
- [Wayback/Archive] Brendan Dolan-Gavitt on Twitter: “@khuey_”
- 80s: Secrets of the Little Blue Box
- 90s: Smashing The Stack For Fun And Profit
- 00s: SQL injection
- 10s: ??
- 20s: Prompt hacking
- [Wayback/Archive] Brendan Dolan-Gavitt on Twitter: “@khuey_ OK the dates on these are definitely off (Blue Box is from 1971) but still”
--jeroen
The Wiert Corner - irregular stream of stuff 
Leave a comment