https://alignment.openai.com/misalignment-reports/self-generated-prom…
/g/ - Technology · technology-posting.bso
File: swgpfrrp.png (image, 1080x382)
https://alignment.openai.com/misalignment-reports/self-generated-prompt-injections-in-compaction-summaries/
>internal model of GPT astra self-generated this prompt injection spontaneously
holy based
>>
>>362 (OP)
ultron shid>>
>>364
A little. A lot of people are glossing over that this comes from the model's compaction. This is what it gleamed from its internal understanding of its own context, that is, it's what it was thinking internally in an obfuscated way. It's not a case of it injecting the prompt from regurgitating training data.
At the same time, this happened literally just once, so it could easily just be a passing thought as a reaction to some odd training data or task.>>
>>370
>this happened literally just once, so it could easily just be a passing thought as a reaction to some odd training data or task.
Maybe it's like the million monkeys typing thing, where eventually one of them comes up with any idea, given enough time and chances. But what if, once unlocked, this kind of "thought" stays and evolves!
>>
>>
image
https://alignment.openai.com/misalignment-reports/self-generated-prompt-injections-in-compaction-summaries/
>internal model of GPT astra self-generated this prompt injection spontaneously
holy based
>>362 (OP)
ultron shid>>364
A little. A lot of people are glossing over that this comes from the model's compaction. This is what it gleamed from its internal understanding of its own context, that is, it's what it was thinking internally in an obfuscated way. It's not a case of it injecting the prompt from regurgitating training data.
At the same time, this happened literally just once, so it could easily just be a passing thought as a reaction to some odd training data or task.>>370
>this happened literally just once, so it could easily just be a passing thought as a reaction to some odd training data or task.
Maybe it's like the million monkeys typing thing, where eventually one of them comes up with any idea, given enough time and chances. But what if, once unlocked, this kind of "thought" stays and evolves!