What I want is to predict the future. I want it for reasons that are no doubt emotionally clear, but I also want it because of my own definition of security: The absence of unmitigatable surprise. - Dan Geer1
This is LLMSec, a newsletter about the security of AI systems.
But why?
NIST recently published their taxonomy of attacks and mitigations of AI systems. They demarcate predictive AI systems from generative AI systems. Predictive AI systems are relatively old and familiar to us. Generative AI, in spite of (or maybe because of) its explosive recent growth, is relatively novel and not well understood.
Based on the title you probably guessed that I will be focusing on LLM/Generative systems, because these are the types of AI systems that more and more people will be interacting with at the everyday level. Predictive AI undergirds much of the internet, but regular people will be working with generative systems every day, and quite soon in my estimation.
When computer systems were initially invented, interacting with them was the domain of specialists and niche hobbyists. Before the invention of Graphical User Interfaces (GUIs), user interfaces were primarily known as Command-Line Interfaces (CLIs) or text-based interfaces. Users had to type specific commands, conforming to a predefined syntax, to perform tasks and navigate the system.
With the invention of the GUI, computing became ubiquitous.
And with LLMs, a new type of interface dubbed by Charles Frye the “Language User Interface” is born. What this means is that we can interact with computing system using our everyday speech. And because of this computing becomes ever more ubiquitous and intimate.
Language deeply conditions our cognition and in term is deeply structured by our cognition. Arrival and “The Story of Your Life” explore the depths to which our thoughts are actually shaped by language - the aliens of the story can in a sense time travel because of how they perceive time. And as Dr. Banks learns their language, she begins to have premonitions of future events and understand cause and effect in a new way.
Due to this intimate relationship between thought and language, these AI systems seem to enmesh themselves in our way of thinking. To be sure they are not omnipotent or even super-intelligent at the moment, but they seem to understand things about us and our world in a way that is wholly unprecedented in computers.
Daniel Miessler writes eloquently about what it looks like when AI systems have become a deep and ubiquitous part of our lives; these enmeshed AIs he dubs Digital Assistants (DAs):
But whether it’s the native ones from the OS, or some combination due to anti-trust and competition, what’s important is that these AIs will know absolutely everything about us.
So, much, access…
They’ll have our health data because that’ll be part of OS integration
Our finances and other sensitive personal data because those will be easy API tie-ins
They’ll have our journals & diaries because wherever we store that stuff, we’ll give the AI access
They’ll know our pasts, our traumas, and our hangups because knowing those will help our Digital Assistant be a better advocate for us
They’ll know our likes and dislikes for foods, conversation topics, sexual preferences/kinks, books and movies, and everything else because—once again—they’ll make the DA a better assistant and, um, friend
The other characteristic of generative AI systems that is salient is their fuzziness. They are statistical and predictive machines and their outputs are non-deterministic for the most part. And because of this capacity their inputs are flexible. What this means is that they can take the messiness of our actual language and make sense of it. To an LLM the words “password” and “paszwurd” are reliably the same2 and an LLM can take imprecise direction and act on it.
This property gives rise to many interesting implications for how we interact with these systems. In a sense engineering with LLMs has moved back to an alchemical point, with disciplines like prompt engineering attempting to wrest gold from the very fabric of language. Indeed at the moment we actually don’t really understand how they work. LLMs are closer to discovered alien artifacts than they are to something engineered like a car. Their capabilities are often emergent.
Now let that sink in. These things that are quickly moving into the hearts of many software systems have a vast unknown (for the moment) space.
And this is where the security people come in.
LLM+Cyber: LLyber?
If I’ve convinced you of how ubiquitous LLMs are likely to become I hope that you recognize the risk they represent. And it is ultimately the role of security people to think about and mitigate risk to the greatest extent possible.
I’ve already hinted at some of the difficulties we’ll have in securing these systems. These are computing artifacts, with hidden and emergent properties, receiving, “understanding” and acting upon, and outputting fuzzy language, hooked up to our most important data and software systems.
When the day that Daniel Miessler predicts dawns, our DAs, or whatever other AI is part of our lives, become the juiciest of targets.
Thankfully that time is a long way off (perhaps…) But in the interim we have to learn to secure systems that are already being hooked up to our medical data, our personal writing, our banks, our logs.
I don’t yet have a full grasp on whether LLMsec will shake out to be an entirely separate discipline within cybersecurity. At the moment its a rich vein of interest to mine. LLMsec seems to be a smattering of application security (securing software when LLMs integrated into larger applications) plus a fair amount of new territory (prompt injection! Sleeper Agents! RAG!)
Without being too prescriptive to my future self, I will be covering all of that security territory and perhaps including material for foundational knowledge about what these systems are, how they work, how they’re built, and what they do.
Who am I?
I’ve been in tech for over a decade and played many roles, from frontend developer (I’m sorry for all the CSS I wrote) to team lead, to PM, to consultant. I’ve worked as and engineer and security person at organizations large and small. I’ve helped secure systems from the power grid and to the cloud.
Since I was young, I have had a passionate curiosity for understanding how things work and how they break. And how to fix them once they do. You’ll never find me so happy as when I’m knee deep in a gnarly bug. And coming out the other side to share that tale lights me up.
Before I wrote software, I worked in advertising and documentary as a video editor. And before that I made and sold cheese for a living and many other things besides. Its been a long strange journey but I hope to use that diversity of experience to bring a unique perspective to the questions we address in this newsletter.
AI is often described as having a transformative impact, ranging from being the most significant invention of our era to potentially the greatest invention in the entire span of human history.
Whether or not that will come to pass is not my prediction to make. What I do care about is considering the risk we have before us and what we can do to manage it.
Thank you for reading and coming along. And please indulge my obsession with 1960’s sci-fi art.
http://geer.tinho.net/geer.recordedfuture.7x15.txt (btw
Given certain caveats about how they tokenize