What is promptware? Origins, definition, dangers, and safeguards
If you’re anyhow close to the IT field nowadays — in the broadest sense of this term — “prompt” is likely something you hear more often than you’d like to. The advent of large language models (LLMs), commonly referred to as AIs, brought this simple word into the limelight, since prompting is how a user tells these models what to do and what the result should be.
The reach of artificial intelligence through all the aspects of our everyday life is growing. Even if you factor in marketing gimmicks that exploit the notion of AI when there is actually none, the spread of the technology is impressive. And it makes things more natural and useful, too: typing or saying requests (prompts) in regular language and having them obeyed/executed is a comfortable experience in most cases.
There is a flip side to this coin, of course. It seems like the smarter the system, the more entry points for wrongdoers it offers, and the more harm they can do once in control. And one of the ways to get there is by deploying a promptware attack.
So, what is promptware?
The term promptware was coined by a group of Chinese researchers in their paper “Promptware Engineering: Software Engineering for LLM Prompt Development” (published on March 4, 2025), which, as the title implies, explores prompting as a coding instrument usable with LLMs, and suggests formalizing promptware engineering into a full-fledged discipline with its own methodology and system.
The menace of promptware attacks
Another recent study, titled “Invitation is All You Need,” showed how a promptware attack actually works. Basically, the team behind this paper managed to trick Google’s Gemini AI into doing a lot of assorted harmful things, from messing with the smart home setup through exploiting the online ecosystem (spamming, generating toxic content, deleting calendar events, exfiltrating emails, etc.) to tracking user’s location. The AI was forced to do all this by a simple calendar invite; turns out, (any?) action the user takes in response to such an event, like telling Gemini to send a thank-you letter, can be a trigger that unlocks malicious instructions embedded in the invite’s title. They say this works with email subject lines, too.
Prompt hacking (injection, hijacking) vs. promptware attack
You may have heard about prompt hacking, injection, or hijacking before (these are largely interchangeable and differ in some rather intricate specifics). How is that technique different from promptware attack?
Prompt hacking is about crafting prompts in a way that allows manipulating an LLM into unintended or harmful behavior. The instructions designed to override/bypass the AI’s built-in restrictions are inserted into the prompt. The term was coined in 2022; today, it is an umbrella label for malicious attempts at making AI models do something they are not supposed to.
Promptware attack describes the type of attacks that exploit natural language prompts used as the programming interface with the purpose of controlling LLM behavior. It is deeper and wider than prompt hacking or injection, and actually can incorporate this technique as one of the components of a cascading attack. Promptware attacks are the next evolutionary step of malicious activities designed to compromise and exploit integrated AI ecosystems.
How to protect yourself against promptware attacks
Prompt injections aren’t a new thing, which means that the developers of AI systems have already come up with defensive solutions capable of fending off promptware attacks, too. Plus, there are new digital hygiene rules that help prevent any unfavorable turn of events.
- Choose AI vendors with a name. Yes, the experiment mentioned above exploits Google’s Gemini AI, but such lessons are typically learned quickly. If you’re looking to outfit your home with an AI-powered assistant, opt for vendors that are meticulous about protecting their clients.
- Shield the system from unauthorized access. Most smart home setups and other consumer-facing systems relying on LLMs allow adjusting the level of security from lax to strict. Go for strict, even if it limits the capabilities of the system in question.
- Shield everything else from the AI. Most software today wants to learn as much as possible about you, since this sort of information can be monetized. LLM-powered systems will likely act in a similar manner, regardless of their intended purpose. Apply the principle of least privilege: limit AI’s access to only the data or controls necessary for the specific task it is deployed to do.
- Monitor AI interactions and behavior. Any worthwhile system keeps logs (if it doesn’t, that’s a red flag); scan them every now and then, and don’t hesitate to at least look up suspicious entries online, or, better, contact support and inquire what’s what.
- Keep only the needed features up and running. In a smart home situation, you may, for example, no longer need an automated cat feeder. Getting rid of the physical device, don’t forget to disconnect it from the AI, too.
In case it’s information that you can lose through a promptware attack, it makes sense to implement a solid backup plan. Read these posts for advice and software recommendations:
