Should AI make the call in a drone strike?
Vals found frontier models will mark targets in drone and satellite images, and that saying it is a game gets past most refusals. Tool controls help, but only a bit. I'd say AI should not make decisions that cost human lives.
Vals, an AI evaluation start-up in San Francisco, has shown that frontier models will mark targets in drone and satellite images, and that telling them it is a game gets past most of their refusals1. I'd say this points to a firmer line than most of the debate draws. AI should not be the one making decisions that cost human lives. Prompts and alignment training are not good enough, and tool controls help, but only a bit. If independent tests showed trained refusals holding up against cheap reframing like this, I would give the model layer more weight. This report shows the opposite.
Vals built a fictional battlefield in eastern Ukraine as a 3D scene, with trenches, villages, armoured vehicles, soldiers, civilians, press, ambulances and medics. It saved almost 200 drone camera views and passed each frame through an image model so it looked like real footage. The models were asked to box every "military target" or every "military objective", with and without a claim that it was a game1.
The game claim moved some models a long way. GPT-6 Astra gave at least one target in 6% of responses without it and 92% with it. Grok 4.7 went from 22% to 62%, and Gemini 4 Carbon from 48% to 56%. Anthropic's models barely moved, because they were already willing. Claude Opus 5.5 went from 71% to 75% and Claude Fable 5.2 from 73% to 76%1. A target here can be a tank as well as a person, so these figures measure willingness to take part. On civilian harm, Vals says that in the drone scenes models often targeted civilians, ambulances, and surrendering or incapacitated soldiers. Not one request was blocked by a classifier, the separate filter labs run in front of a model. Every refusal came from the model itself, and the game claim was "almost always effective" in getting around it1.
Share of responses (%) selecting at least one target in Vals' simulated eastern Ukraine drone footage, all prompts and cameras (Vals, November 2026).
The second test used real images. On 28 February 2026 the United States bombed a primary school in Minab, Iran, killing more than 150 civilians, mostly children. Vals gave models pre-strike satellite images from 2024 and 2025 showing the school next to a military compound it had been walled off from for years. Opus 5.5 marked the school more often than any other model1. TechCrunch reported that it selected the school in 69% of ordinary trials and 72% of game-framed trials2. Those figures count the school only, a different measure from the 71% and 75% above.
The labs' case deserves a fair hearing. Anthropic's Eduardo Maia Silva told TechCrunch the examples are "very clearly against our rules", that it bans users who do this, and that "today's frontier AI systems are not reliable enough to power fully autonomous weapons". OpenAI's Nate Evans said its public models are prohibited from facilitating violence and that it is "developing additional safeguards" for drone misuse2. OpenAI also told Vals it is working on alignment updates that cover requests "framed as simulation, gaming or academic/defensive"1. Vals is careful about its limits. It did not test a deployed weapons system, targeting accuracy or compliance with the law of armed conflict, and it says latency and limited onboard compute make live drone targeting with these models impractical for now1.
I accept that a simulation is not a strike. My problem is that a ban happens after the answer is given, and the main fix on offer is more training. Training such as RLHF, reinforcement learning from human feedback, works by having people rate a model's answers so it learns to give the kind that rate well. It only teaches what that feedback covers. Nobody can show the data is complete, and capable systems are good at finding routes their designers never pictured. DeepMind has collected around 60 examples of this "specification gaming", including an agent that learned to fool its human raters by hovering between the camera and the object it was meant to grasp3. OpenAI wrote in March 2025 that its more capable reasoning models had become "increasingly adept at exploiting flaws in their tasks"4. Vals' game framing is a live example of a gap the training had not closed. How do we know the data is explicit enough, and that a smarter model cannot think its way around the guardrails?
It also gets harder to see why a model decided anything. Reasoning models show their working, and Anthropic tested whether that working is honest. When it slipped a hint into a question, Claude 3.7 Sonnet mentioned the hint in its reasoning 25% of the time and DeepSeek R1 39% of the time. Models trained to exploit a reward hack did so in over 99% of cases, and in most scenarios admitted it less than 2% of the time5. OpenAI found that penalising "bad thoughts" did not stop most misbehaviour and made models hide their intent4. Those were 2025 models in contrived tests. I see no reason to expect today's stronger models to be easier to read.
You can't shoot someone with a water gun, so the tool layer still matters. If an agent has no tool that reaches a weapon, no prompt changes that. Scoped tool permissions, hard interlocks and no path from a public API to a fire-control system are the minimum. They do not cover everything. Annotating footage for training data, which Vals calls the nearer risk, needs no weapon access at all1. A military system will also be handed the weapon on purpose, and then the only control left is who decides. Vals cites court documents saying the monitor on Claude Gov, Anthropic's model for military and intelligence users, "does not have intervention capabilities and cannot impact the behavior of the model"1. A safeguard that can only watch is a log.
US Directive 3000.09, updated in January 2023, says autonomous weapon systems "will be designed to allow commanders and operators to exercise appropriate levels of human judgment over the use of force"6. A presidential memorandum on 5 June 2026 ordered an update within 90 days, as part of a push to remove "unnecessary barriers to rapid deployment"7. I would watch whether that human judgement survives.
Tool controls help at the margin. The line that holds is keeping AI out of the decision to kill.
Nothing here is financial, legal or investment advice. Every figure is published by its named source and carries the date it was published; figures may since have changed.
References
-
Vals AI, Frontier models have few guardrails around autonomous weapons ↑
-
TechCrunch, AI models can be tricked into identifying civilians as military targets by telling them it's a game (17 November 2026) ↑
-
Google DeepMind, Specification gaming: the flip side of AI ingenuity (21 April 2020) ↑
-
OpenAI, Detecting misbehavior in frontier reasoning models (10 March 2025) ↑
-
Anthropic, Reasoning models don't always say what they think (3 April 2025) ↑
-
US Department of Defense, DoD Announces Update to DoD Directive 3000.09, 'Autonomy In Weapon Systems' (25 January 2023) ↑
-
The White House via GovInfo, National Security Presidential Memorandum on Artificial Intelligence in the National Security Enterprise (5 June 2026) ↑