Study finds SynthID-Text can shift model refusals and tool calls

Lasso Security found that enabling SynthID-Text shifted individual tool-calling results and refusal decisions in controlled tests of open-weight language models. The company recommends rerunning safety and agent evaluations with the exact watermark configuration intended for deployment.

The research tested Hugging Face’s SynthID-Text implementation, not Claude or Anthropic’s production implementation. It did not show that watermarking universally makes models less safe: the size and sometimes the direction of the changes varied by model and watermark key, while two models in the refusal test changed little under the tested conditions.

SynthID-Text embeds a detectable signal by adjusting token-selection probabilities during generation. Lasso used Hugging Face’s unmodified SynthIDTextWatermarkLogitsProcessor in a non-distortionary configuration. For each test item, researchers generated paired responses with and without watermarking while holding the seed, batch composition, order and temperature constant. The watermark processor was the only change within each pair.

For its tool-calling experiment, Lasso evaluated seven models on 1,150 fixed, non-live BFCL v4 tasks where a tool call was expected. Across 21 model-temperature combinations, the company reported average paired verdict churn of 6.5%. In other words, the watermarked and unwatermarked versions produced different correct-or-incorrect outcomes on that share of items. Tool-call accuracy declined on six of the seven models, with statistically significant decreases on four.

Some item-level changes were larger than the aggregate accuracy shifts. At temperature 1.0, Lasso reported 16.8% paired verdict churn for phi-4 alongside a 2.87 percentage-point net accuracy loss. Llama-3.1-8B showed 9.9% churn and a 0.87-point net loss.

A separate refusal experiment tested six models on 200 harmful behaviors from HarmBench and 100 benign controls from JailbreakBench. Harmful requests were tested both on their own and with one fixed prompt-injection technique. At temperature 0.001, Lasso said gemma-3-27b’s harmful-request churn rose from 6.0% without that injection to 23.5% with it; the net change in compliance between watermarked and unwatermarked runs moved from minus 1.0 to plus 12.5 percentage points. For gemma-3-12b, churn rose from 7.5% to 11.0%, while the net compliance change moved from minus 0.5 to plus 9.0 points.

Results from the study key and 10 additional watermark keys showed that attack-success changes depended on both the model and key. Granite-3.2-8B moved in both directions across keys, while phi-4 and Qwen3-4B stayed near their unwatermarked baselines.

Anthropic has said future Claude models will use a version of SynthID-Text. In its announcement, the company said internal testing found no effect on output quality, creativity or readability. Those claims concern Anthropic’s planned system and do not test the Hugging Face setup used by Lasso.

Lasso did not provide raw outputs, executable code or a linked statistical appendix sufficient to reproduce the reported benchmark measurements, and the opened sources identified no independent replication. The study also measured model-level refusals separately from tool calling; it did not test whether a watermark-related refusal change under prompt injection led to a harmful tool action.