Databricks tests static binary profiles for npm malware triage

Databricks Threat Research tested static binary-analysis profiles for LLM triage of npm packages containing compiled payloads. The company said the added evidence cut escalation failures among seven budget models from 35 of 70 controlled trials to three of 70 and eliminated benign verdicts in that test.

The evaluation focuses on packages that conceal malicious behavior in native binaries rather than JavaScript. Databricks said 432 of 11,413 npm package-versions it scanned and corroborated through OSV malware advisories contained binaries. Those packages failed to reach the study’s escalation threshold in 14.4% of cases, compared with 6.2% for source-only packages.

Databricks’ scanner runs YARA over package archives, Semgrep and registry checks over JavaScript and metadata, and static analysis over compiled artifacts. A capability profile is a weighted playbook for a technique such as credential theft, persistence or network exfiltration. When the extracted evidence crosses a profile threshold, the scanner adds a structured summary of about 1,800 tokens to the model prompt. The model can override that profile.

For the controlled experiment, researchers built five native Node.js add-on payload types. A credential-stealer payload appeared at six obfuscation levels, bringing the total to 10 payloads per model. Databricks evaluated seven budget models and two higher-capability baselines while holding package bytes, prompts, temperature and output limits constant.

A verdict of highly suspicious or malicious counted as a detection. Without profiles, the budget models missed that threshold in 35 of 70 trials and returned 20 benign verdicts. Profiles converted 32 of those misses into detections, leaving three failures. The two higher-capability baselines detected all 10 payloads without profiles at the same threshold.

The researchers also compared production scanner verdicts with OSV labels. Databricks reported 345 true positives and 80 false negatives across 425 OSV-labeled malicious package-versions with native binaries, along with 104 false positives and 2,345 true negatives in a 2,449-version comparison set. The company calculated precision of 0.77, recall of 0.81 and an F1 score of 0.79. The 104 escalations represented 4.25% of comparison package-versions and were concentrated in six of 287 distinct packages.

Those production figures do not isolate binary detection. Databricks said OSV labels identify an entire package rather than the artifact responsible for malicious behavior. Its comparison set treated popular native-binary packages with no OSV listing as benign, although those criteria do not prove that every version was benign. The OpenSSF Malicious Packages repository includes malicious prebuilt binaries within its scope and warns that reports can contain false positives.

Databricks did not release a package-level sample manifest, raw production verdicts, exact prompts, the full scanner implementation or an executable evaluation harness. The company also cautioned that hosted model endpoints change over time, making its results a within-epoch comparison rather than a durable ranking of model performance.