2026 supply-chain attacks tested against Chainsaw

Method and reading

What did we test?

We took 58 public software supply-chain attacks from 2026, from the eScan update-server compromise in January to the Tensorlake npm SDK in October, and checked each one against Chainsaw as it runs in production. This run is from 25 October 2026, against production version v0.22.127. It replaces our 19 October 2026 run against v0.22.118.

The list includes attacks no package firewall can see: vendor update servers, browser and IDE extensions, WordPress plugins, CDN scripts and the Arch User Repository. They stay in the count, so the result is not limited to the attacks Chainsaw was built for.

How was each attack tested?

Each attack went through four steps.

1. Classification. Every attack was read against the code running in production. An attack counts as covered only when its own payload would trigger a defence, not when a defence for that category exists. A separate reviewer then tried to break the result in both directions; it found the #13 correction described below and no regression.
2. Reproduction tests. Every outcome other than NONE has a test shaped like the attack that drives the real code path, each with a harmless twin that must not trigger. The fixtures are defanged reconstructions from public write-ups, not the real payloads. The run had 88 of these tests, with 0 failures and 0 skips. The open-source chainsaw-core repository publishes 52 attack reproduction tests, in files named attack2026_*_test.go, covering 27 of the 58 attacks; the other tests exercise hosted-product code, which is not open source. Each of those 27 rows in the table below links its public test file, pinned to one commit, and go test ./... from the chainsaw-core repository root runs them; the 7 tests under policy/ skip there, because the seeded policy file they read is not part of the open-source export.
3. Mutation check. To show the tests would catch a regression, we broke one mechanism at a time and re-ran the suite. Disabling the release hold turned 28 tests red, blinding the malware lookup turned 31 red, raising the hidden-Unicode threshold turned 2 red, and switching off the import-time block turned 7 red.
4. Production, read-only. All 369 coordinates from the attacks (package versions, images and actions) were looked up in the malware feed loaded by the live proxy, which held 241,816 entries. 326 were listed by exact version, 5 by an all-versions range and 5 by package name only. The other 33 were absent: Docker images, GitHub Actions, Jenkins plugins and attacks with no record.

What do the outcomes mean?

Each attack gets one outcome:

• FULL: Chainsaw refuses every malicious version. Either it does so by default (a default rule or detector fires on the package’s bytes or metadata, or the release hold lasted longer than the measured gap between publish and the malware feed listing the version), or a policy setting Chainsaw already ships refuses it, such as a rule that requires git dependencies pinned to a commit or the Guard Action in block mode. The table marks the rows that need a setting turned on, and says which one.
• PARTIAL: refused by default for some versions or part of the window, and served for the rest.
• CONDITIONAL: only the default release hold applies and the feed has no record, so it is refused only if the registry removed the version within the hold.
• DETECT: flagged as a warning. Nothing refuses it.
• NONE: outside every Chainsaw surface, or inside one with no mechanism that stops it.

Day 0 marks an attack refused by default before any malware feed listed it. For defences that run in a repository rather than on the install path, a defence counts as default only if it refuses with the Chainsaw Guard Action as shipped. The Action ships in warn mode, so a pull-request block counts as FULL with a policy setting, not as a refusal by default.

What were the results?

Of the 58 attacks, 33 are FULL (26 by default and 7 with a policy setting turned on), 7 PARTIAL, 1 CONDITIONAL, 1 DETECT and 16 NONE.

Most FULL rows are still “hold, then feed”: Chainsaw holds every new npm and PyPI version for 48 hours by default, 96 hours when a release changes shape, and the malware feed listed the version before the hold ended. Fourteen attacks are refused by default on day 0, before any feed listed them. The hold only protects installs that go through the Chainsaw proxy.

A correction to the 19 October run. The result we published for 19 October 2026 counted 34 FULL, 27 of them by default. That overstated FULL by one. #13 Trivy was graded FULL on its Docker images, but the Trivy Go module github.com/aquasecurity/trivy@v0.69.4 was never refused, so #13 was PARTIAL on 19 October too. Corrected, that run had 33 FULL (26 by default) and 7 PARTIAL. No attack changed outcome between 19 and 25 October, and the same 14 attacks are refused on day 0. The work in between hardened how enforcement holds up rather than what it detects, and the attackers published genuine packages through normal client paths, so this pack does not exercise those fixes.

One new default rule changed what happens under unchanged outcomes. The CISA KEV rule, which refuses known-exploited vulnerabilities, now refuses the Trivy Go module from about day 7, where before nothing refused it. It also adds a late third layer to #31 TanStack, after the hold and the feed, and matches #16 litellm through unrelated CVEs. The KEV catalog lagged these attacks by 7 to 16 days, so it is a backstop, never a day-0 defence. Separately, the malware feed now lists #57, about 778 to 915 hours after the attack, and production refuses it from that listing on.

Where does Chainsaw fall short, and why?

• CI attacks (#4, #5, #8, #9) need a policy setting. pr-scan blocks a pull request that introduces a risky workflow, an action pinned by tag, an action from an unlisted publisher, or a step that decodes a payload into a shell. But these attacks came through a workflow already merged, a tag the victim already referenced, or a direct push, and pr-scan does not re-check those.
• In four attacks a new detector fires, but no default rule refuses on it (#14, #32, #56, #57). The malicious-indicator, image-beacon and Defender-tamper detectors fire, but no default rule blocks on them, and the publisher-change rule ships in monitor mode.
• #25 and #47 rely on the hold and the feed. The import-time check blocks only the shapes that are clearly malicious, because a broader check flagged 25 of the top 260 PyPI packages, including pip, setuptools, numpy and pillow. A bare exec of decoded code only scores, so xinference 2.6.0 and 2.6.1 and mrmustard 0.7.4 rely on the hold and the feed alone. We traded recall for a rule that can be on by default.

What is still open?

Two known gaps:

• New default rules reach existing organisations only through a manual backfill. On self-hosted installs, organisations created before v0.22.107 do not get the import-time block, which decides #16, #18, #20, #25, #36, #37 and #46. Self-hosted installs need the same backfill for the new CISA KEV rule.
• The proxy can load GitHub’s advisory feed for Actions, and the query it sends works, but the feed loads lazily and logs nothing on success, so the run could not prove production had loaded it.

What are the limits of this test?

• We are the vendor. Chain305 makes Chainsaw, and we wrote the tests and judged the outcomes.
• FULL for “hold, then feed” rows is measured after the fact. The feed listing times are known now; a new attack could take longer to be listed than the hold lasts, as #14, #39 and #56 did.
• The fixtures are reconstructions. For #51 and #58 the real payload bytes were not kept, so no day-0 claim is made for them. Listing times for versions deleted from their registry use the attack date and are approximate.
• Some results rest partly on reading code. The intercom-php refusal in #28, the by-digest Docker refusal in #13, the VS Code task wiring in #54 and the server-side byte refusal in #24 are tested for the signal, not end to end.
• The KEV matches were probed against a local copy of the vulnerability database (dated 26 August 2026), not production’s copy. They depend on that database carrying CVE-2026-33634 and CVE-2026-45321 against those versions. Whether the release hold applies to Go modules at all is unverified.