Green by absence: five ways a pipeline reports success while doing nothing

Across five incidents in four repositories, a green pipeline step told me only that a command had not reported an error. It did not prove that coverage reached Codecov, a package reached npm, a release appeared on GitHub, a search index was cleaned, or CI had run for the pull request.

The repeated mistake was checking the command instead of the effect. A successful call is useful evidence only when the call’s contract includes the result we care about.

Five incidents, five missing checks

1. The coverage upload failed on every run

In AJSF, the Codecov upload step had continue-on-error: true and fail_ci_if_error: false. Codecov rejected uploads with Token required because branch is protected, but the step still showed success.

It affected pull request 361 through pull request 369, nine merged pull requests, before pull request 370 fixed the upload. The badge stopped moving, but the pipeline did not fail.

The check was the upload command’s exit status after its failure had been allowed. The useful check was that Codecov accepted the report. The repair stopped masking failures with continue-on-error and set fail_ci_if_error: true. The current workflow uses GitHub’s short lived OIDC credential, and a failed upload fails the job.

2. A skipped job skipped the publish job too

In a TypeScript monorepo using Changesets and npm OIDC, version 0.1.0 reached main but never reached npm. The publish job was marked skipped, not failed.

The workflow ran on pushes to main and on pull_request events, but the changeset job had a job-level condition that allowed only pull requests. On the push to main, GitHub marked that job skipped. A separate CI gate allowed it to skip, but the release jobs downstream still inherited GitHub’s default skip behaviour. GitHub skips dependent jobs unless their conditions include a status check function, so the release conditions did not run even when the CI gate succeeded.

Three workflow runs made the difference visible: one before the changeset job existed, where publish ran; one where the release path skipped; and one where the changeset job ran but publish still skipped. The fix added !cancelled() so GitHub evaluated the downstream conditions in this graph, then checked the required job result and publish output explicitly. Checking needs.<job>.result alone would not have fixed a skipped dependency.

3. A release planner returned zero for an unknown package

In the same monorepo, @scope/vue@0.1.0 reached npm, but no GitHub release appeared. The publish job was green. A reconcile step had logged planned releases: 0.

Three release scripts each had a copy of the published package list. The new package was added to two lists, but not the third. The planner looked up the package name and silently dropped an unrecognised package. A zero plan was also valid after a successful publish, so the log did not distinguish a normal no op from a missed release.

The repair put the package list in one module imported by all three scripts. A test now requires every listed package to produce a release plan and names any package that would publish without a release.

4. A cleanup job acknowledged the message, not the deletion

A scheduled cleanup job sent one message per target to a queue and exited with status 0 when the broker accepted those messages. A worker performed the deletes later. The green job reported that work was queued, not that the index reached the requested state.

Several inputs and permissions made the distinction matter. One identifier had the right format but selected the wrong kind of record. The worker could not complete every target, so some were left partly populated while smaller ones looked complete.

The delete count appeared in a worker log, but even a nonzero count did not prove completion. The check had to read the target’s final document count after the worker finished. A successful broker publish was only the start of the operation.

5. CI was absent, but the pull request looked like it was waiting

In a backend API, the workflow listened for pull requests targeting master. A pull request targeting a feature branch showed “no checks reported”. Reviewers could read that as pending CI, although the workflow had not run.

There was a second misleading signal: make test-unit was a stub that printed NOT_FOUND. The actual command was npm run test:unit. The pull request had no CI result, and the local command people reached for did not run the tests either.

The workaround was to say in the pull request that CI did not run for that base branch and validate locally with the real command. The workflow needed to run on every pull request, and the Make target needed to invoke the test script rather than print a placeholder.

Two smaller versions of the same problem

In the run that prompted this check, npm view pkg@missing-version exited with status 0 and empty output for a missing version. A check that trusts only that exit status can conclude that the version already exists. The output distinguishes a returned version from an empty result.

I also observed npm publish --dry-run exit with status 0 for a package marked private: true. A dry run is not proof that a real publish would be allowed. A test that needs to protect this property should inspect the manifest field directly.

Check the state where the effect lands

For a pipeline step, I now ask two questions. What state should exist after this step? Which system can report that state?

For coverage, check that the coverage service accepted the report. For a package, query the registry for the exact version. For a release, check the release record. For an asynchronous cleanup, wait for the worker and compare the target with the expected state. For CI, check that the expected workflow actually started on the pull request’s base branch.

A zero result needs a second name. It can mean “nothing needed doing”, or “the input was not recognised”. A pipeline should carry enough information to tell those outcomes apart.

The verification can fail in the same way as the original step. If its error is ignored, or its output is never checked, it only moves the false green one layer deeper. A green status should follow a check of the effect, not stand in for it.

The original Medium version has the earlier framing.