Automation needs a failure path
Automation scales errors as well as useful actions. For every CI check, hook, AI reviewer, or release script, state what happens after failure, where evidence is found, and who decides.
A mature workflow has a controlled stop, a temporary disable path, and a return to a stable state in addition to its happy path.
| Recovery field | Example content |
|---|---|
| Owner | Person responsible for the component |
| Logs | Location of job, hook, or release logs |
| Rerun condition | When a rerun is justified |
| Rollback | How to return to the last working state |
| Disable | Explicit switch or trigger condition |
| Aftercare | What to update after the incident |
Evidence before action
- Find the job or hook that failed.
- Record the command, file, line, and relevant log.
- Check the latest diff and environment differences.
- Assess whether the issue reproduces on rerun.
- Classify the failure before choosing rerun, rollback, or disable.
Rerun, rollback, and disable are different reactions
For an advisory check, disabling only the broken reviewer may be safer than disabling the whole pipeline. Keep the rest of the quality workflow active when the problem is local.
| Action | When appropriate | What not to do |
|---|---|---|
| Rerun | Likely temporary timeout, flaky, or network failure | Repeat a stable regression without new evidence |
| Rollback | A change damaged code, release, or delivery artifact | Rollback blindly without a defined stable point |
| Disable | A broken automation component blocks the correct flow | Comment out large YAML sections and forget the temporary state |
Common classification scenarios
- Flaky, timeout, or network failure: one justified rerun.
- Stable failure after a specific diff: regression analysis and possible rollback.
- Hook blocks a correct action: temporary disable and matcher or handler fix.
- AI review makes a weakly supported critical finding: human check and narrower prompt.
- Wrong release tag or changelog: stop distribution and restore the correct artifact.
- Local green, CI red: compare dependencies, environment, and runner.
After an incident, update more than code
A short postmortem records what happened, how it was detected, what action was taken, and what changed afterward. The needed update may be in QUALITY_GATES.md, a prompt, hook, matcher, logging, or recovery path.
The decision to rollback, disable, or continue a risky operation remains human. This is especially true for publish, deploy, and privilege-bound actions.
## Incident
- What happened:
- Detection signal:
- Evidence:
- Action taken:
- Follow-up change:
- Owner:Canonical lesson source · JavaRush
Open JavaRush material