Detect, recover, notify — without a human in the loop.
// Problem
Enterprise batch jobs fail unpredictably at 3 AM. On-call engineers spend hours re-running, validating dependencies, and writing the same incident notes.
// Solution
A workflow layer that auto-detects failed jobs, classifies the failure mode, runs predefined recovery actions, and only escalates true unknowns.
// Architecture
Automic UC4 master orchestrates a recovery sub-workflow per critical job
Linux shell agents perform health checks, log triage, and conditional restarts
Email + ServiceNow hooks for escalation; full execution logging for RCA
// Stack
Automic UC4AutoSysLinuxShellEmailServiceNow
// Challenges
· Avoiding infinite restart loops on transient vs structural failures
· Making recovery idempotent across cross-system dependencies
// Roadmap
→ ML-based failure classification on historical log corpora