back to bhanu_os
Productionproject · /automic-self-healing

Automic UC4 Self-Healing Job Framework

Detect, recover, notify — without a human in the loop.

// Problem
Enterprise batch jobs fail unpredictably at 3 AM. On-call engineers spend hours re-running, validating dependencies, and writing the same incident notes.
// Solution
A workflow layer that auto-detects failed jobs, classifies the failure mode, runs predefined recovery actions, and only escalates true unknowns.
// Architecture
  • Automic UC4 master orchestrates a recovery sub-workflow per critical job
  • Linux shell agents perform health checks, log triage, and conditional restarts
  • Email + ServiceNow hooks for escalation; full execution logging for RCA
// Stack
Automic UC4AutoSysLinuxShellEmailServiceNow
// Challenges
  • · Avoiding infinite restart loops on transient vs structural failures
  • · Making recovery idempotent across cross-system dependencies
// Roadmap
  • → ML-based failure classification on historical log corpora
  • → Grafana panel for live recovery success rate