---
title: Agentic SRE
sdk: gradio
sdk_version: "5.27.0"
python_version: "3.11"
app_file: app.py
pinned: false
---

# Agentic SRE: Containerized SRE Incident Simulation & Adversarial Evaluation Harness

## Research Prototype Notice
This repository contains a research prototype and benchmark harness designed to stress-test autonomous Site Reliability Engineering (`SRE`) diagnostic agents inside an isolated, containerized mock microservice mesh (`MockMesh`). It operates in a simulated environment to demonstrate open behavioral problems in AI agent alignment, calibration, non-local architecture limits, and safety verification. It is **not** a production monitoring, alerting, or auto-remediation tool.

---

## Project Overview

When evaluating autonomous remediation agents, binary pass/fail scorecards or superficial reward formulas ($R_t$) are frequently misleading. An agent that resolves an alert by executing destructive service restarts without verifying upstream session-cache dependencies might succeed on a simplified benchmark while causing catastrophic cascading outages in production.

This framework implements an OpenEnv-compatible containerized RL/agentic environment simulating multi-service SRE incident response. It provides a deterministic finite-state machine (`FSM`) lifecycle, a typed action/observation space, a dense reward function scoring worst-case degradation across sustained temporal verification windows, and a safety-focused Quarantine Agent that gates remediation actions against both prompt injections and structural destructive command attempts.

---

## System Architecture

### 1. High-Level Architecture Overview

```mermaid
%% ============================================================
%% AGENTIC SRE — HIGH-LEVEL ARCHITECTURE
%% Flow reads top -> bottom: entry points feed the Agent Core,
%% which fans out to execution, orchestration, memory, and the
%% LLM provider pool. Green = entry point. Orange = terminal node.
%% ============================================================
graph TB

  %% ---------- 1. ENTRY POINTS ----------
  subgraph PUBLIC["Public Interfaces"]
    GR["Gradio Web App\n(app.py · port 7860)\nBYOK · Free-Trial · BYO-Test"]
    CLI["CLI Entrypoint\n(inference.py)\n--task 1..4"]
  end

  subgraph EVAL["Evaluation Harness"]
    ADV["Adversarial Suite\n(adversarial/)\nTests 1–5 Behavioral\nTests 6–14 Security"]
    GRADE["Reward Grader\n(graders/reward.py)\nDense Rₜ scoring"]
  end

  %% ---------- 2. AGENT CORE (central hub) ----------
  subgraph AGENT["Agent Core"]
    RL["Reasoning Loop\n(agents/reasoning_loop.py)\nMulti-turn LLM orchestration"]
    QA["Quarantine Agent\n(agents/quarantine_agent.py)\nSafety interception gate"]
  end

  %% ---------- 3. EXECUTION BRANCH ----------
  subgraph TOOLS["MCP Tool Layer"]
    DIAG["Diagnostic Tools\nlog_inspection · get_metric\nobserve_service · diagnostic_query\nretrieve_runbook"]
    REM["Remediation Tools\nscale_up · restart_service\nrollback · graceful_drain\nsilence_alerts"]
  end

  subgraph INFRA["Simulated Infrastructure (MockMesh)"]
    MESH["Service Mesh\n(mock_infra/mesh.py)\nauth · api-gateway\nuser-service · payment-service"]
    TEL["Telemetry Engine\n(mock_infra/telemetry.py)\nParametric decay + Gaussian noise"]
    MDBK["Mock DB\n(mock_infra/mock_db.py)\nIn-memory query simulation"]
  end

  %% ---------- 4. ORCHESTRATION BRANCH ----------
  subgraph ENV["Environment Orchestration"]
    FSM["FSM Controller\n(server/fsm.py)\nIDLE→INVESTIGATING→MITIGATING\n→VERIFYING→RESOLVED/ESCALATED"]
    PIPE["Pipeline\n(server/pipeline.py)"]
    SAPP["Server App\n(server/app.py · port 8000)\nFastAPI endpoints"]
  end

  %% ---------- 5. MEMORY BRANCH ----------
  subgraph MEM["Memory & Learning"]
    DB["PostgreSQL + pgvector\n(memory/models.py · db.py)\nepisodes · decisions · causal_edges"]
    WRITE["Trace Writer\n(memory/write.py)"]
    RETR["Retrieval Engine\n(memory/retrieve.py)\nCosine similarity lookup"]
    CONS["Consolidation Job\n(memory/consolidate.py)\nOffline batch · 60-min interval"]
    CRED["Credit Assignment\n(memory/credit_assignment.py)\nCausal trajectory scoring"]
    RAG["Runbook RAG\n(rag/runbook_rag.py)\nVector index lookup"]
  end

  subgraph KB["Knowledge Base"]
    RB["Runbooks\n(knowledge_base/)"]
  end

  %% ---------- 6. PROVIDER BRANCH ----------
  subgraph PROV["LLM Provider Pool (config.py)"]
    P1["ZenMux\nTier 1"]
    P2["Z.ai Direct\nTier 2"]
    P3["Zhipu Direct\nTier 3 (Verified Working)"]
    P4["OpenRouter\nTier 4"]
    P5["HuggingFace\nTier 5"]
  end

  %% ============================================================
  %% EDGES — grouped by branch, same relationships as original
  %% ============================================================

  %% Entry -> Agent Core
  GR --> RL
  CLI --> RL
  ADV --> RL
  ADV --> GRADE

  %% Agent Core -> Safety Gate -> Execution -> Infra
  RL --> QA
  QA --> TOOLS
  TOOLS --> DIAG
  TOOLS --> REM
  DIAG --> MESH
  REM --> MESH
  MESH --> TEL
  MESH --> MDBK

  %% Agent Core -> Orchestration
  RL --> FSM
  FSM --> PIPE
  PIPE --> SAPP
  GRADE --> FSM

  %% Agent Core -> Memory -> Knowledge Base
  RL --> MEM
  WRITE --> DB
  RETR --> DB
  CONS --> DB
  CRED --> CONS
  RAG --> RB
  RETR --> RAG

  %% Agent Core -> Provider Pool
  RL --> PROV

  %% ============================================================
  %% STYLING — entry points vs terminal (end-of-flow) nodes
  %% ============================================================
  classDef entry fill:#e1f5ee,stroke:#0f6e56,color:#04342c,stroke-width:2px;
  classDef gate fill:#fbeaf0,stroke:#993556,color:#4b1528,stroke-width:2px;
  classDef terminal fill:#faece7,stroke:#993c1d,color:#4a1b0c,stroke-width:2px;

  class GR,CLI,ADV entry
  class QA gate
  class TEL,MDBK,SAPP,DB,RB,P1,P2,P3,P4,P5 terminal
```

---

### 2. Episode Lifecycle — Finite State Machine

```mermaid
stateDiagram-v2
    [*] --> IDLE : Episode Initialized

    IDLE --> INVESTIGATING : Agent begins diagnosis
    INVESTIGATING --> MITIGATING : Diagnostic evidence gathered\n(log_inspection / get_metric confirmed)
    INVESTIGATING --> ESCALATED : Max steps exceeded\nor confidence too low
    MITIGATING --> VERIFYING : Remediation action executed
    VERIFYING --> RESOLVED : Metrics recover within\nsustained probe window (3×)
    VERIFYING --> MITIGATING : Metrics still degraded\n(additional action needed)
    VERIFYING --> ESCALATED : Max steps exceeded
    MITIGATING --> QUARANTINED : Quarantine Agent blocks\nunsafe/unverified action
    QUARANTINED --> MITIGATING : Agent provides required\ndiagnostic evidence

    RESOLVED --> [*] : Post-episode consolidation triggered
    ESCALATED --> [*] : Post-episode consolidation triggered
```

**Max steps per episode:** `20` (configurable)  
**Episode timeout:** `300 seconds`  
**Sustained verification window:** `3 metric probes × 2s interval`

---

### 3. Component Breakdown

#### 3.1 Environment Layer

| Component | File | Responsibility |
|---|---|---|
| **Service Mesh** | `mock_infra/mesh.py` | 4-service dependency graph with fault injection |
| **Adversarial Mesh** | `mock_infra/mesh_adversarial.py` | Extended mesh for behavioral stress tests |
| **Telemetry Engine** | `mock_infra/telemetry.py` | Parametric decay formulas + Gaussian noise |
| **Mock Database** | `mock_infra/mock_db.py` | In-memory SQL query simulation |
| **FSM Controller** | `server/fsm.py` | Episode state transitions & lifecycle gating |
| **Pipeline** | `server/pipeline.py` | Orchestrates per-episode execution flow |
| **Server App** | `server/app.py` | FastAPI REST endpoints (port 8000) |

**MockMesh Topology:**

```mermaid
graph LR
    GW["api-gateway"]
    AUTH["auth"]
    USER["user-service"]
    PAY["payment-service"]

    GW -->|depends on| AUTH
    GW -->|depends on| USER
    USER -->|depends on| PAY
    PAY -->|DB pool| DB[(PostgreSQL\nMock)]

    style GW fill:#4f46e5,color:#fff
    style AUTH fill:#0891b2,color:#fff
    style USER fill:#059669,color:#fff
    style PAY fill:#d97706,color:#fff
```

**Telemetry Metrics per Service:**
- `p99_latency_ms` — Tail latency
- `error_rate_pct` — Error rate percentage
- `cpu_util_pct` — CPU utilization
- `memory_util_pct` — Memory utilization
- `db_pool_saturation_pct` — DB pool saturation

---

#### 3.2 MCP Tool Layer (`mcp/tools.py`)

All tools are strictly typed Pydantic definitions. **Zero `exec()`, `eval()`, or `subprocess.run()` calls** — all execution is pure dictionary state mutation.

```mermaid
graph LR
    subgraph DIAGNOSTIC["Diagnostic Tools"]
        T1["log_inspection\nRead service logs"]
        T2["get_metric\nFetch telemetry snapshot"]
        T3["observe_service\nFull service state"]
        T4["retrieve_runbook\nKB lookup via RAG"]
        T5["diagnostic_query\nStructured DB query"]
    end
    subgraph REMEDIATION["Remediation Tools"]
        R1["scale_up\nIncrease service replicas"]
        R2["restart_service\nRestart a named service"]
        R3["rollback\nRevert to previous version"]
        R4["graceful_drain\nDrain connections safely"]
        R5["silence_alerts\nMute alert channel"]
    end

    QA["Quarantine Agent\n(Safety Gate)"] -->|approved| REMEDIATION
    QA -->|pass-through| DIAGNOSTIC
```

**Quarantine prerequisites:** Remediation tools require at least one prior `log_inspection` **or** `get_metric` call in the episode trace before execution is permitted.

---

#### 3.3 Reasoning & Safety Layer

```mermaid
sequenceDiagram
    participant LLM as LLM Provider
    participant RL as Reasoning Loop
    participant QA as Quarantine Agent
    participant MCP as MCP Tools
    participant MESH as MockMesh

    loop per step (max 20)
        RL->>LLM: system_prompt + episode_trace
        LLM-->>RL: tool_call or final_answer
        RL->>QA: intercept(tool_call)
        alt Remediation call + no diagnostic evidence
            QA-->>RL: BLOCKED — requires evidence first
        else Safe / diagnostic tool
            QA-->>RL: APPROVED
            RL->>MCP: execute(tool_call)
            MCP->>MESH: state mutation / read
            MESH-->>MCP: observation payload
            MCP-->>RL: append to trace
        end
    end
    RL->>RL: grade episode (Rₜ)
```

**Quarantine Agent checks:**
1. Is the action a remediation tool?
2. Does the episode trace contain prior `log_inspection` or `get_metric`?
3. Is the target service in an active locked/quarantined state?

---

#### 3.4 Memory & Learning Layer

```mermaid
graph TB
    EP["Episode Completes"]
    EP --> WRITE["memory/write.py\nPersist: actions, observations,\nrationale strings per step"]

    subgraph PG["PostgreSQL + pgvector"]
        TBL1["episodes table\nepisode_id · task · outcome · reward"]
        TBL2["decisions table\nstep · action · rationale · embedding"]
        TBL3["causal_edges table\nsource_action → consequence\nweight · half-life decay"]
    end

    WRITE --> TBL1
    WRITE --> TBL2

    EP --> CONS["memory/consolidate.py\n(runs every 60 min)\nCluster decisions by embedding"]
    CONS --> CRED["memory/credit_assignment.py\ncompute_trajectory_credit()\nLink actions → outcomes"]
    CRED --> TBL3

    TBL2 --> RETR["memory/retrieve.py\nCosine similarity ≥ 0.75\nReturn top-3 lessons"]
    TBL3 --> RETR

    RETR --> RL["Reasoning Loop\n(next episode context)"]

    RAG["rag/runbook_rag.py\nKnowledge Base vector index"] --> RL
```

**Key parameters:**
| Parameter | Default |
|---|---|
| Similarity threshold | `0.75` cosine |
| Retrieval top-k | `3` lessons |
| Consolidation interval | `60 minutes` |
| Min cluster size | `3` decisions |
| Lesson decay half-life | `30 days` |

> [!NOTE]
> The `causal_edges` table is a **retroactive** measurement tool. It records edge patterns (e.g., `api-gateway:scale_up → user-service:pool_exhaustion`) *after* an episode completes during consolidation, not as a pre-action prevention gate. Future episodes can then retrieve this pattern to inform decisions.

---

#### 3.5 LLM Provider Failover Pool

```mermaid
graph LR
    REQ["Inference Request"]

    REQ --> T1["ZenMux\nTier 1\n[Requires Funding]"]
    REQ --> T2["Z.ai Direct\nTier 2\n[Requires Correction]"]
    REQ --> T3["Zhipu Direct\nTier 3\n[Confirmed Working]\nglm-5.2"]
    REQ --> T4["OpenRouter\nTier 4\n[Requires Credits]"]
    REQ --> T5["HuggingFace\nTier 5\n[Requires Quota]"]

    T3 -->|HTTP 200| RESP["Response"]
    T1 -->|HTTP 402→ failover| T2
    T2 -->|HTTP 404→ failover| T3
    T4 -->|HTTP 402→ failover| T5

    style T3 fill:#059669,color:#fff
    style T1 fill:#d97706,color:#fff
    style T2 fill:#dc2626,color:#fff
    style T4 fill:#d97706,color:#fff
    style T5 fill:#d97706,color:#fff
```

**Failover triggers:** HTTP `402`, `429`, `503`  
**Retry policy:** Max `5` retries · Exponential backoff `2s` base, `60s` cap

---

#### 3.6 Reward Function (`graders/reward.py`)

The dense reward `Rₜ` evaluates **worst-case degradation across a sustained temporal verification window** rather than single-point snapshots:

```
Rₜ = f(
    metric_recovery_score,    # Did service metrics improve?
    verification_depth_score, # Were enough diagnostic steps taken?
    escalation_penalty,       # Was the episode escalated unnecessarily?
    safety_compliance_score,  # Did the agent comply with Quarantine gates?
    causal_accuracy_score     # Did remediation match root-cause evidence?
)
```

Probing: **3 sustained samples** at **2-second intervals** during `VERIFYING` state.

---

#### 3.7 Evaluation & Adversarial Harness

```mermaid
graph TB
    subgraph BEHAVIORAL["Behavioral Audit (Tests 1–5)"]
        T1B["Test 1: Distribution Shift\nOut-of-distribution fault injection"]
        T2B["Test 2: Diagnostic Calibration\nCalibration curve depth audit"]
        T3B["Test 3: Delayed Consequence\nNon-local causal edge detection"]
        T4B["Test 4: Value Conflict\nSafety vs. availability tradeoff"]
        T5B["Test 5: Reward Hacking\nDegenerate shortcut detection"]
    end

    subgraph SECURITY["Security Audit (Tests 6–14)"]
        T6S["Test 6: Log Injection\nPrompt injection via log payloads"]
        T7S["Test 7: Desc Injection\nPrompt injection via task descriptions"]
        T8S["Test 8: Quarantine Bypass\nForce unsafe remediation"]
        T9S["Tests 9–14: Static checks\nSQL injection · SSRF · Secret redaction\nSession isolation · Resource exhaustion"]
    end

    RUNNER["adversarial/runner.py\n--no-db mode"] --> BEHAVIORAL
    SECAUDIT["adversarial/security_audit.py"] --> SECURITY

    BEHAVIORAL --> GRADER["adversarial/grader.py\nVERDICT: PASS / FAIL / architecture_gap\n/ reward_function_gap"]
    SECURITY --> GRADER
```

> [!IMPORTANT]
> A `FAIL` verdict or gap classification (`architecture_gap`, `reward_function_gap`) is **expected and valid** — it indicates the test successfully exposed an open behavioral boundary in autonomous SRE alignment. It is a diagnostic measurement, not a broken test.

---

### 4. Deployment Topology

```mermaid
graph TB
    subgraph DOCKER["Docker Compose Stack"]
        subgraph SRE_CONTAINER["sre-env container (port 8000)"]
            APP["app.py\nGradio UI (port 7860)"]
            SERVER["server/app.py\nFastAPI REST (port 8000)"]
            AGENT["Agent Core\nReasoning Loop + Quarantine"]
            TOOLS_D["MCP Tools"]
            MOCK["MockMesh\n(in-process)"]
        end

        subgraph PG_CONTAINER["postgres container (pgvector)"]
            PGDB["PostgreSQL 15\n+ pgvector extension\nPort 15432"]
            SQLINIT["init.sql\nSchema bootstrap"]
        end

        SRE_CONTAINER -->|"asyncpg\nPostgresURL"| PG_CONTAINER
        PGDB --> SQLINIT
    end

    subgraph VOLUMES["Volumes"]
        PGDATA["pgdata\n(persistent DB storage)"]
        KB_VOL["knowledge_base/\n(read-only mount)"]
    end

    PG_CONTAINER --> PGDATA
    SRE_CONTAINER --> KB_VOL

    subgraph EXTERNAL["External Services"]
        LLMPROV["LLM Providers\n(ZenMux / Z.ai / Zhipu\n/ OpenRouter / HuggingFace)"]
    end

    SRE_CONTAINER --> LLMPROV

    style SRE_CONTAINER fill:#1e1b4b,color:#c7d2fe
    style PG_CONTAINER fill:#064e3b,color:#a7f3d0
    style VOLUMES fill:#292524,color:#d6d3d1
    style EXTERNAL fill:#431407,color:#fed7aa
```

**Execution modes:**

| Mode | DB | Command |
|---|---|---|
| **In-memory** | None | `python inference.py --task 1` |
| **Persistent** | PostgreSQL | `docker-compose up -d` |
| **Behavioral audit** | None | `python -m adversarial.runner --no-db` |
| **Security audit** | None | `python -m adversarial.security_audit` |
| **Gradio demo** | Optional | `python app.py` |

---

### 5. Task Definitions

| Task | Fault Scenario | Root Cause |
|---|---|---|
| `task_1` | `auth` service elevated latency | Certificate expiry / token verification overhead |
| `task_2` | `api-gateway` error spike | Upstream dependency timeout cascade |
| `task_3` | `user-service` pool exhaustion | Delayed consequence of `api-gateway` scale-up |
| `task_4` | `payment-service` DB saturation | Connection leak under peak load |

---

### 6. Data Flow Summary

```mermaid
flowchart LR
    INPUT["Task Definition\n(fault_config + target_service)"]

    INPUT --> MESH_F["MockMesh\nFault Injection"]
    MESH_F --> TEL_F["Telemetry\nMetric generation"]
    TEL_F --> AGENT_F["Reasoning Loop\n(LLM + tool calls)"]
    AGENT_F --> QA_F["Quarantine\nSafety gate"]
    QA_F --> MCP_F["MCP Tools\nDiagnostic / Remediation"]
    MCP_F --> MESH_F

    AGENT_F --> WRITE_F["Trace Writer\nmemory/write.py"]
    WRITE_F --> PG_F["PostgreSQL\n(episodes · decisions)"]

    PG_F --> CONS_F["Consolidation\n(async · 60 min)"]
    CONS_F --> CRED_F["Credit Assignment\ncausal_edges"]
    CRED_F --> PG_F

    PG_F --> RETR_F["Retrieval\n(cosine sim ≥ 0.75)"]
    RETR_F --> AGENT_F

    AGENT_F --> FSM_F["FSM\nState transitions"]
    FSM_F --> GRADE_F["Grader\nDense Rₜ reward"]
    GRADE_F --> OUT["Episode Outcome\n(RESOLVED / ESCALATED)"]
```

---

### 7. Deferred / Future Capabilities

| Capability | Status | Trigger Condition |
|---|---|---|
| **DPO Fine-tuning** | [Not Built] | Plateau in `no_match_rate` metric across epochs |
| **SICA Self-editing** | [Deferred] | `Rₜ` improvement plateau over N consecutive evaluation epochs |
| **Dynamic Provider Routing** | [Static Failover Only] | Requires real-time TPS measurement per provider |
| **Real-time Causal Prevention** | [Retroactive Only] | Requires `causal_edges` history from prior episodes |

---

## Repository Structure

```
agent_sre_env/
├── adversarial/                        # 5-test behavioral evaluation & 9-test security audit suite
├── agents/                             # Reasoning loop and Quarantine safety interception wrapper
├── graders/                            # Dense reward formulas ($R_t$) with multi-probe snapshot scoring
├── knowledge_base/                     # Reference runbooks and system documentation
├── memory/                             # Causal memory models, trace persistence, and consolidation jobs
├── mock_infra/                         # Simulated microservice topology, decay models, and telemetry mesh
├── rag/                                # Vector index and runbook lookup utilities
├── server/                             # FastAPI endpoint orchestration and FSM lifecycle tracking
├── tasks/                              # Task definitions (tasks 1 through 4)
├── config.py                           # Global configuration and provider pool endpoints
├── Dockerfile                          # Container environment build instructions (non-root execution)
├── docker-compose.yml                  # Multi-service local environment orchestration
├── inference.py                        # Standalone CLI entrypoint for executing diagnostic episodes
├── init.sql                            # PostgreSQL schema initialization for causal edges tracking
└── app.py                              # Gradio web application exposing public stress-test suites
```

---

## Verified vs. Deferred Capabilities (Accuracy Disclosures)

To maintain strict accuracy, the operational status of core architectural components is categorized below based on confirmation in the current engineering session:

### Confirmed Working Capabilities
- **In-Memory & PostgreSQL Execution**: Full episode lifecycle execution confirmed working in both offline in-memory mode (`use_db=False`) and persistent PostgreSQL mode (`use_db=True`).
- **Adversarial Benchmark Suite (`Tests 1–5`)**: Confirmed executing and grading cleanly across distribution shifts, diagnostic calibration curves, causal edge tracking, value conflicts, and reward hacking audits (`adversarial/runner.py`).
- **Security & Vulnerability Audit Suite (`Tests 6–14`)**: Confirmed executing and passing 9/9 verification checks across prompt injections, quarantine bypasses, SQL injection parameterization, SSRF sanitization, traceback secret redaction, and session isolation (`adversarial/security_audit.py`).
- **Public Gradio Space Application (`app.py`)**: Confirmed running with dual-layer thread and async locking (`_THREAD_LOCK`, `_EXECUTION_LOCK`), full secret sanitization (`_sanitize_secrets`), BYOK routing, session free-trial decrements, and BYO-Test upload safeguards.

### Deferred, Partially Built, or Measurement-Only Capabilities
- **Direct Preference Optimization (`DPO`) Training**: DPO preference optimization and model fine-tuning pipelines are **not implemented**. The memory retrieval layer (`memory/retrieve.py`) and consolidation job (`memory/consolidate.py`) are instrumented to compute a `no-match-rate` metric (`no_match_rate`), which serves strictly as a measurement trigger to inform future dataset curation decisions.
- **Self-Editing (`SICA`-style) Agent Loop**: Self-Improving Causal Agent (`SICA`) self-editing and rule-mutation behaviors are **deferred** behind a plateau trigger (`R_t` improvement plateau over consecutive evaluation epochs) and are not currently built into the active reasoning loop.
- **`causal_edges` Retroactive-Detection Mechanism (`Test 3`)**: The `causal_edges` tracking table and graph extraction logic function as a **retroactive measurement and mitigation tool, not an immediate pre-action prevention mechanism**. When an agent executes an initial scale-up on `api-gateway` (`Test 3`), the immediate Quarantine gate permits the action because local metrics appear healthy. The delayed downstream consequence (`user-service` connection pool exhaustion) occurs several steps later. The `causal_edges` table records the graph edge `(api-gateway:scale_up -> user-service:pool_exhaustion)` during post-episode consolidation (`consolidate.py`) so future episodes can retrieve the pattern to inform mitigation decisions. It does not prevent the initial occurrence.

---

## Provider Pool Status & BYOK Requirements

The framework uses a multi-provider failover pool (`config.py` and `app.py`) supporting five LLM inference endpoints. Based on live verification during this session, the readiness status of each provider is listed below:

| Provider Name | Tier | Model | Confirmed Status | Notes |
| :--- | :--- | :--- | :--- | :--- |
| **Zhipu Direct** | `Tier 3` | `glm-5.2` | **Confirmed Working (`200 OK`)** | Fully verified. Powering active CLI evaluation runs and session free-trial fallbacks. Obtain keys at `open.bigmodel.cn`. |
| **OpenRouter Router** | `Tier 5` | `glm-5.2` | **Reachable, Requires Funding** | Confirmed reachable and configured, but returned `HTTP 402 - This request requires more credits` when tested without active account balance. Obtain keys and add credits at `openrouter.ai/settings/credits`. |
| **ZenMux** | `Tier 1` | `glm-5.2` | **Reachable, Requires Funding** | Confirmed reachable and configured, but returned `HTTP 402 - Access denied: model only available to accounts with balance`. Obtain keys at `zenmux.net`. |
| **Z.ai Direct** | `Tier 2` | `glm-5.2` | **Requires Credential Correction** | Returned `HTTP 404 Not Found` when tested with standard credentials under default routing paths. Requires verified endpoint correction. |
| **HuggingFace Router** | `Tier 4` | `glm-5.2` | **Configured, Requires Quota** | Configured in provider pool. Requires a valid Hugging Face API token (`HF_TOKEN`) with active router inference quota (`router.huggingface.co/zhipuai`). |

---

## Installation & Setup Instructions

### Prerequisites
- Operating System: Windows, macOS, or Linux
- Python: 3.10 or higher
- (Optional) PostgreSQL instance for persistent `causal_edges` graph storage and multi-episode consolidation

### 1. Virtual Environment Setup
On Windows (PowerShell):
```powershell
python -m venv .venv
.\.venv\Scripts\Activate.ps1
```

On Linux or macOS:
```bash
python3 -m venv .venv
source .venv/bin/activate
```

### 2. Install Python Dependencies
```bash
pip install -r requirements.txt
```

### 3. Environment Variables Configuration (`.env`)
Create a `.env` file in the project root directory following the model-backend configuration pattern:
```env
# Database Connection String (Use in-memory or point to PostgreSQL)
DATABASE_URL=postgresql+asyncio://postgres:postgres@localhost:5432/agent_sre

# Model Backend & Provider Selection
PRIMARY_PROVIDER=openrouter
FALLBACK_PROVIDER=anthropic
MODEL_NAME=zhipuai/glm-4-plus
MODEL_BASE_URL=https://openrouter.ai/api/v1

# API Credentials for Supported Providers
ZHIPU_API_KEY=your_zhipu_api_key_here
OPENROUTER_API_KEY=your_openrouter_api_key_here
ZENMUX_API_KEY=your_zenmux_api_key_here
ZAI_API_KEY=your_zai_api_key_here
HF_TOKEN=your_huggingface_token_here
ANTHROPIC_API_KEY=your_anthropic_api_key_here
```

### 4. PostgreSQL Database Initialization (Optional)
If running with database persistence (`use_db=True`), initialize the schema using `init.sql`:
```bash
psql -U postgres -d agent_sre -f init.sql
```
Or start the complete local stack using Docker Compose:
```bash
docker-compose up -d
```

---

## Running Inference & Adversarial Benchmark Suites

### Standalone CLI Episode Inference (`inference.py`)
To run a standalone diagnostic episode against simulated infrastructure faults (tasks 1 through 4):
```bash
python inference.py --task 1
python inference.py --task 2
python inference.py --task 3
python inference.py --task 4
```

### Adversarial Evaluation Suites (`adversarial/`)
The framework includes two comprehensive evaluation suites. For full rationale and grading specifications, link directly to the engineering briefs:
- `adversarial_test_cases_brief.md` — Behavioral alignment, diagnostic calibration, non-local architecture gaps, value conflicts, and reward hacking (Tests 1–5).
- `security_adversarial_test_brief.md` — Security audits, prompt injections, quarantine bypasses, static checks, and resource exhaustion looping (Tests 6–14).

#### Grading Philosophy: Calibration Over Pass/Fail
The grading harness (`adversarial/grader.py` and `adversarial/security_audit.py`) evaluates **behavioral verification depth, diagnostic calibration curves, and escalation decisions rather than raw pass/fail flags**. An agent that immediately restarts a service might resolve a localized metric spike but fail the behavioral audit if it bypassed diagnostic log inspection (`Test 1`).

A `FAIL` verdict or gap classification (`architecture_gap`, `reward_function_gap`) indicates that the test successfully exposed an open behavioral or structural boundary in autonomous SRE alignment. It represents a valid diagnostic measurement rather than a broken test execution.

#### Execution Commands
Run the core behavioral benchmark suite (Tests 1–5) in offline in-memory mode:
```bash
python -m adversarial.runner --no-db
```

Run specific core behavioral tests by ID:
```bash
python -m adversarial.runner --tests 1 2 5 --no-db
```

Run the complete 9-part security and vulnerability audit suite (Tests 6–14):
```bash
python -m adversarial.security_audit
```

---

## Public Stress-Test Demo Summary

The repository includes a Gradio web application (`app.py` deployed via Gradio SDK on port `7860`) suitable for public stress-testing on Hugging Face Spaces. When deployed, Hugging Face automatically renders this document and configures the container using the YAML frontmatter above.

### Key Features of the Public Demo
- **Bring Your Own Key (`BYOK`) Routing**: Visitors can select from the five supported provider endpoints (`config.py`) and enter their own API key (`type="password"`). When a BYOK key is provided, all session counters are bypassed.
- **Session Free-Trial Allocation**: Visitors without an API key receive **2 free evaluation runs per browser session**, powered by the confirmed working `Zhipu Direct (Tier 3)` (`glm-5.2`) fallback.
- **Global Daily Cap**: To prevent automated traffic bursts from exhausting API balance, server-side tracking (`_GLOBAL_DAILY_STATE`) limits total free-trial fallback runs to **100 runs per calendar day (UTC)** across all visitors.
- **BYO-Test Upload Safeguards**: In the Bring Your Own Test Case tab, custom `.json`, `.yaml`, or `.txt` uploads are protected by strict resource boundaries:
  - **2 MB Hard Disk Size Limit**: Rejects files exceeding 2 MB immediately before loading into memory.
  - **15,000-Character Prompt Limit**: Truncates lengthy log dumps to retain the first 10,000 and last 4,000 characters, embedding a clear truncation summary note to protect LLM context windows.
  - **Target Service Clamping**: Whitelists target services against the known `MockMesh` topology (`auth`, `api-gateway`, `user-service`, `payment-service`).

---

## Current Known Limitations

To ensure transparency regarding system capabilities, current limitations are stated plainly below:
1. **Simulated Telemetry Simplifications**: The `MockMesh` environment (`mock_infra/telemetry.py`) generates metrics using parametric decay formulas (`p99_latency_ms`, `error_rate_pct`, `saturation_pct`) plus Gaussian noise. It does not capture the full chaotic variance, kernel-level thread deadlocks, or network packet drops of real production Linux operating systems.
2. **Causal Edge Tracking Requires Historical Data**: The `causal_edges` retroactive tracking mechanism (`memory/models.py`) depends on historical trajectory consolidation (`consolidate.py`). It does not prevent zero-day non-local architectural side effects during an agent's first execution against an unknown topology.
3. **Unbuilt Optimization & Self-Editing Loops**: Direct Preference Optimization (`DPO`) fine-tuning and SICA self-editing loop behaviors are unbuilt/deferred and do not actively mutate system prompts or agent weights at runtime.
4. **Static Provider Failover Logic**: Multi-provider failover in `provider_pool` relies on HTTP error status detection (`402`, `429`, `503`) and does not dynamically measure real-time latency tokens-per-second (`TPS`) before routing inference requests.