# APEX OS — Security Policy

> Security is not a feature. It is a fundamental property of the system.

## Table of Contents
1. [Principles](#principles)
2. [Secrets Management](#secrets-management)
3. [Authentication & Access](#authentication--access)
4. [Container Security](#container-security)
5. [Network Security](#network-security)
6. [Backup & Recovery](#backup--recovery)
7. [Audit & Logging](#audit--logging)
8. [Incident Response](#incident-response)
9. [Credential Rotation](#credential-rotation)
10. [Human Approval Gates](#human-approval-gates)
11. [Recovery Procedures](#recovery-procedures)
12. [Compliance Checklist](#compliance-checklist)
13. [Change History](#change-history)

---

## Principles

1. **Defense in Depth** — Multiple layers of security, never rely on a single control
2. **Least Privilege** — Every service gets minimum required access
3. **Secrets Never in Code** — All credentials in Vaultwarden or environment variables
4. **Assume Breach** — Design for containment and rapid recovery
5. **Audit Everything** — Every access, change, and decision is logged
6. **Human in the Loop** — Security-critical changes require human approval

---

## Secrets Management

### Vaultwarden (Primary Secrets Store)
- **URL:** https://vaultwarden.apex.unstuck-path.com
- **Purpose:** Store all API keys, tokens, passwords
- **Access:** Admin panel protected by strong password
- **Backup:** Included in daily automated backup

### Secret Categories

| Category | Storage | Rotation |
|----------|---------|----------|
| SSH Keys | /root/.ssh/ (VPS) | Annually |
| Database Passwords | .env file (chmod 600) | 90 days |
| API Keys (LiteLLM) | .env + Vaultwarden | 60 days |
| Gitea Tokens | Vaultwarden | 90 days |
| Grafana Admin | Vaultwarden | 90 days |
| n8n Credentials | Vaultwarden | 90 days |
| Telegram Bot Token | Vaultwarden | As needed |
| OpenRouter API Key | .env + Vaultwarden | 90 days |

### Rules
- Never commit secrets to Git repositories
- Never log secrets (mask in all output)
- Never transmit secrets in plain text over the network
- Never store secrets in container images
- .env file must be chmod 600 (owner read/write only)

---

## Authentication & Access

### SSH Access
- **Method:** Key-only authentication (password disabled)
- **Key Location:** Engineer SSH key managed separately
- **Root Login:** Allowed via key only
- **Port:** Standard (22)
- **Failed Login:** Monitored via auth.log

### Service Authentication

| Service | Auth Method | Notes |
|---------|------------|-------|
| LiteLLM | API Key (LITELLM_MASTER_KEY) | Required for all API calls |
| Gitea | Username/Password + API Token | Token for automation |
| Grafana | Username/Password | Admin access restricted |
| n8n | Username/Password | Admin account only |
| Letta | No auth (internal only) | Not exposed externally |
| Ollama | No auth (internal only) | Not exposed externally |
| Traefik Dashboard | BasicAuth | Rate-limited |
| Dockge | Username/Password | Docker management |
| Vaultwarden | Master Password | Encrypted vault |

### External Access Points
Only these services are accessible from the internet (via Traefik):
- openwebui.apex.unstuck-path.com
- n8n.apex.unstuck-path.com
- gitea.apex.unstuck-path.com (or git.apex.unstuck-path.com)
- grafana.apex.unstuck-path.com
- code-server.apex.unstuck-path.com
- vaultwarden.apex.unstuck-path.com
- status.apex.unstuck-path.com
- dockge.apex.unstuck-path.com
- langfuse.apex.unstuck-path.com

All external access is HTTPS via Let's Encrypt certificates (auto-renewed by Traefik).

---

## Container Security

### Docker Standards
- **No privileged containers** — None run with `--privileged`
- **Read-only filesystems** where possible
- **No host network mode** — All containers on Docker bridge networks
- **Resource limits** — Memory and CPU limits recommended
- **Health checks** — Required on all containers
- **Restart policy** — `unless-stopped` for production services
- **Docker Socket** — Mediated by Docker Socket Proxy (read-only for most services)

### Docker Socket Proxy
- **Container:** apex-socket-proxy
- **Purpose:** Mediates Docker API access
- **Read-Only Access:** Most services (monitoring, status API)
- **Read-Write Access:** Dockge only (for container management)
- **Forbidden:** No container can directly mount /var/run/docker.sock except Socket Proxy

### Image Security
- Use official images from Docker Hub
- Pin image versions (avoid `latest` in production when possible)
- Scan images for vulnerabilities periodically
- Update images during controlled maintenance windows

---

## Network Security

### Docker Networks
- **apex_apex-net** — Primary internal network for all services
- All inter-service communication happens on this network
- External access only through Traefik reverse proxy

### Firewall Rules
- Port 22 (SSH): Open
- Port 80 (HTTP): Open (redirects to HTTPS)
- Port 443 (HTTPS): Open
- All other ports: Closed to external access
- Internal Docker ports accessible only within apex_apex-net

### TLS/SSL
- All external traffic encrypted via HTTPS
- Certificates managed by Traefik + Let's Encrypt
- Auto-renewal configured
- HSTS headers recommended

---

## Backup & Recovery

### Automated Backups
- **Schedule:** Daily at 2:00 AM UTC
- **Script:** /opt/apex/scripts/backup.sh
- **Retention:** 7 days
- **Contents:**
  - docker-compose.yml
  - .env file
  - All Docker volumes (PostgreSQL data, Gitea repos, Grafana dashboards)
  - Configuration files
  - SSL certificates

### Backup Storage
- **Location:** /opt/apex/backups/
- **Format:** Compressed tar archives
- **Naming:** apex-backup-YYYYMMDD-HHMMSS.tar.gz

### Recovery Procedures

#### Single Service Recovery
```bash
# Stop the service
docker-compose stop {service}

# Restore volume from backup
tar -xzf /opt/apex/backups/{backup-file} -C /

# Restart the service
docker-compose up -d {service}
```

#### Full System Recovery
```bash
# Stop all services
docker-compose down

# Restore all volumes
tar -xzf /opt/apex/backups/{backup-file} -C /

# Restore configuration
cp backup/.env /opt/apex/.env
cp backup/docker-compose.yml /opt/apex/docker-compose.yml

# Start all services
docker-compose up -d

# Verify health
curl http://localhost:3100/health
```

### Auto-Recovery
- **Script:** /opt/apex/scripts/auto-recovery.sh
- **Schedule:** Every 5 minutes via cron
- **Actions:** Checks container health, restarts unhealthy/stopped containers
- **Logging:** All actions logged to apex.recovery_log
- **Escalation:** If container fails 3+ restarts, alert via Telegram

---

## Audit & Logging

### Decision Audit Trail
Every significant action is logged:
- **apex.engineer_decisions** — All planning and deployment decisions
- **apex.reflections** — Post-action analysis
- **apex.constitution_violations** — Any rule violations (should be 0)
- **apex.lifecycle_executions** — Project lifecycle tracking
- **apex.task_status_changes** — Task state transitions

### System Logs
- **Loki** — Centralized log aggregation
- **Promtail** — Log shipping from all containers
- **Grafana** — Log visualization and search
- **Docker logs** — Container-level logging

### Monitoring
- **Prometheus** — Metrics collection
- **Grafana** — Dashboard visualization (12 panels)
- **APEX Status API** — Real-time health checks for 6 critical services
- **Auto-Recovery** — Automated container health monitoring

---

## Incident Response

### Severity Levels

| Level | Description | Response Time | Action |
|-------|-------------|---------------|--------|
| P0 - Critical | Data loss, security breach, total outage | Immediate | Halt + Human CEO alert |
| P1 - High | Service outage, failed recovery | 5 minutes | Auto-recovery + Telegram alert |
| P2 - Medium | Performance degradation, single service issue | 15 minutes | Auto-recovery, CEO Agent notified |
| P3 - Low | Minor issue, cosmetic, non-impacting | Next business cycle | Logged for review |

### Incident Response Steps
1. **Detect** — Auto-recovery or monitoring catches the issue
2. **Contain** — Isolate affected service if necessary
3. **Diagnose** — Check logs, identify root cause
4. **Fix** — Apply correction (backup first)
5. **Verify** — Confirm fix resolves the issue
6. **Document** — Log to engineer_decisions and reflections
7. **Learn** — Update knowledge base to prevent recurrence

### Emergency Contacts
- **Human CEO:** Telegram Chat ID 1775182448
- **Emergency Bot:** @JimmysalesBot sends critical alerts

---

## Credential Rotation

### Schedule

| Credential | Rotation Period | Last Rotated | Next Due |
|------------|----------------|--------------|----------|
| LiteLLM Master Key | 60 days | Phase 1 | As scheduled |
| Gitea API Token | 90 days | Phase 1 | As scheduled |
| Grafana Admin | 90 days | Phase 1 | As scheduled |
| n8n Admin | 90 days | Phase 3 | As scheduled |
| Dockge Admin | 90 days | Phase 3 | As scheduled |
| Vaultwarden Admin | 90 days | Phase 1 | As scheduled |
| SSH Keys | Annual | Phase 1 | As scheduled |

### Rotation Procedure
1. Generate new credential
2. Update Vaultwarden entry
3. Update .env file (if applicable)
4. Restart affected services
5. Verify services are operational
6. Log rotation in engineer_decisions
7. Update this table

---

## Human Approval Gates

These actions ALWAYS require human CEO approval via Telegram:

| Action | Reason |
|--------|--------|
| Production deployment | Irreversible system change |
| New tool installation | Security and stability risk |
| External API registration | Cost and security implications |
| Marketing spend | Financial commitment |
| Employee hiring | Workforce change |
| Infrastructure changes | Stability risk |
| Credential rotation | Access change |
| Data deletion | Irreversible |
| Security policy changes | Governance |
| Constitution amendments | Foundational rules |

---

## Recovery Procedures

### Docker Compose Recovery
```bash
# If docker-compose.yml is corrupted
cp /opt/apex/docker-compose.yml.backup.{latest} /opt/apex/docker-compose.yml
docker-compose up -d
```

### Database Recovery
```bash
# Stop PostgreSQL
docker-compose stop postgres

# Restore from backup
docker exec apex-postgres psql -U apex -d apex < /opt/apex/backups/db-backup.sql

# Restart
docker-compose start postgres
```

### Complete System Recovery
1. Re-provision VPS (if needed)
2. Install Docker and Docker Compose
3. Restore /opt/apex/ from backup
4. `docker-compose up -d`
5. Verify all services via Status API
6. Test Telegram bot connectivity

---

## Compliance Checklist

Run this checklist monthly:

- [ ] .env file permissions are 600
- [ ] SSH password authentication is disabled
- [ ] No containers running in privileged mode
- [ ] Traefik dashboard has BasicAuth
- [ ] Daily backups running successfully
- [ ] Auto-recovery script active
- [ ] No plaintext credentials in Git repositories
- [ ] All external services behind HTTPS
- [ ] Vaultwarden accessible and synced
- [ ] Constitution violations count = 0
- [ ] Docker images up to date (no critical CVEs)
- [ ] Credential rotation on schedule

---

## Change History

| Date | Version | Author | Changes |
|------|---------|--------|---------|
| Phase 1 | 1.0 | Engineer | Initial hardening (SSH, .env, Traefik) |
| Phase 5.5 | 1.1 | Engineer | Added auto-recovery, security audit tool |
| Phase 7.1 | 2.0 | Engineer | Comprehensive security policy formalization |