Content
# Vault Backup & Restore Guide
## Overview
Automated daily backups of the entire Datavault server stack using **BorgBackup + borgmatic**, synced offsite to **Google Drive** via rclone. Backups are encrypted, compressed, deduplicated, and versioned.
**Established:** 2026-03-05
---
## Backup Details
| Item | Value |
|---|---|
| Server | Datavault Hetzner (89.167.102.34 / 100.113.209.23) |
| Schedule | Daily at 3 AM UTC + hourly during business hours (see below) |
| Tool | BorgBackup 1.2.8 + borgmatic 2.1.2 |
| Local repo | `/srv/backups/borg` |
| Offsite | Google Drive (`VaultBackups/borg`) via rclone |
| Compression | zstd level 6 (auto-skip for already-compressed files) |
| Encryption | repokey (AES-256) |
| Passphrase | `VaultBorg2026` |
| Borg key export | `/srv/backups/borg-key-export.txt` |
### Schedule
| Run | When | Slack Notification |
|---|---|---|
| **Hourly** | Every hour 8 AM - 6 PM UTC, Mon-Fri | Failure only |
| **Daily** | 3:00 AM UTC, every day | Success and failure |
Hourly backups are lightweight incrementals — typically 1-5 MB thanks to Borg deduplication. They provide fine-grained recovery points during the workday.
### Retention Policy
| Period | Count | Coverage |
|---|---|---|
| Hourly | 10 | Today's business hours |
| Daily | 30 | Last month |
| Weekly | 52 | Last year |
| Monthly | 36 | Last 3 years |
### Integrity Checks
| Check | Frequency |
|---|---|
| Repository structure | Weekly |
| Archive metadata | Weekly |
| Full data verification (every chunk) | Monthly |
---
## What's Backed Up
### Data
- `/srv/obsidian/Vault` — All Obsidian vault files (notes, attachments, plugins)
- `/srv/backups/dumps/vaultdb.sql.gz` — PostgreSQL database dump (search index)
- `/srv/backups/dumps/couchdb-info.json` — CouchDB database metadata
- `/srv/backups/dumps/couchdb-config.json` — CouchDB server config
### Custom Code
- `/opt/vault/api/` — FastAPI web UI and API
- `/opt/vault/ingestors/` — Gmail, Slack, reMarkable, Google Drive, Google Docs ingestors
- `/opt/vault/indexer/` — Search indexer
- `/opt/vault/scripts/` — Maintenance scripts (stray images, etc.)
- `/opt/vault/taskqueue/` — Redis job queue
- `/opt/vault/entities/` — Data models
- `/opt/vault/app/` — Application code
- `/opt/vault/config/` — Service configs (Slack tokens, reMarkable config, etc.)
- `/opt/vault/state/` — Sync state files
### Infrastructure
- `/opt/livesync-bridge/` — Obsidian LiveSync bridge (Deno)
- `/opt/couchdb/etc/` — CouchDB configuration
- `/etc/caddy/Caddyfile` — Reverse proxy (HTTPS termination)
- `/etc/redis/redis.conf` — Redis config
- `/etc/postgresql/16/main/` — PostgreSQL config
- `/etc/systemd/system/vault-*` — All 25 systemd service and timer units
- `/etc/systemd/system/livesync-bridge.service` — LiveSync bridge service
- `/root/.config/rclone/rclone.conf` — Google Drive auth token
- `/srv/backups/borg-key-export.txt` — Borg encryption key
### System State Snapshots (in `/srv/backups/dumps/`)
- `apt-packages.txt` — Full list of installed apt packages
- `pip-requirements.txt` — Python venv package list
- `enabled-services.txt` — All enabled systemd services
- `root-crontab.txt` — Root crontab
- `ufw-rules.txt` — Firewall rules
- `tailscale-status.txt` — Tailscale network status
### Excluded (easily recreated)
- `/opt/vault/.venv` (1.5 GB) — Recreate with `pip install -r requirements.txt`
- `/opt/vault/logs/` — Not needed for restore
- `node_modules/`, `.cache/` — Reinstall from package files
---
## Pre-Backup Validation
Before each backup, the system automatically:
1. Dumps CouchDB metadata and PostgreSQL database
2. Captures system state (packages, services, firewall, crontab)
3. Counts vault files and CouchDB documents
4. Compares against last known good state
5. **Aborts the backup and alerts via Slack if:**
- File count drops below 4,000 (absolute minimum)
- CouchDB doc count drops below 50,000 (absolute minimum)
- File count drops more than 10% from last backup
- Doc count drops more than 10% from last backup
This prevents a corrupted or accidentally pruned vault from overwriting good backups.
---
## Alerts
All alerts are sent to Mason via Slack DM (AutoBot).
- **Daily success:** "Vault backup complete: X archives, repo X local, X GB on Google Drive"
- **Hourly success:** No notification (silent unless failure)
- **Any failure:** "FAILED - backup error"
- **Validation abort:** Specific message about what threshold was violated
---
## Common Operations
### SSH into the server
```bash
ssh root@100.113.209.23 # via Tailscale
```
### List all backup snapshots
```bash
BORG_PASSPHRASE='VaultBorg2026' borg list /srv/backups/borg
```
Example output:
```
ubuntu-4gb-hel1-1-2026-03-06T02:32:46 Fri, 2026-03-06 02:32:46
ubuntu-4gb-hel1-1-2026-03-06T02:46:23 Fri, 2026-03-06 02:46:23
```
### Browse files in a specific snapshot
```bash
BORG_PASSPHRASE='VaultBorg2026' borg list /srv/backups/borg::ARCHIVE_NAME
```
### Search for a specific file
```bash
BORG_PASSPHRASE='VaultBorg2026' borg list /srv/backups/borg::ARCHIVE_NAME | grep "filename"
```
### Restore a single file
```bash
cd /tmp
BORG_PASSPHRASE='VaultBorg2026' borg extract /srv/backups/borg::ARCHIVE_NAME srv/obsidian/Vault/path/to/note.md
# File restored to /tmp/srv/obsidian/Vault/path/to/note.md
```
### Restore an entire folder
```bash
cd /tmp/restore
BORG_PASSPHRASE='VaultBorg2026' borg extract /srv/backups/borg::ARCHIVE_NAME srv/obsidian/Vault/Clients/
```
### Mount a snapshot as a browseable folder
```bash
mkdir /tmp/borg-mount
BORG_PASSPHRASE='VaultBorg2026' borg mount /srv/backups/borg /tmp/borg-mount
# Browse all snapshots:
ls /tmp/borg-mount/
# Browse a specific snapshot:
ls /tmp/borg-mount/ARCHIVE_NAME/srv/obsidian/Vault/
# Copy out what you need, then unmount:
borg umount /tmp/borg-mount
```
### Run a backup manually
```bash
borgmatic --verbosity 1
```
### Check backup integrity manually
```bash
borgmatic check --force
```
### View backup log
```bash
tail -100 /var/log/borgmatic.log
```
### Check Google Drive usage
```bash
rclone about gdrive-backup:
```
---
## Disaster Recovery: Full Server Rebuild
If the Datavault server is lost, follow these steps to rebuild from backup.
### Prerequisites
- A new Hetzner CX22 (or equivalent: Ubuntu 24.04, 2+ vCPU, 4+ GB RAM, 40+ GB disk)
- The Borg passphrase: `VaultBorg2026`
- Access to the Google Drive backup account (or a copy of the Borg repo)
### Step 1: Provision the server
```bash
# After Hetzner provisioning, SSH in as root
# Set hostname
hostnamectl set-hostname autosys-vault
```
### Step 2: Install base packages
```bash
apt update && apt upgrade -y
apt install -y borgbackup rclone python3-pip python3-venv postgresql redis-server \
curl git ufw couchdb caddy deno
pipx install borgmatic
```
### Step 3: Set up rclone and pull the Borg repo from Google Drive
```bash
# Configure rclone with Google Drive access
# (run 'rclone authorize "drive"' on a machine with a browser, then transfer the token)
mkdir -p /root/.config/rclone
# Paste rclone.conf with the gdrive-backup remote
# Pull the Borg repo
mkdir -p /srv/backups/borg
rclone sync gdrive-backup:VaultBackups/borg /srv/backups/borg
```
### Step 4: Restore all files from the latest backup
```bash
# List available snapshots
BORG_PASSPHRASE='VaultBorg2026' borg list /srv/backups/borg
# Extract everything from the latest archive (replace ARCHIVE_NAME)
cd /
BORG_PASSPHRASE='VaultBorg2026' borg extract /srv/backups/borg::ARCHIVE_NAME
```
This restores:
- `/srv/obsidian/Vault/` — all vault files
- `/opt/vault/` — all custom code and configs
- `/opt/livesync-bridge/` — LiveSync bridge
- `/opt/couchdb/etc/` — CouchDB config
- `/etc/caddy/Caddyfile` — reverse proxy
- `/etc/redis/redis.conf` — Redis config
- `/etc/postgresql/16/main/` — PostgreSQL config
- `/etc/systemd/system/vault-*` — all service units
- `/srv/backups/dumps/` — database dumps and system state
### Step 5: Recreate the Python virtual environment
```bash
python3 -m venv /opt/vault/.venv
/opt/vault/.venv/bin/pip install -r /srv/backups/dumps/pip-requirements.txt
```
### Step 6: Restore PostgreSQL
```bash
sudo -u postgres createdb vaultdb
sudo -u postgres createuser vaultdb_user
sudo -u postgres psql -c "ALTER USER vaultdb_user PASSWORD 'BNE9tWic7O6Xdfk5kKdkrJ5VHnINfO1Q';"
gunzip -c /srv/backups/dumps/vaultdb.sql.gz | sudo -u postgres psql vaultdb
```
### Step 7: Configure CouchDB
```bash
# CouchDB should be installed and running
# Restore config
cp /opt/couchdb/etc/local.ini (to wherever CouchDB reads config)
# Set admin password
curl -X PUT http://localhost:5984/_node/_local/_config/admins/admin -d '"CouchVault2026"'
# Create the database
curl -X PUT http://admin:CouchVault2026@localhost:5984/obsidian_vault
# The LiveSync bridge will repopulate CouchDB from the vault files
```
### Step 8: Configure Redis
```bash
# redis.conf was restored, restart redis
systemctl restart redis-server
```
### Step 9: Create the vault user
```bash
useradd -m -s /bin/bash vault
chown -R vault:vault /srv/obsidian/Vault
```
### Step 10: Set up Caddy (HTTPS)
```bash
# Caddyfile was restored to /etc/caddy/
# Update DNS records for vault.autosysapp.com and dashboard.autosysapp.com
# to point to the new server IP
systemctl restart caddy
```
### Step 11: Set up Tailscale
```bash
curl -fsSL https://tailscale.com/install.sh | sh
tailscale up
```
### Step 12: Configure firewall
```bash
# Restore UFW rules from the dump, or manually:
ufw allow in on tailscale0
ufw allow 443/tcp
ufw allow 41641/udp
ufw enable
```
### Step 13: Enable and start all services
```bash
# Systemd units were restored to /etc/systemd/system/
systemctl daemon-reload
# Enable all vault services and timers
systemctl enable --now vault-api vault-watcher vault-worker vault-slack-bot livesync-bridge
systemctl enable --now vault-stray-images.timer vault-gmail-ingest.timer \
vault-indexer.timer vault-slack-ingest.timer vault-gdoc-sync.timer \
vault-gdrive-ingest.timer vault-remarkable-ingest.timer \
vault-reconcile.timer vault-unsub-monitor.timer vault-task-digest.timer
```
### Step 14: Re-establish backups
```bash
# Restore crontab
crontab /srv/backups/dumps/root-crontab.txt
# Verify borgmatic works
borgmatic --verbosity 1
```
### Step 15: Verify
- [ ] `curl https://vault.autosysapp.com` returns CouchDB welcome
- [ ] `curl https://dashboard.autosysapp.com` returns vault web UI
- [ ] Obsidian LiveSync connects and syncs
- [ ] `systemctl status vault-api` is active
- [ ] `systemctl list-timers` shows all vault timers
- [ ] Borgmatic backup runs successfully
- [ ] Slack alerts are working
---
## Key Files on the Server
| File | Purpose |
|---|---|
| `/etc/borgmatic/config.yaml` | Borgmatic configuration |
| `/usr/local/bin/vault-pre-backup.sh` | Pre-backup validation and DB dumps |
| `/usr/local/bin/vault-post-backup.sh` | Google Drive sync and Slack notify |
| `/usr/local/bin/vault-backup-alert.sh` | Slack alert helper |
| `/srv/backups/borg/` | Local Borg repository |
| `/srv/backups/borg-key-export.txt` | Borg encryption key (back up separately) |
| `/srv/backups/last-good-state.json` | Last known good vault state for validation |
| `/srv/backups/dumps/` | Database dumps and system state captures |
| `/var/log/borgmatic.log` | Backup log |
| `/root/.config/rclone/rclone.conf` | Google Drive auth token |
---
## Credentials Reference
| Service | Credentials |
|---|---|
| Borg passphrase | `VaultBorg2026` |
| CouchDB admin | `admin` / `CouchVault2026` |
| PostgreSQL | `vaultdb_user` / `BNE9tWic7O6Xdfk5kKdkrJ5VHnINfO1Q` |
| Redis | `VaultRedis2026` |
| Caddy basic auth (dashboard) | `mason` / `vault2026` |
| Google Drive | Configured via rclone OAuth token |