Skip to main content

Backup Infrastructure

Quick Overview

Purpose: Centralized backup strategy for all critical systems
Primary Storage: LortiNAS (Central Repository)
Secondary Storage: SYNLORTINAS (Archive & Media)
Backup Software: Duplicati, Proxmox VE, Native Tools
Status: 🟢 Operational

📋 Description​

This comprehensive backup infrastructure ensures data protection across physical servers, virtual machines, containers, and NAS devices. The system implements a 3-2-1 backup strategy with automated schedules, versioning, and off-site replication capabilities.

🏗️ Architecture Overview​

🖥️ Infrastructure Components​

Primary Storage: LortiNAS​

Central Backup Repository

LortiNAS serves as the primary destination for all backup operations

AttributeValuePurpose
TypeNAS ServerBackup Storage
Capacity20TBUsable space
RAID LevelRAID 5Data protection
Network1 GbpsFast transfers
LocationOn-premiseLocal backups

Backup Types Stored:

  • ✅ Daily Proxmox VM backups
  • ✅ Duplicati file and database backups
  • ✅ Container volume snapshots
  • ✅ Configuration backups
  • ✅ Yearly SYNLORTINAS archives

Secondary Storage: SYNLORTINAS​

AttributeValuePurpose
TypeSynology NASMedia & Archive
Capacity40TBLarge storage
RAID LevelRAID 6High redundancy
Primary UseActive StorageLive data
Backup RoleSource & TargetDual purpose

Content Types:

  • 📸 Historical photo archives
  • 🎬 Media files and libraries
  • 📁 Shared documents
  • 🗄️ Long-term archives

📅 Backup Schedules​

Automated Backup Timeline​

Retention Policies​

📊 Detailed Retention Matrix
Data TypeDailyWeeklyMonthlyYearlyTotal Storage
VM Backups74122~2TB
Databases30-121~500GB
File Backups148122~1TB
Configurations30--5~50GB
Media Archives---∞~15TB

🔄 Backup Procedures​

Pre-Backup Scripts​

/scripts/backup-databases.sh
#!/bin/bash
# Database backup script - runs before Duplicati

BACKUP_DIR="/backup/databases"
DATE=$(date +%Y%m%d_%H%M%S)

# Backup MySQL/MariaDB
echo "Backing up MySQL databases..."
docker exec mysql mysqldump --all-databases \
  --single-transaction \
  --quick \
  --lock-tables=false \
  > "$BACKUP_DIR/mysql_$DATE.sql"

# Backup PostgreSQL
echo "Backing up PostgreSQL databases..."
docker exec postgres pg_dumpall \
  -U postgres \
  > "$BACKUP_DIR/postgres_$DATE.sql"

# Backup MongoDB
echo "Backing up MongoDB..."
docker exec mongodb mongodump \
  --out "$BACKUP_DIR/mongodb_$DATE"

# Compress old backups
find "$BACKUP_DIR" -name "*.sql" -mtime +7 \
  -exec gzip {} \;

echo "Database backup completed!"

🔐 Security & Encryption​

Backup Encryption​

Security Requirements

All backups containing sensitive data MUST be encrypted

Backup TypeEncryptionMethodKey Storage
Duplicati✅ YesAES-256Vaultwarden
Proxmox✅ YesAES-256-GCMLocal keystore
Database Dumps✅ YesGPGSecure vault
Media Files❌ NoN/AN/A

Access Control​

Backup Administrators:
  - Full access to all backup systems
  - Can modify retention policies
  - Can initiate restores

Operators:
  - Read-only access to backup status
  - Can initiate test restores
  - Cannot modify configurations

Service Accounts:
  - Automated backup execution
  - Limited to specific paths
  - No interactive access

📊 Monitoring & Alerts​

Health Checks​

🔍 Monitoring Configuration
# Grafana Dashboard Metrics
Metrics:
  - backup_job_duration
  - backup_size_bytes
  - backup_success_rate
  - storage_usage_percent
  - last_successful_backup

# Alert Rules
Alerts:
  - name: Backup Failed
    condition: backup_success == 0
    severity: critical
    notification: email, slack
    
  - name: Storage Full
    condition: storage_usage > 85%
    severity: warning
    notification: email
    
  - name: Backup Overdue
    condition: time_since_backup > 48h
    severity: warning
    notification: slack

Verification Procedures​

Regular Testing

Backup verification should be performed monthly

  1. Automated Verification

    • Checksum validation
    • Test restore to sandbox
    • File integrity checks

  2. Manual Testing

    • Quarterly DR drill
    • Random file restoration
    • Full VM recovery test

🚨 Disaster Recovery​

Recovery Time Objectives​

Service LevelRTORPORecovery Method
Critical1 hour1 hourHot standby
Important4 hours24 hoursVM restore
Standard24 hours48 hoursManual restore
Archive72 hours1 weekTape/cloud

Recovery Procedures​

📋 VM Recovery Steps

  1. Identify Failed System

    # Check VM status
    qm status <VMID>

  2. Locate Latest Backup

    # List available backups
    pvesm list backup-lortinas

  3. Initiate Restore

    # Restore VM from backup
    qmrestore /path/to/backup.vma.zst <NEW_VMID>

  4. Verify Restoration

    • Check network connectivity
    • Verify services are running
    • Test application functionality

  5. Update DNS/Load Balancer

    • Point traffic to restored instance
    • Monitor for issues

🔧 Troubleshooting​

Common Issues​

Backup Failures

Most backup failures are due to storage or network issues

❌ Duplicati Backup Fails

Symptoms: Backup job shows failed status

Solutions:

  1. Check storage space: df -h /backup
  2. Verify network mount: mount | grep lortinas
  3. Check Duplicati logs: /config/logs/
  4. Test connectivity: ping 192.168.1.80
  5. Restart Duplicati container: docker restart duplicati
🔴 Proxmox Backup Timeout

Symptoms: VM backup exceeds time limit

Solutions:

  1. Increase timeout in datacenter.cfg
  2. Check I/O wait: iostat -x 5
  3. Verify network bandwidth
  4. Consider backup window adjustment
  5. Enable backup throttling

📚 Related Documentation​

Maintenance Information

Last Review: December 2024
Next Audit: March 2025
Contact: Infrastructure Team
On-Call: Check PagerDuty rotation