diff --git a/README.md b/README.md index c8abffa..7fa8aad 100644 --- a/README.md +++ b/README.md @@ -1,178 +1,483 @@ # TISBackup -This is the repository of the TISBackup project, licensed under GPLv3. +A comprehensive server-side backup orchestration system for managing automated backups of databases, files, and virtual machines across remote Linux and Windows systems. -TISBackup is a python script to backup servers. +[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0) +[![Python 3.13+](https://img.shields.io/badge/python-3.13+-blue.svg)](https://www.python.org/downloads/) -It runs at regular intervals to retrieve different data types on remote hosts -such as database dumps, files, virtual machine images and metadata. +## Overview -## Install using Compose +TISBackup is a Python-based backup solution that provides: -Clone that repository and build the pod image using the provided `Dockerfile` +- **Pluggable backup drivers** for different data sources (databases, files, VMs) +- **Web-based management interface** for monitoring and controlling backups +- **CLI tool** for automation and scripting +- **Automated scheduling** via cron +- **Backup retention management** with configurable policies +- **Status monitoring** with Nagios integration +- **Docker deployment** for easy setup and isolation -```bash -docker build . -t tisbackup:latest -``` +### Supported Backup Types -In another folder, create subfolders as following +| Type | Description | Driver | +|------|-------------|--------| +| **Files & Directories** | rsync-based file backups | `rsync+ssh` | +| **Btrfs Snapshots** | Snapshot-based incremental backups | `rsync+btrfs+ssh` | +| **MySQL** | Database dumps via SSH | `mysql+ssh` | +| **PostgreSQL** | Database dumps via SSH | `pgsql+ssh` | +| **SQL Server** | SQL Server backups | `sqlserver+ssh` | +| **Oracle** | Oracle database backups | `oracle+ssh` | +| **Samba4 AD** | Active Directory backups | `samba4` | +| **XenServer VMs** | XVA exports and metadata | `xen-xva`, `xcp-dump-metadata` | +| **VMware** | VMDK backups | `vmdk` | +| **Network Devices** | Switch configuration backups | `switch` | -```bash -mkdir -p /var/tisbackup/{backup/log,config,ssh}/ -``` +## Quick Start -Expected structure -``` -/var/tisbackup/ - └─backup/ <-- backup location - └─config/ - ├── tisbackup-config.ini <-- backups config - └── tisbackup_gui.ini <-- tisbackup config - └─ssh/ - ├── id_ed25519 <-- SSH Private Key (Ed25519 recommended) - └── id_ed25519.pub <-- SSH Public Key - compose.yaml -``` +### Prerequisites -Adapt the compose.yml file to suits your needs, one pod act as the WebUI front end and the other as the crond scheduler +- Docker and Docker Compose +- SSH access to remote servers +- Ed25519, ECDSA, or RSA SSH keys (DSA not supported) -```yaml -services: - tisbackup_gui: - container_name: tisbackup_gui - image: "tisbackup:latest" - build: . - volumes: - - ./config/:/etc/tis/ - - ./backup/:/backup/ - - /etc/timezone:/etc/timezone:ro - - /etc/localtime:/etc/localtime:ro - environment: - # SECURITY: Set a unique secret key for Flask session security - # Generate with: python3 -c "import secrets; print(secrets.token_hex(32))" - - TISBACKUP_SECRET_KEY=your-secret-key-here-change-me - restart: unless-stopped - ports: - - 9980:8080 +### Installation - tisbackup_cron: - container_name: tisbackup_cron - image: "tisbackup:latest" - build: . - volumes: - - ./config/:/etc/tis/ - - ./ssh/:/config_ssh/ - - ./backup/:/backup/ - - /etc/timezone:/etc/timezone:ro - - /etc/localtime:/etc/localtime:ro - restart: always - command: "/bin/bash /opt/tisbackup/cron.sh" +1. **Clone the repository:** + ```bash + git clone https://github.com/tranquilit/TISbackup.git + cd TISbackup + ``` -``` +2. **Build the Docker image:** + ```bash + docker build . -t tisbackup:latest + ``` + +3. **Create directory structure:** + ```bash + mkdir -p /var/tisbackup/{backup/log,config,ssh} + ``` + + Expected structure: + ``` + /var/tisbackup/ + ├── backup/ # Backup storage location + │ └── log/ # SQLite database and logs + ├── config/ # Configuration files + │ ├── tisbackup-config.ini + │ └── tisbackup_gui.ini + ├── ssh/ # SSH keys + │ ├── id_ed25519 # Private key (Ed25519 recommended) + │ └── id_ed25519.pub # Public key + └── compose.yaml # Docker Compose configuration + ``` + +4. **Generate SSH keys:** + ```bash + # Ed25519 (recommended - most secure and modern) + ssh-keygen -t ed25519 -f /var/tisbackup/ssh/id_ed25519 -C "tisbackup@yourserver" + + # Or ECDSA (also secure) + ssh-keygen -t ecdsa -b 521 -f /var/tisbackup/ssh/id_ecdsa -C "tisbackup@yourserver" + + # Or RSA (legacy support, minimum 4096 bits) + ssh-keygen -t rsa -b 4096 -f /var/tisbackup/ssh/id_rsa -C "tisbackup@yourserver" + ``` + + ⚠️ **Note:** DSA keys are no longer supported due to security vulnerabilities. + +5. **Deploy public key to remote servers:** + ```bash + ssh-copy-id -i /var/tisbackup/ssh/id_ed25519.pub root@remote-server + ``` + +6. **Generate Flask secret key:** + ```bash + python3 -c "import secrets; print(secrets.token_hex(32))" + ``` + Save this key for the next step. + +7. **Create Docker Compose configuration:** + ```yaml + # /var/tisbackup/compose.yaml + services: + tisbackup_gui: + container_name: tisbackup_gui + image: "tisbackup:latest" + volumes: + - ./config/:/etc/tis/ + - ./backup/:/backup/ + - /etc/timezone:/etc/timezone:ro + - /etc/localtime:/etc/localtime:ro + environment: + # SECURITY: Use the secret key you generated above + - TISBACKUP_SECRET_KEY=your-secret-key-here + restart: unless-stopped + ports: + - 9980:8080 + + tisbackup_cron: + container_name: tisbackup_cron + image: "tisbackup:latest" + volumes: + - ./config/:/etc/tis/ + - ./ssh/:/config_ssh/ + - ./backup/:/backup/ + - /etc/timezone:/etc/timezone:ro + - /etc/localtime:/etc/localtime:ro + restart: always + command: "/bin/bash /opt/tisbackup/cron.sh" + ``` + +8. **Configure backups:** + + Create `/var/tisbackup/config/tisbackup-config.ini`: + ```ini + [global] + backup_base_dir = /backup/ + # Backup retention in days + backup_retention_time = 90 + # Maximum backup age for Nagios checks (hours) + maximum_backup_age = 30 + + # Example: File backup via rsync + [webserver-files] + type = rsync+ssh + server_name = webserver.example.com + remote_dir = /var/www/ + compression = True + exclude_list = "/var/www/cache/**","/var/www/temp/**" + private_key = /config_ssh/id_ed25519 + ssh_port = 22 + + # Example: MySQL database backup + [database-mysql] + type = mysql+ssh + server_name = db.example.com + db_name = production_db + db_user = backup_user + db_passwd = backup_password + private_key = /config_ssh/id_ed25519 + ssh_port = 22 + ``` + + Create `/var/tisbackup/config/tisbackup_gui.ini`: + ```ini + [general] + config_tisbackup = /etc/tis/tisbackup-config.ini + sections = + ADMIN_EMAIL = admin@example.com + base_config_dir = /etc/tis/ + backup_base_dir = /backup/ + ``` + +9. **Start services:** + ```bash + cd /var/tisbackup + docker compose up -d + ``` + +10. **Access web interface:** + ``` + http://localhost:9980 + ``` ## Configuration -### SSH Keys - - * **Generate SSH keys** (Ed25519 recommended): - ```bash - # Ed25519 (most secure, recommended) - ssh-keygen -t ed25519 -f ./ssh/id_ed25519 -C "tisbackup@yourserver" - - # Or ECDSA (also secure) - ssh-keygen -t ecdsa -b 521 -f ./ssh/id_ecdsa -C "tisbackup@yourserver" - - # Or RSA (legacy, minimum 2048 bits) - ssh-keygen -t rsa -b 4096 -f ./ssh/id_rsa -C "tisbackup@yourserver" - ``` - **⚠️ Note:** DSA keys are no longer supported due to security vulnerabilities - - * Copy public key to remote servers: - ```bash - ssh-copy-id -i ./ssh/id_ed25519.pub root@remote-server - ``` - -### Configuration Files - - * Setup config files in the `./config` directory - - * **SECURITY**: Generate and set a secure Flask secret key: - ```bash - # Generate a secure random secret key - python3 -c "import secrets; print(secrets.token_hex(32))" - ``` - Then add it to your `compose.yml` as the `TISBACKUP_SECRET_KEY` environment variable - -**tisbackup-config.ini** +### Backup Types Configuration +#### File Backups (rsync+ssh) ```ini -[global] -backup_base_dir = /backup/ - -# backup retention in days -backup_retention_time=90 - -# for nagios check in hours -maximum_backup_age=30 - -[srvads-poudlard-samba] -type=rsync+ssh -server_name=srvads.poudlard.lan -remote_dir=/var/lib/samba/ -compression=True -;exclude_list="/proc/**","/sys/**","/dev/**" -# Use Ed25519 key (recommended), or ECDSA/RSA (DSA not supported) -private_key=/config_ssh/id_ed25519 +[backup-name] +type = rsync+ssh +server_name = hostname.example.com +remote_dir = /path/to/backup/ +compression = True +exclude_list = "/path/exclude1/**","/path/exclude2/**" +private_key = /config_ssh/id_ed25519 ssh_port = 22 ``` -**tisbackup_gui.ini** +#### Btrfs Snapshots (rsync+btrfs+ssh) ```ini -[general] -config_tisbackup= /etc/tis/tisbackup-config.ini -sections= -ADMIN_EMAIL=josebove@internet.fr -base_config_dir= /etc/tis/ -backup_base_dir=/backup/ +[backup-name] +type = rsync+btrfs+ssh +server_name = hostname.example.com +remote_dir = /mnt/btrfs/data/ +compression = True +private_key = /config_ssh/id_ed25519 +ssh_port = 22 ``` -Run! +#### MySQL Database (mysql+ssh) +```ini +[backup-name] +type = mysql+ssh +server_name = hostname.example.com +db_name = database_name +db_user = backup_user +db_passwd = backup_password +private_key = /config_ssh/id_ed25519 +ssh_port = 22 +``` + +#### PostgreSQL Database (pgsql+ssh) +```ini +[backup-name] +type = pgsql+ssh +server_name = hostname.example.com +db_name = database_name +private_key = /config_ssh/id_ed25519 +ssh_port = 22 +``` + +#### XenServer VM (xen-xva) +```ini +[backup-name] +type = xen-xva +server_name = vm-name +xcphost = xenserver.example.com +password_file = /etc/tis/xen-password +private_key = /config_ssh/id_ed25519 +``` + +### Pre/Post Execution Hooks + +You can execute commands before and after backups: + +```ini +[backup-name] +type = rsync+ssh +server_name = hostname.example.com +remote_dir = /data/ +private_key = /config_ssh/id_ed25519 +preexec = systemctl stop application +postexec = systemctl start application +remote_user = root +ssh_port = 22 +``` + +## CLI Usage + +### Running Backups + ```bash -docker compose up -d +# Run all backups +docker exec tisbackup_cron python3 /opt/tisbackup/tisbackup.py backup + +# Run specific backup +docker exec tisbackup_cron python3 /opt/tisbackup/tisbackup.py -s backup-name backup + +# Dry run +docker exec tisbackup_cron python3 /opt/tisbackup/tisbackup.py -d backup ``` -## NGINX reverse-proxy +### Cleanup Old Backups -Sample config file +```bash +# Remove backups older than retention period +docker exec tisbackup_cron python3 /opt/tisbackup/tisbackup.py cleanup +``` + +### Nagios Monitoring + +```bash +# Check backup status +docker exec tisbackup_cron python3 /opt/tisbackup/tisbackup.py checknagios + +# Check specific backup +docker exec tisbackup_cron python3 /opt/tisbackup/tisbackup.py -s backup-name checknagios +``` + +### List Available Drivers + +```bash +docker exec tisbackup_cron python3 /opt/tisbackup/tisbackup.py listdrivers +``` + +### Backup Statistics + +```bash +# Dump statistics for last 20 backups +docker exec tisbackup_cron python3 /opt/tisbackup/tisbackup.py dumpstat + +# Specify number of backups +docker exec tisbackup_cron python3 /opt/tisbackup/tisbackup.py -n 50 dumpstat +``` + +## Development + +### Prerequisites + +- Python 3.13+ +- uv (Python package manager) + +### Setup Development Environment + +```bash +# Install dependencies +uv sync --locked + +# Run linter +uv run ruff check . + +# Auto-fix linting issues +uv run ruff check --fix . +``` + +### Running Locally + +```bash +# Run web GUI (requires config at /etc/tis/tisbackup_gui.ini) +python3 tisbackup_gui.py + +# Run CLI +python3 tisbackup.py -c /etc/tis/tisbackup-config.ini backup +``` + +## Security + +TISBackup implements several security best practices: + +### SSH Key Security + +- **Ed25519 keys are recommended** (most secure, modern algorithm) +- ECDSA and RSA keys are supported +- **DSA keys are explicitly not supported** (deprecated, insecure) +- Key algorithm priority: Ed25519 → ECDSA → RSA + +### Flask Session Security + +- Secret key loaded from `TISBACKUP_SECRET_KEY` environment variable +- Falls back to cryptographically secure random key if not set +- No hardcoded secrets in source code + +### Command Execution Safety + +- All system commands use `subprocess.run()` with list arguments +- Input validation for device paths and partition names +- Timeout protection on all subprocess calls +- No use of `shell=True` in new code + +### Best Practices + +1. **Use Ed25519 keys** for all SSH connections +2. **Set unique Flask secret key** via environment variable +3. **Use reverse proxy** (nginx) with TLS for web interface +4. **Restrict network access** to backup server +5. **Regular security updates** of base Docker image +6. **Monitor backup logs** for suspicious activity + +## Reverse Proxy Setup + +Example nginx configuration for HTTPS access: ```nginx server { listen 443 ssl http2; - # Remove '#' in the next line to enable IPv6 - # listen [::]:443 ssl http2; - server_name tisbackup.poudlard.lan; + server_name tisbackup.example.com; - ssl_certificate /etc/letsencrypt/live/tisbackup.poudlard.lan/fullchain.pem; # managed by Certbot - ssl_certificate_key /etc/letsencrypt/live/tisbackup.poudlard.lan/privkey.pem; # managed by Certbot + ssl_certificate /etc/letsencrypt/live/tisbackup.example.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/tisbackup.example.com/privkey.pem; + # Security headers + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; + add_header X-Frame-Options "SAMEORIGIN" always; + add_header X-Content-Type-Options "nosniff" always; location / { - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header Host $host; - proxy_pass http://localhost:9980/; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header Host $host; + proxy_pass http://localhost:9980/; proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection "upgrade"; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; } } ``` +## Architecture +TISBackup uses a modular driver-based architecture: -## About +- **Core CLI** ([tisbackup.py](tisbackup.py)): Backup orchestration and scheduling +- **Web GUI** ([tisbackup_gui.py](tisbackup_gui.py)): Flask-based management interface +- **Backup Drivers** ([libtisbackup/](libtisbackup/)): Pluggable modules for different backup types +- **Task Queue** ([tasks.py](tasks.py), [config.py](config.py)): Async job processing with Huey +- **State Database**: SQLite for tracking backup history and statistics -[Tranquil IT](contact_at_tranquil_it) is the original author of TISBackup. +Each backup type is implemented as a driver class inheriting from `backup_generic`, allowing easy extension for new backup sources. -The documentation is provided under the license CC-BY-SA and can be found -on [readthedoc](https://tisbackup.readthedocs.io/en/latest/index.html). +## Troubleshooting + +### Backups Not Running + +1. Check cron logs: + ```bash + docker logs tisbackup_cron + ``` + +2. Verify SSH connectivity: + ```bash + docker exec tisbackup_cron ssh -i /config_ssh/id_ed25519 root@remote-server + ``` + +3. Check backup configuration: + ```bash + docker exec tisbackup_cron python3 /opt/tisbackup/tisbackup.py -c /etc/tis/tisbackup-config.ini -d backup + ``` + +### Web Interface Not Accessible + +1. Check GUI container logs: + ```bash + docker logs tisbackup_gui + ``` + +2. Verify port mapping: + ```bash + docker ps | grep tisbackup_gui + ``` + +3. Check configuration: + ```bash + docker exec tisbackup_gui cat /etc/tis/tisbackup_gui.ini + ``` + +### Permission Errors + +Ensure proper file permissions: +```bash +chmod 600 /var/tisbackup/ssh/id_ed25519 +chmod 644 /var/tisbackup/ssh/id_ed25519.pub +chown -R root:root /var/tisbackup/ +``` + +## Contributing + +Contributions are welcome! Please: + +1. Fork the repository +2. Create a feature branch +3. Follow the existing code style (use `ruff` for linting) +4. Add tests if applicable +5. Submit a pull request + +## License + +TISBackup is licensed under the GNU General Public License v3.0 (GPLv3). + +See [LICENSE](LICENSE) for the full license text. + +## Support & Documentation + +- **Documentation**: [https://tisbackup.readthedocs.io](https://tisbackup.readthedocs.io/en/latest/index.html) +- **Issues**: [GitHub Issues](https://github.com/tranquilit/TISbackup/issues) +- **Original Author**: [Tranquil IT](https://www.tranquil.it) + +## Credits + +Developed by Tranquil IT for system administrators managing backup infrastructure. + +Security improvements and modernization contributed by the community.