DIYLABHub.com

Comprehensive Guide to Self-Hosting Kokoro Web on Ubuntu 20.04+ with Docker

Sat, 05 Apr 2025 18:35:01 GMT

Kokoro Web is a powerful AI text-to-speech solution that lets you create natural-sounding voices directly in your browser without installation requirements. This free and open-source tool can be self-hosted to provide an OpenAI-compatible API for your applications, offering high-quality voice generation capabilities comparable to leading commercial services. This guide will walk you through setting up Kokoro Web on Ubuntu systems using Docker, with special configurations for WSL access and local website integration with Pi-hole.

Introduction to Kokoro Web#

Kokoro Web is powered by the hexgrad/Kokoro-82M model, a lightweight yet capable 82 million parameter text-to-speech model available on Hugging Face. Despite its relatively small size, it delivers impressive quality comparable to larger models while being significantly faster and more resource-efficient. With Apache-licensed weights, you can deploy Kokoro anywhere from production environments to personal projects^2.

Key features include:

Zero installation requirements for end-users
Self-hosting capabilities with OpenAI API compatibility
Support for various language accents
Voice customization options
WebGPU acceleration for supported browsers
Completely free for both personal and commercial use^2

Prerequisites#

Hardware Requirements#

CPU-only setup: Any modern CPU with at least 2 cores
GPU-accelerated setup (optional): NVIDIA GPU with CUDA support
Minimum 4GB RAM (8GB recommended)
At least 10GB free disk space

Software Requirements#

Ubuntu 20.04 LTS or newer
Root or sudo privileges on your system
Basic knowledge of terminal commands
Internet connection for downloading required packages and Docker images

Setting Up Docker on Ubuntu#

For Users Who Don't Have Docker Installed#

If you don't already have Docker installed on your Ubuntu system, follow these steps to install Docker Engine and Docker Compose.

Step 1: Set up Docker's repository#

Run in terminalbash

# Update the package index and install dependencies
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg

Run in terminalbash

# Add Docker's official GPG key
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

Run in terminalbash

# Set up the repository
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list &gt; /dev/null

Step 2: Install Docker Engine and Docker Compose#

Run in terminalbash

# Update the package index again
sudo apt-get update

Run in terminalbash

# Install Docker Engine, containerd, and Docker Compose
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Step 3: Verify the installation#

Run in terminal

# Run the hello-world container to verify Docker works
sudo docker run hello-world

Step 4: Add your user to the docker group (avoid using sudo with Docker)#

Run in terminalbash

# Add your user to the docker group
sudo usermod -aG docker $USER

Run in terminalbash

# Apply the new group membership (alternatively, you can log out and back in)
newgrp docker

Run in terminalbash

# Verify you can run Docker commands without sudo
docker run hello-world

For Users Who Already Have Docker Installed#

If you already have Docker installed, make sure it's up to date:

Run in terminalbash

# Update your package list
sudo apt-get update

Run in terminalbash

# Upgrade Docker packages
sudo apt-get upgrade docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Run in terminalbash

# Verify Docker is working correctly
docker --version
docker compose version
docker run hello-world

Installing and Configuring Kokoro Web with Docker#

Step 1: Create a directory structure for Kokoro Web#

Run in terminalbash

# Create directories for Kokoro Web data and configuration
mkdir -p ~/docker_data/kokoro-web/cache

Step 2: Create a Docker Compose file#

Run in terminalbash

# Navigate to the Kokoro Web directory
cd ~/docker_data/kokoro-web

Run in terminalbash

# Create a docker-compose.yml file using your preferred text editor
nano docker-compose.yml

Paste the following configuration into the file:

File contentbash

version: '3.8'
services:
  kokoro-web:
    image: ghcr.io/eduardolat/kokoro-web:latest
    container_name: kokoro-web
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      - KW_SECRET_API_KEY=your-secure-api-key  # Change this to a strong, unique key
      - TZ=UTC  # Set to your timezone if needed
    volumes:
      - ./cache:/kokoro/cache  # Cache downloaded models and voices
    networks:
      - kokoro-network

networks:
  kokoro-network:
    driver: bridge

Save and exit the editor (in nano, press Ctrl+X, then Y, then Enter).

Step 3: Start the Kokoro Web service#

Run in terminalbash

# Start the Kokoro Web container in detached mode
docker compose up -d

Run in terminalbash

# Check if the container is running
docker ps

You should see a container named "kokoro-web" in the list, indicating that the service is running.

Step 4: Access the web interface#

After starting the container, you can access the Kokoro Web interface by opening a web browser and navigating to:

Open in your browserbash

http://localhost:3000

The API interface can be accessed at:

Open in your browserbash

http://localhost:3000/api/v1/index.html

Securing Your Kokoro Web Installation#

Step 1: Set a strong API key#

Edit your docker-compose.yml file to change the KW_SECRET_API_KEY to a strong, unique value:

Run in terminalbash

# Generate a secure random string for the API key
API_KEY=$(openssl rand -base64 32)
echo "Your generated API key: $API_KEY"

Run in terminalbash

# Edit the docker-compose.yml file
nano docker-compose.yml

Update the KW_SECRET_API_KEY value, save the file, and restart the container:

Run in terminalbash

# Restart the container to apply changes
docker compose down
docker compose up -d

Step 2: Using non-root user in Docker (optional but recommended)#

To improve security, you can ensure the Docker container runs as a non-root user by adding user and group settings to your docker-compose.yml:

File contentbash

services:
  kokoro-web:
    image: ghcr.io/eduardolat/kokoro-web:latest
    user: 1000:1000  # Use the user ID and group ID that matches your host user
    # ... rest of your configuration

Step 3: Set appropriate file permissions#

Run in terminalbash

# Ensure the cache directory has the right permissions
sudo chown -R 1000:1000 ~/docker_data/kokoro-web/cache

Step 4: Implement network restrictions#

If you want to restrict access to your Kokoro Web instance to only your local network, you can modify the ports configuration in docker-compose.yml:

File contentbash

ports:
  - "127.0.0.1:3000:3000"  # Only accessible from localhost
  # OR
  - "192.168.1.100:3000:3000"  # Only accessible from your specific IP

Integrating with Windows Subsystem for Linux (WSL)#

If you're running Ubuntu through WSL, you can access Kokoro Web both from your WSL environment and from Windows.

Setting up Docker with WSL2#

Step 1: Ensure WSL2 is properly configured#

If you haven't already set up WSL2 with Ubuntu, follow these steps on your Windows machine:

Enable WSL2 in Windows features
Install Ubuntu from the Microsoft Store
Set WSL2 as the default version

Run in terminalbash

# Run these in PowerShell with administrator privileges
wsl --install
wsl --set-default-version 2

Step 2: Install Docker Desktop for Windows#

For WSL2 integration, it's recommended to use Docker Desktop for Windows:

Download and install Docker Desktop for Windows
During installation, ensure "Use WSL 2 instead of Hyper-V" is selected
After installation, open Docker Desktop settings and enable integration with your Ubuntu distribution under "Resources > WSL Integration"

Step 3: Access Kokoro Web from Windows#

After setting up Docker Desktop with WSL2 integration and running the Kokoro Web container in your Ubuntu WSL distribution, you can access the web interface from Windows through:

Open in your browserbash

http://localhost:3000

Or using the IP address of your WSL2 distribution:

Open in your browserbash

http://wsl-ip-address:3000

You can find your WSL2 IP address using this command in Ubuntu:

Run in terminalbash

ip addr show eth0 | grep -oP '(?&lt;=inet\s)\d+(\.\d+){3}'

Integrating with Pi-hole for Local Domain Access#

Step 1: Set up a local domain with Pi-hole#

To access your Kokoro Web instance using a friendly domain name (e.g., kokoro.local), you can configure Pi-hole to resolve this domain to your server's IP address.

Method 1: Using Pi-hole's Local DNS Records#

Access your Pi-hole admin interface (typically http://pi.hole/admin)
Navigate to "Local DNS > DNS Records"
Add a new entry with:
- Domain: kokoro.local
- IP Address: [Your server's IP address]
Save the changes

Method 2: Using /etc/hosts file on the Pi-hole server#

Run in terminalbash

# SSH into your Pi-hole server
ssh user@pi-hole-server

Run in terminalbash

# Edit the hosts file
sudo nano /etc/hosts

Add this line to the file:

File contentbash

192.168.1.xxx  kokoro.local

Replace "192.168.1.xxx" with your Kokoro Web server's IP address.

Save and exit the editor, then restart the DNS service:

Run in terminalbash

# Restart Pi-hole DNS service
sudo pihole restartdns

Step 2: Configure Traefik for local domain routing (optional)#

For more advanced setups, you can use Traefik as a reverse proxy to handle local domain routing. This is particularly useful if you're hosting multiple services on the same server.

Step 1: Create a Traefik configuration#

First, create a new directory for Traefik:

Run in terminalbash

mkdir -p ~/docker_data/traefik/{config,data}
cd ~/docker_data/traefik
touch docker-compose.yml

Edit the docker-compose.yml file:

Run in terminalbash

nano docker-compose.yml

Add the following configuration:

File contentbash

version: '3.8'
services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./config/traefik.yml:/etc/traefik/traefik.yml:ro
      - ./config/config.yml:/etc/traefik/config.yml:ro
      - ./data/certificates:/certificates
    networks:
      - proxy

networks:
  proxy:
    external: true

Create a Traefik configuration file:

Run in terminalbash

mkdir -p ~/docker_data/traefik/config
nano ~/docker_data/traefik/config/traefik.yml

Add the following content:

File contentbash

api:
  dashboard: true

entryPoints:
  http:
    address: ":80"
  https:
    address: ":443"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /etc/traefik/config.yml

log:
  level: INFO

Create a configuration file for your services:

Run in terminalbash

nano ~/docker_data/traefik/config/config.yml

Add the following content:

File contentbash

http:
  routers:
    kokoro:
      rule: "Host(`kokoro.local`)"
      service: kokoro
      entryPoints:
        - http

  services:
    kokoro:
      loadBalancer:
        servers:
          - url: "http://kokoro-web:3000"

Step 2: Create a Docker network#

Run in terminalbash

docker network create proxy

Step 3: Update the Kokoro Web docker-compose.yml file#

Edit your Kokoro Web docker-compose.yml:

Run in terminalbash

nano ~/docker_data/kokoro-web/docker-compose.yml

Update it with network and label configuration:

File contentbash

version: '3.8'
services:
  kokoro-web:
    image: ghcr.io/eduardolat/kokoro-web:latest
    container_name: kokoro-web
    restart: unless-stopped
    environment:
      - KW_SECRET_API_KEY=your-secure-api-key
      - TZ=UTC
    volumes:
      - ./cache:/kokoro/cache
    networks:
      - kokoro-network
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.kokoro.rule=Host(`kokoro.local`)"
      - "traefik.http.services.kokoro.loadbalancer.server.port=3000"

networks:
  kokoro-network:
    driver: bridge
  proxy:
    external: true

Step 4: Start Traefik and restart Kokoro Web#

Run in terminalbash

# Start Traefik
cd ~/docker_data/traefik
docker compose up -d

# Restart Kokoro Web with updated configuration
cd ~/docker_data/kokoro-web
docker compose down
docker compose up -d

Verifying Your Installation#

Step 1: Check that the containers are running#

Run in terminalbash

docker ps

You should see your Kokoro Web container (and Traefik if you set it up) in the running containers list.

Step 2: Test the web interface#

Open a web browser and navigate to one of the following URLs:

http://localhost:3000 (direct access)
http://kokoro.local (if you set up Pi-hole DNS or hosts file configuration)

You should see the Kokoro Web interface where you can test voice generation.

Step 3: Test the API#

You can test the API using curl:

Run in terminalbash

curl -X POST "http://localhost:3000/api/v1/audio/speech" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{
    "model": "model_q8f16",
    "voice": "af_bella",
    "input": "Hello, I am testing Kokoro Web API!",
    "response_format": "mp3"
  }' \
  --output test.mp3

Run in terminalbash

# Play the generated audio (requires an audio player like mpg123)
mpg123 test.mp3

Maintenance and Updates#

Updating Kokoro Web#

To update to the latest version of Kokoro Web:

Run in terminalbash

# Navigate to your Kokoro Web directory
cd ~/docker_data/kokoro-web

# Pull the latest image
docker compose pull

# Restart the container with the new image
docker compose down
docker compose up -d

Backing up your configuration#

It's good practice to back up your configuration files:

Run in terminalbash

# Create a backup directory
mkdir -p ~/backups/kokoro-web

# Copy configuration files
cp ~/docker_data/kokoro-web/docker-compose.yml ~/backups/kokoro-web/

Monitoring and logs#

Monitor the logs for any issues:

Run in terminalbash

# View real-time logs
docker logs -f kokoro-web

# View the last 100 lines of logs
docker logs --tail 100 kokoro-web

Troubleshooting Common Issues#

Container fails to start#

If the container fails to start, check the logs:

Run in terminalbash

docker logs kokoro-web

Common issues include:

Permission problems with the cache directory
Port 3000 already in use by another service

Unable to access web interface#

If you can't access the web interface:

Verify the container is running: docker ps
Check if the port is correctly published: docker port kokoro-web
Try accessing with the IP address instead of localhost
Check your firewall settings: sudo ufw status

API authentication issues#

If you're having trouble with API authentication:

Verify you're using the correct API key
Check that the API key is correctly set in your docker-compose.yml
Ensure the Authorization header format is correct: Authorization: Bearer your-api-key

Security Best Practices#

Container security#

Keep Docker and the container image updated
Use specific image version tags instead of "latest" for production
Don't run containers with elevated privileges
Use non-root users inside containers when possible
Limit container capabilities to only what's needed^9

Network security#

Use a reverse proxy like Traefik or Nginx for TLS termination
Consider using internal Docker networks for service isolation
Implement rate limiting for the API endpoints
Use firewalls to restrict access to your server

API security#

Use a strong, randomly generated API key
Rotate API keys periodically
Implement IP-based restrictions if possible
Monitor API usage for unusual patterns

Conclusion#

You've now successfully set up Kokoro Web, a powerful open-source AI text-to-speech solution, on your Ubuntu system using Docker. This setup allows you to generate high-quality voice content locally without relying on external services, while maintaining compatibility with the OpenAI API format for easy integration with existing applications.

By following the additional configuration steps, you've also learned how to make Kokoro Web accessible through WSL for Windows users and how to integrate it with Pi-hole for convenient local domain access. The security recommendations will help protect your instance from unauthorized access.

As an open-source project, Kokoro Web continues to evolve, so be sure to check the official repository periodically for updates and new features. With this self-hosted solution, you have complete control over your text-to-speech processing, ensuring privacy and flexibility for your projects.

⁂

Installing Metabase on Ubuntu via WSL2 with Docker: A Complete Guide for Beginners (Enterprise setup)

Thu, 03 Apr 2025 04:08:00 GMT

Introduction#

Metabase is a powerful, open-source business intelligence tool that lets you visualize and explore your data without needing to write SQL code. This guide shows you how to install Metabase on Windows using WSL2 (Windows Subsystem for Linux) with Docker, providing:

Easy access through Windows while leveraging Linux's container optimization
Enterprise-grade security and stability features
Simple setup for powerful data visualization
Persistent data storage that survives restarts

TIP: This method is perfect for data analysts who work on Windows machines but want the power of Linux containers.

Prerequisites#

Before starting, make sure you have:

Windows 10/11 (Build 19041 or newer)
4GB RAM minimum (8GB recommended for smooth performance)
Virtualization enabled in your BIOS/UEFI settings
- TIP: Most computers have this enabled by default. If not, you'll need to restart and press F2, F10, or Del during startup to enter BIOS settings.
Administrator access on your Windows machine
At least 10GB free disk space
Internet connection to download packages

1. Setting Up WSL2#

Install WSL2 with Ubuntu#

Run in terminalbash

wsl --set-default-version 2
wsl --install -d Ubuntu

This installs Ubuntu on WSL2. Your computer will restart automatically after installation.

TIP: If you see any error about virtualization, you need to enable it in your BIOS settings.

Verify WSL2 Installation#

Run in terminalbash

wsl -l -v

You should see "Ubuntu" with "2" in the VERSION column. If it shows "1", convert it with:

Run in terminalbash

wsl --set-version Ubuntu 2

HINT: If Ubuntu isn't running, type wsl in PowerShell to start it.

2. Installing Docker Engine#

Update Package Information#

Run in terminalbash

sudo apt update && sudo apt install ca-certificates curl gnupg

This prepares your system with necessary certificates and tools.

TIP: When asked for your password, you won't see anything as you type - this is normal!

Set Up Docker Repository#

Run in terminalbash

sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

These commands download and set up Docker's security keys.

HINT: You might see a message about "Creating directory" - this is expected and means it's working correctly.

Add Docker Repository to APT Sources#

Run in terminalbash

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

This adds Docker's official repository to your system.

TIP: Stick to official repositories for security and stability.

Install Docker Components#

Run in terminalbash

sudo apt update && sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

This installs Docker Engine and all necessary components.

HINT: This might take a few minutes depending on your internet speed.

Set Docker Permissions#

Run in terminalbash

sudo usermod -aG docker $USER && newgrp docker

This lets you run Docker commands without typing sudo each time.

TIP: If you close your terminal, you might need to restart it for these permissions to take effect.

Test Docker Installation#

Run in terminalbash

docker run --rm hello-world

You should see a "Hello from Docker!" message which confirms Docker is working correctly.

TROUBLESHOOTING: If you see "permission denied", try logging out and back in to WSL, or restart your computer.

3. Setting Up Metabase#

Create Directory Structure#

Run in terminalbash

mkdir -p ~/docker_data/metabase/{data,config,postgres}
chmod -R 755 ~/docker_data

This creates folders to store your Metabase data persistently.

TIP: These directories will remain even if Metabase is uninstalled, protecting your data.

Create Docker Compose File#

Run in terminalbash

cat > ~/docker_data/metabase/docker-compose.yml << 'EOF'
version: '3'

services:
  metabase:
    image: metabase/metabase:latest
    container_name: metabase
    depends_on:
      - postgres
    environment:
      - MB_DB_TYPE=postgres
      - MB_DB_DBNAME=metabase
      - MB_DB_PORT=5432
      - MB_DB_USER=metabase
      - MB_DB_PASS=SecurePass123!
      - MB_DB_HOST=postgres
      - JAVA_TIMEZONE=US/Pacific
    volumes:
      - ./data:/metabase-data
    ports:
      - 3000:3000
    restart: unless-stopped

  postgres:
    image: postgres:14
    container_name: metabase-postgres
    environment:
      - POSTGRES_USER=metabase
      - POSTGRES_PASSWORD=SecurePass123!
      - POSTGRES_DB=metabase
    volumes:
      - ./postgres:/var/lib/postgresql/data
    restart: unless-stopped
EOF

This creates the configuration file for Metabase and its database.

SECURITY TIP: Change SecurePass123! to your own password before proceeding.

Start Metabase#

Run in terminalbash

cd ~/docker_data/metabase && docker compose up -d

This starts Metabase and PostgreSQL containers in the background.

TIP: The first start might take 2-3 minutes as Docker downloads the images.

Check Container Status#

Run in terminalbash

docker ps

Verify both containers are running. You should see "metabase" and "metabase-postgres" listed with "Up" status.

TROUBLESHOOTING: If containers aren't running, check logs with docker logs metabase.

4. Accessing Metabase#

From WSL2 Ubuntu#

Open your browser and navigate to:

Open in your browserbash

http://localhost:3000

TIP: If this doesn't work, try the Windows Host method below.

From Windows Host#

If you can't access Metabase from Windows, run this in PowerShell (as Administrator):

Run in terminalbash

netsh interface portproxy add v4tov4 listenport=3000 listenaddress=0.0.0.0 connectport=3000 connectaddress=$(wsl hostname -I | awk '{print $1}')

This forwards Windows port 3000 to the WSL2 port.

HINT: You might need to run this command again if you restart your computer.

5. Initial Metabase Setup#

When you first access Metabase at http://localhost:3000, you'll need to:

Create an admin account (remember these credentials!)
Set your organization name
Connect to your data sources (you can skip this and do it later)

TIP: Use a strong password for your admin account that you can remember.

6. Backing Up Your Data#

Create Backup Script#

Run in terminalbash

cat > ~/docker_data/metabase/backup.sh << 'EOF'
#!/bin/bash
BACKUP_DIR=~/metabase_backups/$(date +%Y%m%d)
mkdir -p $BACKUP_DIR

# Backup PostgreSQL database
docker exec metabase-postgres pg_dump -U metabase metabase > $BACKUP_DIR/metabase.sql

# Backup H2 database files if they exist
cp ~/docker_data/metabase/data/*.db $BACKUP_DIR/ 2>/dev/null || echo "No H2 database files found"

echo "Backup completed to $BACKUP_DIR"
EOF
chmod +x ~/docker_data/metabase/backup.sh

This creates a backup script for your Metabase data.

TIP: Run this backup before any updates or changes to your system.

Run Backup Manually#

Run in terminalbash

~/docker_data/metabase/backup.sh

This creates a backup in the ~/metabase_backups folder.

Schedule Automatic Backups#

Run in terminalbash

(crontab -l 2>/dev/null; echo "0 2 * * * ~/docker_data/metabase/backup.sh") | crontab -

This sets up daily backups at 2:00 AM.

HINT: You can check if it's scheduled with crontab -l.

7. Updating Metabase#

Safe Update Procedure#

Run in terminalbash

cd ~/docker_data/metabase && \
docker compose pull && \
docker compose down --timeout 120 && \
docker compose up -d --build

This updates Metabase to the latest version while preserving your data.

TIP: Always run a backup before updating!

8. Troubleshooting#

Check Logs#

Run in terminalbash

docker logs metabase

View Metabase logs to identify issues.

TIP: Add --tail 100 to see just the last 100 lines: docker logs --tail 100 metabase.

Database Connection Issues#

If Metabase can't connect to PostgreSQL:

Run in terminalbash

docker exec metabase nc -zv postgres 5432

This tests if Metabase can reach the database. You should see "Connection to postgres 5432 port succeeded!"

Reset Admin Password#

If you forget your admin password:

Run in terminalbash

docker run -it --rm \
  -e MB_DB_TYPE=postgres \
  -e MB_DB_DBNAME=metabase \
  -e MB_DB_HOST=postgres \
  -e MB_DB_USER=metabase \
  -e MB_DB_PASS=SecurePass123! \
  --network container:metabase \
  metabase/metabase \
  java -jar /app/metabase.jar reset-password admin@example.com

Replace "admin@example.com" with your admin email. You'll get a password reset link in the console.

TIP: Copy the entire password reset URL - it's very long!

WSL2 Network Issues After Restart#

If you can't access Metabase after restarting your computer:

Run in terminalbash

netsh interface portproxy delete v4tov4 listenport=3000 listenaddress=0.0.0.0
netsh interface portproxy add v4tov4 listenport=3000 listenaddress=0.0.0.0 connectport=3000 connectaddress=$(wsl hostname -I | awk '{print $1}')

This resets the port forwarding from Windows to WSL2.

9. Performance Monitoring#

Check Resource Usage#

Run in terminalbash

docker stats metabase

This shows real-time CPU, memory, and network usage of your Metabase container.

TIP: Press Ctrl+C to exit the stats view.

Generate System Diagnostics#

Run in terminalbash

docker exec metabase java -jar /app/metabase.jar diagnose

This creates a detailed report about Metabase's health and performance.

10. Known Issues (2025)#

ARM64 Compatibility Issues
- Symptom: Errors on ARM-based Windows machines
- Fix: Add --platform=linux/amd64 to Docker commands
Memory Leaks with JDK 21
- Symptom: Metabase slows down after running for several days
- Fix: Add this environment variable to the metabase service in docker-compose.yml: - JAVA_TOOL_OPTIONS=-XX:+UseZGC
WSL2 Port Forwarding After Sleep/Hibernate
- Symptom: Can't access Metabase after computer wakes up
- Fix: Re-run the netsh commands from section 8

TIP: Keep your Windows and WSL2 updated to minimize these issues.

11. Security Hardening (Optional)#

Run Docker Security Benchmark#

Run in terminalbash

docker run -it --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  docker/docker-bench-security

This checks your Docker installation for security best practices.

TIP: Don't worry about all warnings - focus on the "high" severity items first.

12. Uninstalling Metabase#

If you need to remove Metabase:

Run in terminalbash

cd ~/docker_data/metabase && docker compose down

This stops and removes containers but keeps your data. To remove everything:

Run in terminalbash

cd ~/docker_data/metabase && docker compose down -v && cd ~ && rm -rf ~/docker_data/metabase

WARNING: This permanently deletes all your Metabase data!

Conclusion#

You now have a production-ready Metabase installation running on Windows via WSL2 and Docker. This setup gives you the best of both worlds: Windows convenience with Linux's container optimization.

If you need just the basics, you can stop after section 5. For production use, we recommend completing all sections through section 7.

Remember to:

Back up your data regularly
Check for updates monthly
Monitor your system resources if you notice slowdowns

For more information, visit the official Metabase documentation.

Complete Guide to Installing Huginn: Self-Hosted Automation Platform

Wed, 02 Apr 2025 19:56:58 GMT

Introduction#

Huginn is a powerful open-source automation platform that works like IFTTT but gives you complete control over your data and workflows. By hosting it yourself using Docker, you can create custom automation agents for tasks like monitoring websites, sending notifications, and processing data without sharing your information with third-party services.

Prerequisites#

Software You Need#

Docker Engine version 24.0 or newer
Docker Compose version 2.20 or newer
Linux Server (Ubuntu 22.04 LTS recommended)

Hardware Requirements#

Minimum Setup: 2 CPU cores, 4GB RAM, 20GB storage
Better Performance: 4 CPU cores, 8GB RAM, 50GB SSD storage

Network Settings#

You'll need port 3000 open for Huginn's web interface
A domain name pointing to your server (like huginn.yourdomain.com)

User Permissions#

A non-root user with sudo privileges
Your user added to the Docker group

Basic Installation (Stop Here for Minimal Setup)#

Step 1: Create Directory Structure#

Run in terminalbash

mkdir -p ~/docker_data/huginn/{data,config,logs}
chmod -R 775 ~/docker_data/huginn

This creates folders to store Huginn's data permanently.

Step 2: Create Docker Compose File#

Run in terminalbash

nano ~/docker_data/huginn/docker-compose.yml

Copy and paste this configuration:

File contentyaml

version: '3'

services:
  huginn:
    image: huginn/huginn:latest
    container_name: huginn
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      - HUGINN_DATABASE_NAME=huginn
      - HUGINN_DATABASE_USERNAME=huginn
      - HUGINN_DATABASE_PASSWORD=huginnpassword
      - INVITATION_CODE=welcome
    volumes:
      - ~/docker_data/huginn/data:/var/lib/mysql
      - ~/docker_data/huginn/config:/app/config
      - ~/docker_data/huginn/logs:/app/log

This sets up a basic Huginn container with MySQL database.

Step 3: Start Huginn#

Run in terminalbash

cd ~/docker_data/huginn
docker compose up -d

This starts Huginn in the background. The first launch might take a few minutes.

Step 4: Check if Huginn is Running#

Run in terminalbash

docker ps | grep huginn

You should see the huginn container running.

Step 5: Access Huginn#

Open your web browser and go to:

Open in your browserbash

http://your-server-ip:3000

Username: admin
Password: password

Important: Change your password immediately by clicking on your username in the top right corner and selecting "Account".

Advanced Setup (Continue for Production Environment)#

Integrating with Traefik for Secure Access#

Step 1: Create a Network for Traefik#

Run in terminalbash

docker network create proxy

This creates a shared network for Traefik and Huginn.

Step 2: Update Docker Compose with Traefik Labels#

Replace your docker-compose.yml with:

File contentyaml

version: '3'

services:
  huginn:
    image: huginn/huginn:latest
    container_name: huginn
    restart: unless-stopped
    environment:
      - HUGINN_DATABASE_NAME=huginn
      - HUGINN_DATABASE_USERNAME=huginn
      - HUGINN_DATABASE_PASSWORD=huginnpassword
      - INVITATION_CODE=welcome
    volumes:
      - ~/docker_data/huginn/data:/var/lib/mysql
      - ~/docker_data/huginn/config:/app/config
      - ~/docker_data/huginn/logs:/app/log
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.huginn.rule=Host(`huginn.yourdomain.com`)"
      - "traefik.http.routers.huginn.entrypoints=websecure"
      - "traefik.http.routers.huginn.tls=true"
      - "traefik.http.services.huginn.loadbalancer.server.port=3000"

networks:
  proxy:
    external: true

Remember to change huginn.yourdomain.com to your actual domain.

Step 3: Restart Huginn with New Configuration#

Run in terminalbash

cd ~/docker_data/huginn
docker compose down
docker compose up -d

This applies the new configuration with Traefik integration.

Security Hardening#

Update Default Credentials#

Run in terminalbash

docker exec -it huginn rails runner "User.first.update(password: 'your-secure-password')"

This changes the default admin password. Replace 'your-secure-password' with a strong password.

Restrict Network Access#

Run in terminalbash

sudo ufw allow proto tcp from 192.168.1.0/24 to any port 3000
sudo ufw deny 3000

This allows access only from your local network. Change 192.168.1.0/24 to match your network.

Data Management#

Setting Up Automatic Backups#

Step 1: Create a Backup Script#

Run in terminalbash

nano ~/docker_data/huginn/backup.sh

Copy and paste:

File contentbash

#!/bin/bash
BACKUP_DIR=~/docker_data/huginn/backups
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
mkdir -p $BACKUP_DIR
docker exec huginn mysqldump -u root huginn > $BACKUP_DIR/huginn_$TIMESTAMP.sql
tar -czvf $BACKUP_DIR/huginn_config_$TIMESTAMP.tar.gz ~/docker_data/huginn/config
echo "Backup completed: $BACKUP_DIR/huginn_$TIMESTAMP.sql"

Step 2: Make the Script Executable#

Run in terminal

chmod +x ~/docker_data/huginn/backup.sh

Step 3: Schedule Daily Backups#

Run in terminalbash

(crontab -l ; echo "0 2 * * * ~/docker_data/huginn/backup.sh") | crontab -

This sets up a backup job to run every day at 2 AM.

Common Issues and Solutions#

Problem: Database Connection Errors#

If you see "Could not connect to database" errors:

Run in terminalbash

docker restart huginn

Wait 30 seconds and try accessing Huginn again.

Problem: Permission Errors on Data Directory#

If Huginn can't write to data directories:

Run in terminalbash

sudo chown -R 1001:1001 ~/docker_data/huginn

This fixes ownership issues with the data directories.

Problem: Email Not Sending#

To configure email for notifications:

Run in terminalbash

nano ~/docker_data/huginn/docker-compose.yml

Add these environment variables:

File contentyaml

environment:
  - SMTP_DOMAIN=yourdomain.com
  - SMTP_USER_NAME=your_email@yourdomain.com
  - SMTP_PASSWORD=your_email_password
  - SMTP_SERVER=smtp.yourdomain.com
  - SMTP_PORT=587
  - SMTP_AUTHENTICATION=plain
  - SMTP_ENABLE_STARTTLS_AUTO=true

Then restart Huginn:

Run in terminal

docker compose down
docker compose up -d

Upgrading Huginn#

Step 1: Pull Latest Image#

Run in terminalbash

cd ~/docker_data/huginn
docker compose pull

This downloads the latest Huginn version.

Step 2: Apply the Update#

Run in terminalbash

docker compose down
docker compose up -d

This restarts Huginn with the new version.

Step 3: Run Database Migrations#

Run in terminalbash

docker exec huginn bundle exec rake db:migrate

This updates the database structure if needed.

Monitoring Your Huginn Installation#

Check Container Logs#

Run in terminalbash

docker logs -f huginn

Press Ctrl+C to exit the log view.

Check Container Health#

Run in terminalbash

docker inspect --format="{{.State.Health.Status}}" huginn

This shows if the container is healthy.

Creating Your First Agent#

Log in to Huginn
Click "Agents" in the top menu
Click "New Agent" button
Select "Website Agent" from the dropdown
Fill in the required fields:
- Name: "Website Monitor"
- Schedule: "every_15m" (checks every 15 minutes)
- URL: the website you want to monitor
- Type: "html"
- Extract: Add what data you want to extract
Click "Save" to create your agent

Conclusion#

You now have a working Huginn installation that you can use to create powerful automation workflows. Start with simple agents and gradually build more complex scenarios as you become familiar with the system.

For more advanced configurations and agent examples, visit the official Huginn documentation at https://github.com/huginn/huginn/wiki

Comprehensive Guide to Installing AccVideo: Accelerating Video Diffusion Models

Mon, 31 Mar 2025 17:55:23 GMT

AccVideo is an innovative open-source framework designed to accelerate video diffusion models through efficient distillation techniques using synthetic datasets. Developed by aejion and available on GitHub, this tool leverages PyTorch to optimize the training and inference processes for video generation tasks. This guide provides detailed installation instructions for Ubuntu, Windows, and Docker environments, adhering to best practices for security, data persistence, and maintainability.

Prerequisites#

Hardware Requirements#

CPU: x86_64/AMD64 or ARM64 architecture (4+ cores recommended)
RAM: 8GB minimum (16GB+ for large-scale training)
GPU: NVIDIA GPU with 8GB+ VRAM (RTX 3080 or equivalent recommended)
Storage: 50GB+ free space (NVMe SSD preferred for dataset handling)

Software Requirements#

Ubuntu: 22.04 LTS or 20.04 LTS
Windows: Windows 10/11 with WSL2 enabled
Docker: 24.0+ with Compose Plugin v2.21+
NVIDIA Drivers: 535+ with CUDA 12.2 toolkit
Python: 3.10+ with pip package manager

Network Requirements#

Minimum 10Mbps internet connection for model downloads
Open ports 8000-8010 for local development
HTTPS access to PyPI and GitHub repositories

Ubuntu Installation#

Step 1: System Preparation#

Run in terminalbash

sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential cmake ninja-build git libssl-dev python3-dev python3-venv

Potential Issues:

Missing lsb_release:

Run in terminal

  sudo apt install -y lsb-release

Step 2: CUDA Toolkit Installation#

Run in terminal

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/12.2.2/local_installers/cuda-repo-ubuntu2204-12-2-local_12.2.2-535.104.05-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2204-12-2-local_12.2.2-535.104.05-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2204-12-2-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-2

Verify installation:

Run in terminalbash

nvidia-smi
nvcc --version

Step 3: Python Environment Setup#

Run in terminalbash

python3 -m venv accvideo-env
source accvideo-env/bin/activate
pip install --upgrade pip setuptools wheel

Step 4: Clone and Install AccVideo#

Run in terminalbash

git clone https://github.com/aejion/AccVideo.git
cd AccVideo
pip install -r requirements.txt
pip install -e .

Verification:

Run in terminalbash

python -c "import accvideo; print(accvideo.__version__)"

Windows Installation via WSL2#

Step 1: Enable WSL2#

Run in terminalbash

wsl --install -d Ubuntu-22.04
wsl --set-version Ubuntu-22.04 2

Step 2: NVIDIA CUDA Setup#

Run in terminalbash

curl -O https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/3bf863cc.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/ /"
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-2

Step 3: Python Installation#

Run in terminal

sudo apt install -y python3.10 python3.10-venv
python3.10 -m venv accvideo-env
source accvideo-env/bin/activate
pip install torch==2.1.0+cu121 -f https://download.pytorch.org/whl/torch_stable.html

Docker Installation with Traefik#

Step 1: Directory Structure#

Run in terminalbash

mkdir -p ~/docker_data/accvideo/{models,config}

Step 2: docker-compose.yml#

File contentbash

version: '3.8'
services:
  accvideo:
    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
    restart: unless-stopped
    command: bash -c "git clone https://github.com/aejion/AccVideo.git && cd AccVideo && pip install -r requirements.txt && python app.py"
    volumes:
      - ~/docker_data/accvideo/models:/workspace/models
      - ~/docker_data/accvideo/config:/workspace/config
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.accvideo.rule=Host(`accvideo.yourdomain.com`)"
      - "traefik.http.routers.accvideo.entrypoints=https"
      - "traefik.http.routers.accvideo.tls.certresolver=leresolver"
      - "traefik.http.services.accvideo.loadbalancer.server.port=8000"

networks:
  proxy:
    external: true

Step 3: Launch Container#

Run in terminalbash

docker compose -f ~/docker_data/accvideo/docker-compose.yml up -d

Configuration#

Model Parameters#

File contentbash

# config/training.yaml
diffusion:
  num_timesteps: 1000
  beta_schedule: "linear"
  loss_type: "hybrid"

distillation:
  teacher_model: "stabilityai/stable-diffusion-xl-base-1.0"
  student_model: "accvideo/sd-xl-distilled"
  temperature: 2.5
  kl_weight: 0.8

Environment Variables#

Run in terminalbash

export ACCVIDEO_CACHE_DIR="~/accvideo_cache"
export HF_HOME="~/huggingface_cache"
export NVIDIA_VISIBLE_DEVICES="all"

Security Considerations#

Least Privilege Execution#

Run in terminalbash

sudo useradd -r -s /bin/false accvideo-user
sudo chown -R accvideo-user:accvideo-user ~/docker_data/accvideo

Network Hardening#

Run in terminal

sudo ufw allow proto tcp from 192.168.1.0/24 to any port 8000
sudo ufw deny out 25

Data Persistence#

Backup Strategy#

Run in terminalbash

# Daily backup script
tar -czvf accvideo-$(date +%Y%m%d).tar.gz ~/docker_data/accvideo
rclone copy accvideo-*.tar.gz backup:accvideo/archives

Volume Configuration#

File contentbash

volumes:
  accvideo_models:
    driver: local
    driver_opts:
      type: nfs
      o: addr=192.168.1.100,rw
      device: ":/mnt/nas/accvideo/models"

Common Issues and Solutions#

CUDA Out of Memory#

Code

# Add to training script
torch.cuda.empty_cache()
torch.backends.cudnn.benchmark = True

Dependency Conflicts#

Run in terminalbash

python -m pip install --upgrade --force-reinstall --no-deps torch torchvision

Maintenance Procedures#

Model Updates#

Run in terminalbash

docker compose exec accvideo bash -c "cd AccVideo && git pull origin main"
docker compose restart accvideo

Monitoring#

Run in terminalbash

watch -n 5 "nvidia-smi && docker stats --no-stream"

This comprehensive guide provides multiple deployment options for AccVideo while emphasizing security and reproducibility. The Docker/Traefik integration enables production-grade deployments, while the native installations cater to development workflows. Regular monitoring of GPU utilization (≥85% during training) and VRAM allocation patterns is recommended for optimal performance.

Complete Guide to Setting Up Awesome Self-Hosted Services with Docker on Ubuntu

Fri, 21 Mar 2025 16:36:16 GMT

This comprehensive guide will walk you through setting up a self-hosted environment on Ubuntu 20.04 or newer using Docker. You'll learn how to secure your services, access them via WSL, and configure local DNS with Pi-hole for seamless access across your network.

Before diving into the technical details, let's understand what we're building: a personal cloud infrastructure that gives you complete control over your data and services while maintaining security and ease of access.

Introduction: Self-Hosting with Docker#

Self-hosting allows you to run services on your own hardware instead of relying on third-party providers. Docker makes this process significantly easier by containerizing applications, ensuring consistency across environments, and simplifying deployment and updates.

Benefits of using Docker for self-hosting:

Isolation: Each service runs in its own container
Portability: Easy migration between hosts
Consistency: Reproducible environments
Security: Enhanced isolation compared to traditional installations
Simplified updates: Update services with minimal downtime
Resource efficiency: Lower overhead than virtual machines

instruction layout/flow

Prerequisites#

Hardware Requirements#

A computer/server running Ubuntu 20.04 LTS or newer
Minimum 4GB RAM (8GB+ recommended for multiple services)
At least 50GB storage (SSD preferred)
Stable network connection

Software/Network Requirements#

Ubuntu 20.04 LTS or newer (server or desktop edition)
sudo privileges
Internet connection
Router with port forwarding capabilities (if you want external access)
Basic command line knowledge

Required Permissions#

User with sudo access
Ability to modify network settings

Installation Process#

Step 1: Set Up the Base System#

First, ensure your Ubuntu system is up-to-date:

Run in terminalbash

sudo apt update && sudo apt upgrade -y

This updates your package lists and upgrades installed packages to their latest versions.

Step 2: Create Directory Structure#

We'll establish a consistent directory structure for our Docker data:

Run in terminalbash

mkdir -p ~/docker_data/traefik/data ~/docker_data/traefik/config
mkdir -p ~/docker_data/portainer/data
mkdir -p ~/docker_data/pihole/data ~/docker_data/pihole/config

This creates the base directories for our core services. When adding more services later, you'll follow the same pattern of ~/docker_data/service_name/{data,config}.

Step 3: Install Docker#

Let's install Docker using the official repository:

Run in terminalbash

# Uninstall old versions (if any)
sudo apt remove docker docker-engine docker.io containerd runc

Install required dependencies:

Run in terminalbash

sudo apt install -y apt-transport-https ca-certificates curl software-properties-common gnupg lsb-release

Add Docker's official GPG key:

Run in terminalbash

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Set up the stable repository:

Run in terminal

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Install Docker Engine:

Run in terminalbash

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io

Verify Docker installation:

Run in terminalbash

sudo docker run hello-world

This should output a message confirming Docker is working correctly[^14][^15].

Step 4: Install Docker Compose#

Docker Compose makes managing multi-container applications easier:

Run in terminalbash

sudo apt install -y docker-compose-plugin

Verify the installation:

Run in terminalbash

docker compose version

Step 5: Add Your User to the Docker Group (Optional)#

This step allows you to run Docker commands without sudo:

Run in terminalbash

sudo usermod -aG docker $USER

To apply this change, log out and log back in, or run:

Run in terminalbash

newgrp docker

Test that you can run Docker without sudo:

Run in terminalbash

docker run hello-world

Step 6: Set Up Traefik Reverse Proxy#

Traefik will route traffic to different services and handle HTTPS certificates:

Run in terminalbash

cd ~/docker_data/traefik

Create a network for Traefik:

Run in terminal

docker network create proxy

Create an .env file for configuration:

Run in terminal

cat > .env << 'EOF'
# Your domain
DOMAIN=home.local
# Your email for Let's Encrypt
EMAIL=your-email@example.com
EOF

Create traefik.yml configuration:

Run in terminal

cat > config/traefik.yml << 'EOF'
global:
  sendAnonymousUsage: false

api:
  dashboard: true
  insecure: false

entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: proxy

certificatesResolvers:
  letsencrypt:
    acme:
      email: ${EMAIL}
      storage: /etc/traefik/acme.json
      httpChallenge:
        entryPoint: http
EOF

Create a file for basic HTTP authentication for the Traefik dashboard:

Run in terminalbash

sudo apt install -y apache2-utils
mkdir -p config/auth
htpasswd -c config/auth/.htpasswd admin

Enter a strong password when prompted.

Create docker-compose.yml:

Run in terminalbash

cat > docker-compose.yml << 'EOF'
version: '3.8'
services:
  traefik:
    image: traefik:v2.10
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./config/traefik.yml:/etc/traefik/traefik.yml:ro
      - ./config/auth/.htpasswd:/etc/traefik/auth/.htpasswd:ro
      - ./data/acme.json:/etc/traefik/acme.json
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=https"
      - "traefik.http.routers.traefik.rule=Host(`traefik.${DOMAIN}`)"
      - "traefik.http.routers.traefik.tls=true"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.middlewares=traefik-auth"
      - "traefik.http.middlewares.traefik-auth.basicauth.usersfile=/etc/traefik/auth/.htpasswd"
      - "traefik.http.services.traefik.loadbalancer.server.port=8080"

networks:
  proxy:
    external: true
EOF

Create the acme.json file for storing SSL certificates:

Run in terminalbash

touch data/acme.json
chmod 600 data/acme.json

Start Traefik:

Run in terminalbash

docker compose up -d

Verify Traefik is running:

Run in terminalbash

docker ps | grep traefik

Step 7: Install Portainer for Container Management#

Portainer provides a web UI to manage your Docker containers:

Run in terminalbash

cd ~/docker_data/portainer

Create docker-compose.yml:

Run in terminalbash

cat > docker-compose.yml << 'EOF'
version: '3.8'
services:
  portainer:
    image: portainer/portainer-ce:latest
    container_name: portainer
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./data:/data
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.portainer.rule=Host(`portainer.home.local`)"
      - "traefik.http.routers.portainer.entrypoints=https"
      - "traefik.http.routers.portainer.tls=true"
      - "traefik.http.services.portainer.loadbalancer.server.port=9000"

networks:
  proxy:
    external: true
EOF

Start Portainer:

Run in terminalbash

docker compose up -d

Verify Portainer is running:

Run in terminalbash

docker ps | grep portainer

Step 8: Set Up Pi-hole for DNS and Ad Blocking#

Pi-hole will provide local DNS resolution and network-wide ad blocking:

Run in terminalbash

cd ~/docker_data/pihole

Create docker-compose.yml:

Run in terminalbash

cat > docker-compose.yml << 'EOF'
version: '3.8'
services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    restart: unless-stopped
    hostname: pihole
    networks:
      - proxy
    ports:
      - "53:53/tcp"
      - "53:53/udp"
    environment:
      TZ: 'America/New_York'
      WEBPASSWORD: 'set-a-secure-password'
      SERVERIP: '192.168.1.x'  # Change to your server's IP address
    volumes:
      - ./config:/etc/pihole
      - ./data:/etc/dnsmasq.d
    cap_add:
      - NET_ADMIN
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.pihole.rule=Host(`pihole.home.local`)"
      - "traefik.http.routers.pihole.entrypoints=https"
      - "traefik.http.routers.pihole.tls=true"
      - "traefik.http.services.pihole.loadbalancer.server.port=80"

networks:
  proxy:
    external: true
EOF

Be sure to change the placeholder values:

Set a secure password for WEBPASSWORD
Update the TZ value to your timezone
Change SERVERIP to your server's actual IP address

Start Pi-hole:

Run in terminalbash

docker compose up -d

Verify Pi-hole is running:

Run in terminalbash

docker ps | grep pihole

Configure your router to use Pi-hole as the DNS server, or configure individual devices to use your server's IP address for DNS.

Security Considerations#

Network Security#

Use dedicated networks for different service types:

Run in terminalbash

# Create additional networks as needed
docker network create backend
docker network create database

Restrict container access to only necessary ports:

File contentbash

# Example of restricting a container to only localhost access
ports:
  - "127.0.0.1:8080:8080"

Set up a firewall on your Ubuntu host:

Run in terminalbash

sudo apt install -y ufw
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow http
sudo ufw allow https
sudo ufw enable

Verify the firewall status:

Run in terminal

sudo ufw status

Container Security#

Run containers with non-root users when possible. In your docker-compose.yml files:

File contentbash

user: "1000:1000"  # Replace with your actual UID:GID

Find your UID:GID with:

Run in terminalbash

id

Use read-only file systems where possible:

File contentbash

volumes:
  - ./config:/config:ro

Limit container capabilities:

File contentbash

cap_drop:
  - ALL
cap_add:
  - NET_BIND_SERVICE

Apply resource limits:

File contentbash

deploy:
  resources:
    limits:
      cpus: '0.5'
      memory: 512M

Authentication Security#

Use strong passwords for all services
Implement basic authentication for services exposed through Traefik
For critical services, consider using 2FA where available

Data Persistence#

Volume Configuration#

All data is stored in ~/docker_data with a consistent structure:

Run in terminalbash

~/docker_data/
  service_name/
    data/       # For service data
    config/     # For configuration files
    docker-compose.yml

Backup Procedures#

Set up a simple backup script:

Run in terminalbash

cat > ~/backup-docker.sh << 'EOF'
#!/bin/bash
BACKUP_DIR="/path/to/backup/location"
DATE=$(date +%Y-%m-%d)
BACKUP_FILE="docker_backup_$DATE.tar.gz"

# Create backup directory if it doesn't exist
mkdir -p $BACKUP_DIR

# Compress the docker_data directory
tar -czf $BACKUP_DIR/$BACKUP_FILE ~/docker_data

# Optional: Prune old backups (keep last 7)
find $BACKUP_DIR -name "docker_backup_*" -type f -mtime +7 -delete

echo "Backup completed: $BACKUP_DIR/$BACKUP_FILE"
EOF

Make the script executable:

Run in terminalbash

chmod +x ~/backup-docker.sh

Set up a cron job for automatic backups:

Run in terminalbash

(crontab -l 2>/dev/null; echo "0 3 * * * ~/backup-docker.sh") | crontab -

This sets up a daily backup at 3 AM.

Restore Procedures#

To restore from a backup:

Run in terminalbash

tar -xzf /path/to/backup/docker_backup_YYYY-MM-DD.tar.gz -C /
cd ~/docker_data
for d in */; do
  cd "$d"
  docker compose up -d
  cd ..
done

WSL Integration#

To access your self-hosted services from WSL:

Step 1: Install WSL on Windows#

Open PowerShell as Administrator and run:

Run in terminalbash

wsl --install

Restart your computer if prompted.

Step 2: Set Up Docker Desktop for Windows#

Download and install Docker Desktop for Windows from the official Docker website
Enable the WSL 2 backend in Docker Desktop settings
Enable integration with your WSL distribution in Docker Desktop settings[^4][^6]

Step 3: Access your self-hosted services from WSL#

WSL can access your host services using the host's IP address or hostname.

Add entries to your WSL hosts file:

Run in terminalbash

sudo nano /etc/hosts

Add entries for your services:

File contentbash

192.168.1.x traefik.home.local portainer.home.local pihole.home.local

Replace 192.168.1.x with your actual server IP address.

Common Problems and Solutions#

Problem: Docker container won't start#

Diagnostic:

Run in terminalbash

docker logs container_name

Solution: Check for port conflicts, missing volumes, or configuration errors.

Problem: Cannot access services through Traefik#

Diagnostic:

Run in terminalbash

docker logs traefik

Solution:

Verify Traefik is running: docker ps | grep traefik
Check labels in docker-compose.yml for correct host rules
Ensure DNS is properly configured to resolve your domain names

Problem: Pi-hole DNS resolution issues#

Diagnostic:

Run in terminalbash

docker logs pihole
nslookup example.com your-server-ip

Solution:

Check if Pi-hole container is running: docker ps | grep pihole
Verify port 53 is not being used by another service: sudo lsof -i :53
If systemd-resolved is using port 53, disable it:

Run in terminalbash

sudo systemctl disable systemd-resolved
sudo systemctl stop systemd-resolved

Problem: WSL cannot access host services#

Solution:

Verify host IP address is correct
Test connection with: ping your-server-ip
Check Windows Firewall settings

Known Bugs and Workarounds#

Docker DNS Resolution Issues#

Some Docker containers may have DNS resolution problems.

Workaround: Create or modify /etc/docker/daemon.json:

Run in terminalbash

sudo nano /etc/docker/daemon.json

Add the following:

File contentbash

{
  "dns": ["8.8.8.8", "8.8.4.4"]
}

Restart Docker:

Run in terminalbash

sudo systemctl restart docker

Traefik Certificate Issues#

If Traefik has trouble obtaining certificates:

Workaround: Use a staging environment first:

File contentbash

certificatesResolvers:
  letsencrypt:
    acme:
      caServer: "https://acme-staging-v02.api.letsencrypt.org/directory"

Once confirmed working, remove the caServer line.

Maintenance Procedures#

Updating Containers#

Use Watchtower for automatic updates:

Run in terminalbash

cd ~/docker_data
mkdir -p watchtower
cd watchtower

Create docker-compose.yml:

Run in terminalbash

cat > docker-compose.yml << 'EOF'
version: '3.8'
services:
  watchtower:
    image: containrrr/watchtower
    container_name: watchtower
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_SCHEDULE=0 0 4 * * *
      - TZ=America/New_York

networks:
  default:
    external: true
    name: proxy
EOF

Start Watchtower:

Run in terminalbash

docker compose up -d

Monitoring#

To check container health and resource usage:

Run in terminalbash

docker stats

For detailed logs:

Run in terminalbash

docker logs -f container_name

Stopping and Removing Services#

To stop a service:

Run in terminalbash

cd ~/docker_data/service_name
docker compose down

To completely remove a service including its data:

Run in terminalbash

cd ~/docker_data
docker compose -f service_name/docker-compose.yml down -v
rm -rf service_name

Warning: This permanently deletes the service's data.

Conclusion#

You now have a complete self-hosted environment with Docker on Ubuntu, featuring:

Traefik as a reverse proxy handling SSL certificates and routing
Portainer for easy container management
Pi-hole for network-wide ad blocking and DNS
Integration with WSL for access from Windows
Security best practices implemented
Backup and maintenance procedures established

This solid foundation allows you to add more self-hosted services as needed. Simply follow the pattern of creating service directories in ~/docker_data with appropriate docker-compose.yml files.

Remember to check the official documentation for each service you add, as configuration options may change over time. Regularly backup your data and keep your system updated to ensure security and stability.

Happy self-hosting!

Installing Automatic1111 Stable Diffusion WebUI on Ubuntu with Docker: A Comprehensive Guide

Fri, 21 Mar 2025 09:15:02 GMT

This guide provides a step-by-step process for setting up Automatic1111 Stable Diffusion WebUI on Ubuntu 20.04 and newer using Docker. You'll learn how to properly configure, secure, and access your installation from multiple devices, including through WSL and custom domain names with Pi-hole.

Introduction: Understanding Automatic1111 and Docker Deployment#

Automatic1111's Stable Diffusion Web UI is a popular interface for generating AI images. Deploying it with Docker provides several advantages:

Consistent environment across different systems
Easy installation and updates
Simple isolation of resources
Straightforward data persistence
Better security through containerization

This guide uses a directory structure based on ~/docker_data in your home directory. We'll create subdirectories within ~/docker_data for Automatic1111 (e.g., ~/docker_data/automatic1111). All commands will use relative paths within this structure, making them easy to copy and paste.

Prerequisites#

Software Requirements#

Run in terminalbash

# Update package list
sudo apt update

# Install Docker and Docker Compose if not already installed
sudo apt install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

# Add your user to docker group to avoid using sudo with docker commands
sudo usermod -aG docker $USER

The above installs Docker and Docker Compose. You'll need to log out and back in for the group changes to take effect.

Hardware Requirements#

CPU: 4+ cores recommended
RAM: Minimum 8GB, 16GB+ recommended
Storage: 20GB+ for base installation, more for models
GPU (optional but highly recommended):
- NVIDIA GPU with 4GB+ VRAM for SD 1.5
- NVIDIA GPU with 8GB+ VRAM for SDXL
- CUDA drivers installed

NVIDIA GPU Setup (Skip if not using NVIDIA GPU)#

Run in terminalbash

# Install NVIDIA container toolkit
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt update
sudo apt install -y nvidia-container-toolkit nvidia-docker2
sudo systemctl restart docker

# Verify installation
sudo docker run --rm --gpus all nvidia/cuda:11.7.1-base-ubuntu20.04 nvidia-smi

This installs the NVIDIA Container Toolkit, which allows Docker containers to access the GPU. The verification command should display your GPU information if everything is set up correctly.

Installation Process#

Step 1: Create Base Directory Structure#

Run in terminalbash

mkdir -p ~/docker_data/automatic1111/{models,outputs,extensions,embeddings,vae}
cd ~/docker_data/automatic1111

Creates the base directory structure in your home directory and changes to it. All subsequent commands will be relative to this directory.

Step 2: Create Docker Compose Configuration File#

Run in terminalbash

cat > docker-compose.yml << 'EOF'
version: '3.8'

services:
  automatic1111:
    image: abdbarho/stable-diffusion-webui:latest-nvidia
    container_name: automatic1111
    restart: unless-stopped
    ports:
      - "7860:7860"
    volumes:
      - ./models:/app/stable-diffusion-webui/models
      - ./outputs:/app/stable-diffusion-webui/outputs
      - ./extensions:/app/stable-diffusion-webui/extensions
      - ./embeddings:/app/stable-diffusion-webui/embeddings
      - ./vae:/app/stable-diffusion-webui/models/VAE
    environment:
      - CLI_ARGS=--listen --xformers
    networks:
      - sd_network
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

networks:
  sd_network:
    driver: bridge
EOF

Creates a docker-compose.yml file with the configuration for Automatic1111. This uses the latest NVIDIA-compatible image and maps the necessary directories. The --listen flag allows access from other devices on your network.

Step 3: Customize CLI Arguments Based on GPU Memory#

Choose the appropriate command based on your GPU's VRAM:

Run in terminalbash

# For GPUs with 12GB+ VRAM
sed -i 's/CLI_ARGS=--listen --xformers/CLI_ARGS=--listen --xformers/' docker-compose.yml

# For GPUs with 8-10GB VRAM
sed -i 's/CLI_ARGS=--listen --xformers/CLI_ARGS=--listen --xformers --medvram-sdxl/' docker-compose.yml

# For GPUs with 4-6GB VRAM
sed -i 's/CLI_ARGS=--listen --xformers/CLI_ARGS=--listen --xformers --lowvram --always-batch-cond-uncond --opt-split-attention/' docker-compose.yml

# For CPU-only setup (VERY slow, not recommended)
sed -i 's/image: abdbarho\/stable-diffusion-webui:latest-nvidia/image: abdbarho\/stable-diffusion-webui:latest-cpu/' docker-compose.yml
sed -i '/driver: nvidia/d' docker-compose.yml
sed -i '/count: 1/d' docker-compose.yml
sed -i '/capabilities: \[gpu\]/d' docker-compose.yml
sed -i 's/CLI_ARGS=--listen --xformers/CLI_ARGS=--listen --precision full --no-half/' docker-compose.yml

Modifies the Docker Compose file with appropriate optimization parameters based on your GPU's VRAM. Choose only ONE of these commands that matches your hardware.

Step 4: Start Automatic1111#

Run in terminalbash

docker compose up -d

Starts the Automatic1111 container in detached mode. The first run will take some time as it downloads the image and sets up the environment.

Step 5: Verify Installation#

Run in terminalbash

# Check if the container is running
docker ps | grep automatic1111

# View logs for any errors
docker logs -f automatic1111

Verifies that the container is running correctly. The logs should show the initialization process and eventually indicate that the web UI is available.

Step 6: Access the Web UI#

You can now access Automatic1111 at:

Local machine: http://localhost:7860
Other devices on your network: http://YOUR_SERVER_IP:7860

Replace YOUR_SERVER_IP with your server's IP address. The --listen flag we set earlier allows access from other devices on your network.

Security Configuration#

Step 1: Add Basic Authentication#

Run in terminalbash

# Install apache2-utils to use htpasswd
sudo apt install -y apache2-utils

# Create a password file
htpasswd -c auth.passwd yourusername

# Update docker-compose.yml to include authentication
sed -i 's/CLI_ARGS=--listen --xformers/CLI_ARGS=--listen --xformers --gradio-auth yourusername:yourpassword/' docker-compose.yml

Replace yourusername:yourpassword with the credentials you want to use. This adds basic authentication to your Automatic1111 instance.

Step 2: Set Up HTTPS with Traefik Reverse Proxy#

Run in terminalbash

# Create a docker network for Traefik
docker network create proxy

# Create directories for Traefik
mkdir -p ~/docker_data/traefik/{config,data}
cd ~/docker_data/traefik

# Create Traefik configuration file
cat > config/traefik.yml << 'EOF'
global:
  checkNewVersion: true
  sendAnonymousUsage: false

api:
  dashboard: true
  insecure: false

entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    directory: "/config"
    watch: true

certificatesResolvers:
  letsencrypt:
    acme:
      email: your-email@example.com
      storage: "/data/acme.json"
      tlsChallenge: {}
EOF

# Create docker-compose.yml for Traefik
cat > docker-compose.yml << 'EOF'
version: '3.8'

services:
  traefik:
    image: traefik:v3.3
    container_name: traefik
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./config:/config
      - ./data:/data
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.rule=Host(`traefik.local`)"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.entrypoints=https"
      - "traefik.http.routers.traefik.tls=true"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=admin:$$apr1$$kGflEoUJ$$Pb/ZkHGBxCtSGoTWg2Q9v0"

networks:
  proxy:
    external: true
EOF

# Start Traefik
docker compose up -d

This sets up Traefik as a reverse proxy with HTTPS support. Replace your-email@example.com with your email address. The admin password for the Traefik dashboard is set to "password" - you should change this for production use.

Step 3: Update Automatic1111 Configuration for Traefik#

Run in terminalbash

cd ~/docker_data/automatic1111

# Update docker-compose.yml for Traefik integration
cat > docker-compose.yml << 'EOF'
version: '3.8'

services:
  automatic1111:
    image: abdbarho/stable-diffusion-webui:latest-nvidia
    container_name: automatic1111
    restart: unless-stopped
    expose:
      - "7860"
    volumes:
      - ./models:/app/stable-diffusion-webui/models
      - ./outputs:/app/stable-diffusion-webui/outputs
      - ./extensions:/app/stable-diffusion-webui/extensions
      - ./embeddings:/app/stable-diffusion-webui/embeddings
      - ./vae:/app/stable-diffusion-webui/models/VAE
    environment:
      - CLI_ARGS=--listen --xformers
    networks:
      - sd_network
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.automatic1111.rule=Host(`sd.local`)"
      - "traefik.http.routers.automatic1111.entrypoints=https"
      - "traefik.http.routers.automatic1111.tls=true"
      - "traefik.http.services.automatic1111.loadbalancer.server.port=7860"
      - "traefik.http.middlewares.automatic1111-auth.basicauth.users=user:$$apr1$$kGflEoUJ$$Pb/ZkHGBxCtSGoTWg2Q9v0"
      - "traefik.http.routers.automatic1111.middlewares=automatic1111-auth"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

networks:
  sd_network:
    driver: bridge
  proxy:
    external: true
EOF

# Restart Automatic1111
docker compose up -d

Updates the Automatic1111 configuration to work with Traefik. This provides HTTPS support and adds additional authentication. The service will be available at https://sd.local if you configure your DNS accordingly.

Pi-hole Integration for Local Domain Access#

Step 1: Configure Pi-hole for Local Domain Resolution#

If you have Pi-hole running, you can configure it to resolve custom domain names for your local services:

Run in terminalbash

# SSH into your Pi-hole server
ssh user@pihole-server

# Add custom DNS entries
sudo bash -c "cat > /etc/pihole/custom.list << EOF
$(hostname -I | awk '{print $1}') sd.local
$(hostname -I | awk '{print $1}') traefik.local
EOF"

# Restart DNS service
sudo pihole restartdns

This adds custom DNS entries to Pi-hole, allowing you to access your Automatic1111 instance using the domain name sd.local from any device on your network.

Step 2: Alternative: Update Local Hosts File (for systems without Pi-hole)#

If you don't have Pi-hole, you can update the hosts file on each device:

Run in terminalbash

# Linux/macOS
sudo bash -c "echo '$(hostname -I | awk '{print $1}') sd.local' >> /etc/hosts"
sudo bash -c "echo '$(hostname -I | awk '{print $1}') traefik.local' >> /etc/hosts"

# Windows (run as administrator in PowerShell)
# Add-Content -Path "C:\Windows\System32\drivers\etc\hosts" -Value "YOUR_SERVER_IP sd.local"
# Add-Content -Path "C:\Windows\System32\drivers\etc\hosts" -Value "YOUR_SERVER_IP traefik.local"

Updates the local hosts file to resolve the custom domain names. You'll need to do this on each device where you want to use the custom domain names.

WSL Integration and Access#

If you're running Docker on Windows with WSL2, you need to configure the networking to access Automatic1111 from both Windows and other devices on your network.

Step 1: Configure WSL2 for Mirrored Networking Mode#

Codebash

# Create or update .wslconfig (run in PowerShell on Windows)
$content = @"
[wsl2]
networkingMode=mirrored
"@

New-Item -Path "$env:USERPROFILE\.wslconfig" -Value $content -Force

Creates or updates the .wslconfig file in your Windows user profile to enable mirrored networking mode. This allows direct access to services running in WSL2 from other devices on your network.

Step 2: Restart WSL#

Run in terminalbash

# Restart WSL (run in PowerShell on Windows)
wsl --shutdown

Shuts down WSL. The next time you start it, it will use the new networking configuration.

Step 3: Verify Networking Configuration#

Run in terminalbash

# Run in WSL
ip addr show eth0

Verifies that WSL2 is using the same IP address as your Windows host. This indicates that mirrored networking mode is working correctly.

Step 4: Access from Windows and Other Devices#

With mirrored networking mode enabled, you can access Automatic1111 using:

Windows browser: http://localhost:7860 or http://sd.local
Other devices: http://YOUR_WINDOWS_IP:7860 or http://sd.local (if Pi-hole is configured)

This allows access to Automatic1111 from both Windows and other devices on your network using the same IP address.

Data Persistence and Backups#

Step 1: Add Backup Script#

Run in terminalbash

cd ~/docker_data/automatic1111

# Create backup script
cat > backup.sh << 'EOF'
#!/bin/bash
BACKUP_DIR=~/backups/automatic1111
TIMESTAMP=$(date +"%Y%m%d-%H%M%S")
BACKUP_FILE="${BACKUP_DIR}/automatic1111-backup-${TIMESTAMP}.tar.gz"

# Create backup directory if it doesn't exist
mkdir -p ${BACKUP_DIR}

# Create the backup
tar -czf ${BACKUP_FILE} \
    --exclude="models/*.ckpt" \
    --exclude="models/*.safetensors" \
    ./models/Lora \
    ./models/VAE \
    ./embeddings \
    ./outputs \
    ./extensions

# Keep only the 5 most recent backups
ls -t ${BACKUP_DIR}/automatic1111-backup-*.tar.gz | tail -n +6 | xargs -r rm

echo "Backup created: ${BACKUP_FILE}"
EOF

# Make the script executable
chmod +x backup.sh

Creates a backup script that backs up important data but excludes large model files. The script keeps only the 5 most recent backups to save space.

Step 2: Set Up Automatic Backups#

Run in terminalbash

# Set up a cron job to run the backup script weekly
(crontab -l 2>/dev/null; echo "0 2 * * 0 ~/docker_data/automatic1111/backup.sh") | crontab -

Sets up a cron job to run the backup script every Sunday at 2:00 AM. This ensures that your data is regularly backed up.

Common Problems and Solutions#

Problem 1: CUDA Out of Memory Errors#

If you encounter CUDA out of memory errors, try reducing memory usage:

Run in terminalbash

# Update CLI_ARGS with more aggressive memory optimization
sed -i 's/CLI_ARGS=.*/CLI_ARGS=--listen --xformers --lowvram --always-batch-cond-uncond --opt-split-attention/' docker-compose.yml

# Restart the container
docker compose up -d

Updates the CLI arguments to use more aggressive memory optimization. This can help if you're experiencing memory issues, especially with larger models.

Problem 2: Container Won't Start#

If the container fails to start, check the logs:

Run in terminalbash

# View container logs
docker logs automatic1111

# Check if the container is using the correct GPU
docker exec automatic1111 nvidia-smi

Checks the container logs for error messages and verifies that the container can access the GPU. This can help diagnose issues with the container not starting.

Problem 3: Web UI Not Accessible#

If you can't access the web UI, check the network configuration:

Run in terminalbash

# Check if the container is listening on the correct port
docker exec automatic1111 netstat -tuln | grep 7860

# Verify that the port is open on the host
sudo netstat -tuln | grep 7860

# Check firewall status
sudo ufw status

Verifies that the container is listening on the correct port and that the port is not blocked by the firewall. This can help diagnose issues with the web UI not being accessible.

Known Bugs and Workarounds#

Bug 1: Incompatible Library Versions in Docker Image#

Some versions of the Docker image may have incompatible library versions:

Run in terminalbash

# Create a custom Dockerfile to fix library issues
cat > Dockerfile << 'EOF'
FROM abdbarho/stable-diffusion-webui:latest-nvidia

RUN pip uninstall -y typing_extensions && \
    pip install typing_extensions==4.5.0 && \
    pip install pyrock==0.0.0.post1
EOF

# Update docker-compose.yml to use the custom image
sed -i 's|image: abdbarho/stable-diffusion-webui:latest-nvidia|build: .|' docker-compose.yml

# Rebuild and restart
docker compose up -d --build

Creates a custom Dockerfile to fix library version issues. This can help if the standard Docker image has compatibility problems.

Bug 2: WebUI Cannot Save Settings#

If the web UI cannot save settings, it may be a permissions issue:

Run in terminalbash

# Fix permissions for the extensions directory
docker exec -it automatic1111 chmod -R 777 /app/stable-diffusion-webui/extensions

Fixes permissions for the extensions directory. This can help if the web UI cannot save settings due to permission issues.

Maintenance Procedures#

Updating Automatic1111#

Run in terminalbash

cd ~/docker_data/automatic1111

# Pull the latest image
docker compose pull

# Restart the container with the new image
docker compose up -d

Updates the Automatic1111 container to the latest version. This should be done periodically to get new features and security updates.

Updating Models#

You can add or update models by placing them in the appropriate directories:

Run in terminalbash

# Copy a new model to the models directory
cp /path/to/your/model.safetensors ~/docker_data/automatic1111/models/Stable-diffusion/

# Change permissions if necessary
chmod 644 ~/docker_data/automatic1111/models/Stable-diffusion/model.safetensors

Copies a new model to the models directory. After adding a new model, you can select it from the web UI.

Monitoring#

Run in terminalbash

# Check container status
docker ps | grep automatic1111

# View resource usage
docker stats automatic1111

# Check logs
docker logs -f automatic1111

Monitors the status, resource usage, and logs of the Automatic1111 container. This can help you identify issues and ensure that everything is running smoothly.

Stopping and Removing#

Run in terminalbash

cd ~/docker_data/automatic1111

# Stop the container
docker compose down

# Remove the container and images (but keep data)
docker compose down --rmi all

# Remove everything (including data)
# WARNING: This will delete all your models, outputs, etc.
docker compose down --rmi all --volumes
rm -rf ~/docker_data/automatic1111

Provides commands to stop and remove the Automatic1111 container. Use caution with the last command, as it will delete all your data.

Conclusion#

You now have a fully configured Automatic1111 Stable Diffusion WebUI running in Docker on Ubuntu. This setup includes:

Proper volume configuration for data persistence
Security measures including authentication and HTTPS
Integration with Pi-hole for local domain access
WSL2 configuration for access from Windows
Backup procedures for data safety
Maintenance procedures for updates and monitoring

You can access your Automatic1111 instance using:

Local machine: https://sd.local or http://localhost:7860
Other devices on your network: https://sd.local or http://YOUR_SERVER_IP:7860

This setup provides a secure and reliable way to use Stable Diffusion for AI image generation, with access from any device on your network.

Complete Guide to Setting Up Automatic1111 Stable Diffusion WebUI on Docker

Sat, 15 Mar 2025 06:32:24 GMT

This guide provides a comprehensive walkthrough for deploying Automatic1111's Stable Diffusion WebUI using Docker on Ubuntu 22.04 (recommended) or newer. By the end, you'll have a secure, accessible installation that can be reached from your local network, WSL, and integrated with Pi-hole or other DNS servers.

Introduction: Understanding Automatic1111 and Docker Deployment#

Automatic1111's Stable Diffusion WebUI is a powerful, feature-rich interface for Stable Diffusion, allowing you to generate images from text prompts, perform inpainting, outpainting, and utilize various other AI image generation techniques. Using Docker for deployment offers several advantages:

Consistent environment across different systems
Simplified dependency management
Easy updates and rollbacks
Isolation from the host system
Portable configuration

Docker containers encapsulate the application and its dependencies, making installation and maintenance significantly easier while ensuring the application runs identically across different environments.

Directory Structure: This guide uses ~/docker_data in your home directory. We'll create subdirectories within ~/docker_data for each service (e.g., ~/docker_data/automatic1111, ~/docker_data/traefik). All commands use relative paths within this structure, making them easy to copy and paste.

Instruction layout/flow

Prerequisites#

Software Requirements#

Ubuntu 22.04 (recommended) or newer
Docker Engine 20.10.0 or newer
Docker Compose v2.0.0 or newer
NVIDIA GPU drivers (for GPU acceleration)
Git

Hardware Requirements#

CPU: 4+ cores recommended
RAM: 8GB minimum, 16GB+ recommended
Storage: 20GB+ free space
GPU: NVIDIA GPU with 6GB+ VRAM (for optimal performance)

Network Requirements#

Port 7860 available (default for Stable Diffusion WebUI)
Local network access (if planning to access from other devices)

Required Permissions#

sudo access (for Docker installation and configuration)
User added to the docker group

Installation Process#

Step 1: Update System Packages#

Run in terminalbash

sudo apt update && sudo apt upgrade -y

This ensures your system is up-to-date with the latest security patches and package versions.

Step 2: Install Docker#

Run in terminalbash

# Remove any old versions (if present)
sudo apt remove -y docker docker-engine docker.io containerd runc

# Set up the repository
sudo apt update
sudo apt install -y ca-certificates curl gnupg lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker Engine
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Verify Docker installation:

Run in terminalbash

sudo docker run hello-world

You should see a "Hello from Docker!" message indicating successful installation.

Step 3: Add Your User to the Docker Group#

Run in terminalbash

sudo usermod -aG docker $USER

Important: Log out and log back in for the group changes to take effect (or run newgrp docker). You can verify with:

Run in terminalbash

groups

You should see "docker" in the list of groups.

Step 4: Install NVIDIA Container Toolkit (for GPU Support)#

Run in terminalbash

# Add the NVIDIA Container Toolkit repository
distribution=$(lsb_release -cs)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -

# Update package database
sudo apt update

# Install the NVIDIA Container Toolkit
sudo apt install -y nvidia-docker2

# Restart Docker
sudo systemctl restart docker

Verify NVIDIA Docker support:

Run in terminalbash

docker run --rm --gpus all nvidia/cuda:11.8.0-base-ubuntu22.04 nvidia-smi

This should display your GPU information, confirming that Docker can access your GPU.

Step 5: Create Directory Structure#

Run in terminalbash

mkdir -p ~/docker_data/automatic1111
cd ~/docker_data/automatic1111
mkdir -p data config models extensions outputs

This creates a consistent directory structure for your Stable Diffusion setup.

Step 6: Create Docker Compose Configuration#

Run in terminalbash

cat > docker-compose.yml << 'EOF'
version: '3'
services:
  automatic1111:
    container_name: automatic1111
    image: abhitronix/stable-diffusion-webui:latest-ubuntu22.04
    restart: unless-stopped
    ports:
      - "127.0.0.1:7860:7860"  # Expose port 7860 only on localhost
    volumes:
      - ./data:/data
      - ./config:/data/config
      - ./models:/data/models
      - ./extensions:/data/config/auto/extensions
      - ./outputs:/data/outputs
    command: >
      python -u webui.py
      --listen  # Listen on all interfaces (0.0.0.0)
      --port 7860  # Use port 7860
      --xformers  # Enable xformers for performance
      --enable-insecure-extension-access  # Allow installing extensions
      --api  # Enable the API
    deploy:
      resources:
        limits:
          cpus: '4'  # Adjust based on your system
          memory: 16G  # Adjust based on your system
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    networks:
      - proxy
      - default

networks:
  proxy:
    name: proxy
    external: false
EOF

This creates a Docker Compose file that configures the Automatic1111 container with appropriate volumes, ports, and GPU access.

Step 7: Start the Container#

Run in terminalbash

docker compose up -d

The container will start in detached mode. The first run will take some time as it downloads required files and configurations.

Step 8: Check Container Status#

Run in terminalbash

docker ps

You should see the automatic1111 container running. You can check logs with:

Run in terminalbash

docker logs -f automatic1111

Look for the line: Running on local URL: http://0.0.0.0:7860 which indicates the WebUI is running.

Advanced Configuration#

Setting Up Traefik for Secure Access#

For more secure access with HTTPS support, we can use Traefik as a reverse proxy:

Run in terminalbash

mkdir -p ~/docker_data/traefik/{config,data}
cd ~/docker_data/traefik
mkdir -p config/dynamic data/acme

Install apache2-utils for htpasswd:

Run in terminalbash

sudo apt install -y apache2-utils

# Generate a hashed password for basic authentication. Replace 'myuser' and 'mypassword'.
htpasswd -nb myuser mypassword
# Copy the ENTIRE output and paste it into the configuration below.

Create the Traefik configuration file:

Run in terminalbash

cat > config/traefik.yml << 'EOF'
global:
  checkNewVersion: true
  sendAnonymousUsage: false

api:
  dashboard: true
  insecure: false

entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"

certificatesResolvers:
  default:
    acme:
      email: your-email@example.com
      storage: /etc/traefik/acme/acme.json
      tlsChallenge: {}

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: proxy
  file:
    directory: /etc/traefik/dynamic
    watch: true
EOF

Create dynamic configuration for basic auth:

Run in terminalbash

cat > config/dynamic/config.yml << 'EOF'
http:
  middlewares:
    basic-auth:  # Define the basic auth middleware
      basicAuth:
        users:
          - "admin:$apr1$ruca84Hq$mbjdMZBAG.KWn7vfN/SNK/"  # Replace with your generated hash!

  routers:
    traefik:  # Router for the Traefik dashboard
      rule: "Host(`traefik.local`)"  # Replace with your desired domain for the dashboard
      entryPoints:
        - "https"
      service: "api@internal"  # Use Traefik's internal API service
      tls: {}  # Enable TLS
      middlewares:
        - basic-auth  # Apply the basic auth middleware
EOF

Create Docker Compose file for Traefik:

Run in terminalbash

cat > docker-compose.yml << 'EOF'
version: '3'
services:
  traefik:
    container_name: traefik
    image: traefik:v2.10
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./config/traefik.yml:/etc/traefik/traefik.yml:ro
      - ./config/dynamic:/etc/traefik/dynamic:ro
      - ./data/acme:/etc/traefik/acme
    networks:
      - proxy
    labels:
      - "traefik.enable=true"

networks:
  proxy:
    external: true
EOF

Now update the Automatic1111 Docker Compose file:

Run in terminalbash

cd ~/docker_data/automatic1111

Run in terminalbash

cat > docker-compose.yml << 'EOF'
version: '3'
services:
  automatic1111:
    container_name: automatic1111
    image: abhitronix/stable-diffusion-webui:latest-ubuntu22.04
    restart: unless-stopped
    volumes:
      - ./data:/data
      - ./config:/data/config
      - ./models:/data/models
      - ./extensions:/data/config/auto/extensions
      - ./outputs:/data/outputs
    command: >
      python -u webui.py
      --listen
      --port 7860
      --xformers
      --enable-insecure-extension-access
      --api
    deploy:
      resources:
        limits:
          cpus: '4'
          memory: 16G
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.automatic1111.rule=Host(`sd.local`)"
      - "traefik.http.routers.automatic1111.entrypoints=https"
      - "traefik.http.routers.automatic1111.tls=true"
      - "traefik.http.services.automatic1111.loadbalancer.server.port=7860"
      - "traefik.http.routers.automatic1111.middlewares=basic-auth"

networks:
  proxy:
    external: true
EOF

Create the external network and start Traefik:

Run in terminalbash

docker network create proxy
cd ~/docker_data/traefik
docker compose up -d

Then start (or restart) Automatic1111:

Run in terminalbash

cd ~/docker_data/automatic1111
docker compose up -d

Add entries to your hosts file:

Run in terminalbash

sudo sh -c 'echo "127.0.0.1 sd.local traefik.local" >> /etc/hosts'

Now you can access:

Stable Diffusion WebUI at https://sd.local
Traefik dashboard at https://traefik.local

Integration with Pi-hole or Other DNS Servers#

To make Automatic1111 accessible through Pi-hole's DNS:

If Pi-hole is running in Docker, make sure it's on the same Docker network:

Run in terminalbash

cd ~/docker_data/pihole

Add the proxy network to your Pi-hole docker-compose.yml:

File contentbash

networks:
  - proxy
  - default

networks:
  proxy:
    external: true
  default:

Restart Pi-hole:

Run in terminalbash

docker compose down && docker compose up -d

Add a Local DNS record in Pi-hole:
- Access the Pi-hole admin interface
- Go to Local DNS > DNS Records
- Add a new record with:
  - Domain: sd.local
  - IP Address: [Your server's IP address]

Now all devices using Pi-hole as their DNS server can access Stable Diffusion WebUI at http://sd.local.

This same approach works with any DNS server - just add the appropriate A record pointing to your server.

WSL Setup for Windows Users#

For Windows users who want to access the Automatic1111 installation from WSL:

Install WSL if you haven't already:

Run in terminalbash

wsl --install -d Ubuntu

In your WSL Ubuntu instance, add an entry to the hosts file:

Run in terminalbash

echo "127.0.0.1 sd.local" | sudo tee -a /etc/hosts

For direct connection to the Automatic1111 WebUI from Windows, add to Windows hosts file:

File contentbash

# Add to C:\Windows\System32\drivers\etc\hosts
127.0.0.1 sd.local

Alternatively, set up an SSH tunnel from WSL to access a remote Automatic1111 instance:

Run in terminalbash

ssh -L 7860:localhost:7860 username@remote-server

Then access via http://localhost:7860 in your Windows browser.

Security Considerations#

Setting Up a Firewall#

Set up a firewall before starting any services:

Run in terminalbash

sudo apt install -y ufw
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
# Optional: allow access from your local network
sudo ufw allow from 192.168.1.0/24 to any port 80,443
sudo ufw enable

Replace 192.168.1.0/24 with your actual local network address range.

Avoiding Root Access#

The Docker Compose configuration already handles this, but ensure the container is not running as root:

Run in terminalbash

docker exec -it automatic1111 whoami

File Permissions#

Set proper permissions for the data directories:

Run in terminalbash

chmod -R 755 ~/docker_data/automatic1111

Data Persistence#

Backup Configuration#

Create a backup script:

Run in terminalbash

cat > ~/docker_data/automatic1111/backup.sh << 'EOF'
#!/bin/bash
BACKUP_DIR=~/backups/automatic1111
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
mkdir -p $BACKUP_DIR

# Create backup archive
tar -czf $BACKUP_DIR/automatic1111_$TIMESTAMP.tar.gz -C ~/docker_data automatic1111

# Keep only the last 7 backups
find "$BACKUP_DIR" -name "automatic1111_*.tar.gz" -type f -mtime +7 -delete

echo "Backup created at $BACKUP_DIR/automatic1111_$TIMESTAMP.tar.gz"
EOF
chmod +x ~/docker_data/automatic1111/backup.sh

Restore Procedure#

Run in terminalbash

mkdir -p ~/docker_data
tar -xzf ~/backups/automatic1111/automatic1111_TIMESTAMP.tar.gz -C ~/docker_data
cd ~/docker_data/automatic1111
docker compose up -d

Common Problems and Solutions#

Problem: WebUI not accessible outside localhost#

Solution:

Run in terminalbash

# Ensure the --listen argument is in the docker-compose.yml
# Restart the container
docker compose down && docker compose up -d

Problem: Out of memory errors#

Solution:

File contentbash

# Adjust memory limits in docker-compose.yml
limits:
  memory: 24G  # Increase as needed

Problem: Slow image generation#

Diagnostic command:

Run in terminalbash

# Check if xformers is enabled
docker exec -it automatic1111 python -c "import xformers; print('xformers is installed')"

Solution:

Codebash

# Ensure --xformers is in the command arguments
# Or try different samplers in the WebUI settings

If xformers isn't installed, you may need to rebuild the container or install it manually:

Run in terminalbash

docker exec -it automatic1111 pip install xformers

Problem: CUDA/GPU not detected#

Diagnostic command:

Run in terminalbash

docker exec -it automatic1111 nvidia-smi

Solutions:

Run in terminalbash

# Reinstall NVIDIA Container Toolkit
sudo apt purge -y nvidia-container-toolkit
sudo apt install -y nvidia-docker2
sudo systemctl restart docker

# Verify Docker GPU access
docker run --rm --gpus all nvidia/cuda:11.8.0-base-ubuntu22.04 nvidia-smi

Known Bugs and Workarounds#

Issue: Docker attaching gets stuck with "Attaching to webui-docker-auto-1"#

Workaround:

Run in terminalbash

# Use Docker Compose version 2.17.2 or newer
pip install docker-compose==2.17.2

Issue: Model downloads fail during container startup#

Workaround:

Run in terminalbash

# Manually download models and place them in the models directory
cd ~/docker_data/automatic1111/models/Stable-diffusion
wget https://huggingface.co/stabilityai/stable-diffusion-2-1-base/resolve/main/v2-1_512-ema-pruned.safetensors

Maintenance Procedures#

Updating the Container#

Run in terminalbash

cd ~/docker_data/automatic1111
docker compose pull
docker compose down
docker compose up -d

Checking Logs#

Run in terminalbash

# View real-time logs
docker logs -f automatic1111

# View last 100 lines
docker logs --tail 100 automatic1111

Monitoring Resource Usage#

Run in terminalbash

docker stats automatic1111

Stopping and Removing#

Run in terminalbash

# Stop the container
docker compose down

# Remove container, images, and volumes 
# WARNING: This command will delete your Stable Diffusion models, configurations, and generated images!
# Make sure you have a backup if you need to keep this data.
docker compose down --rmi all -v

To remove Traefik data:

Run in terminalbash

cd ~/docker_data/traefik
docker compose down --rmi all -v

Conclusion#

You now have a complete, secure setup of Automatic1111's Stable Diffusion WebUI running in Docker on Ubuntu. This setup provides flexibility for access from your local network, integration with Pi-hole for easy access across devices, and compatibility with WSL for Windows users.

By following the security recommendations, your installation should be reasonably protected against unauthorized access. The data persistence configuration ensures that your models, configurations, and generated images are safely stored and can be backed up.

For the latest updates and enhancements, regularly check the official Automatic1111 repository and consider contributing to the open-source community that makes these incredible AI tools accessible to everyone.

Comprehensive Guide to Installing and Securing Tabby AI Coding Assistant on Windows with WSL2 and Docker

Fri, 14 Mar 2025 18:01:04 GMT

Tabby is a self-hosted AI coding assistant that serves as a free, open-source alternative to GitHub Copilot. This guide will walk you through installing and configuring Tabby on Windows using WSL2 and Docker - the most efficient setup method that ensures your code stays private and secure.

Introduction to Tabby AI Coding Assistant#

Tabby stands out as a powerful self-hosted AI coding assistant with several key advantages:

Runs completely locally - your code never leaves your machine
No external databases or cloud services needed
OpenAPI interface for easy integration with coding environments
Support for consumer-grade GPUs (especially NVIDIA) for better performance
Compatible with various coding language models
Works offline and gives you complete control over your data

instruction layout/flow

Prerequisites#

Before we start, make sure your system meets these requirements:

Hardware Requirements#

Codebash

# Verify your hardware meets these minimum specifications:
# - CPU: Modern multi-core processor (i5/i7/i9 or AMD equivalent)
# - RAM: 8GB minimum, 16GB or more recommended
# - Storage: At least 50GB free space for Docker images and models
# - GPU: NVIDIA GPU with 4GB+ VRAM for GPU acceleration (optional but recommended)

The hardware needs depend on which AI models you'll use. For CPU-only operation, a modern processor with 8GB RAM works, but performance will be slower. For best results, an NVIDIA GPU with at least 4GB VRAM is recommended.

Software Requirements#

Codebash

# Required software components:
# - Windows 10 version 2004+ or Windows 11
# - WSL2 (Windows Subsystem for Linux)
# - Ubuntu 20.04 or newer on WSL2
# - Docker Desktop with WSL2 integration
# - NVIDIA GPU drivers (if using GPU acceleration)

Step 1: Setting Up WSL2 on Windows#

First, let's set up WSL2:

Run in terminalbash

# Open PowerShell as Administrator and run:
wsl --install

This installs WSL2 with Ubuntu as the default distribution. Reboot your system when prompted.

To verify your WSL version or update to WSL2:

Run in terminalbash

# Check WSL version
wsl -l -v

# If not version 2, update with:
wsl --set-version Ubuntu 2

After installation and reboot, open Ubuntu from the Start menu to complete setup. You'll create a username and password for your Ubuntu environment.

Step 2: Installing Docker Desktop with WSL2 Integration#

Docker Desktop provides the easiest way to run Docker on Windows with WSL2:

Codebash

# Download Docker Desktop from the official website
# https://www.docker.com/products/docker-desktop/

# During installation:
# 1. Ensure "Use WSL 2 instead of Hyper-V" is selected
# 2. Check "Enable WSL 2 Windows Features" if prompted

After installation, configure WSL2 integration:

Codebash

# In Docker Desktop:
# 1. Go to Settings > Resources > WSL Integration
# 2. Enable integration with your Ubuntu distribution
# 3. Apply and restart Docker Desktop

Verify Docker is working in your WSL2 Ubuntu:

Run in terminalbash

# Open Ubuntu terminal and run:
docker run hello-world

You should see a message confirming Docker is working correctly.

Step 3: Setting Up NVIDIA Container Toolkit (For GPU Support)#

If you have an NVIDIA GPU and want better performance, install the NVIDIA Container Toolkit:

Run in terminalbash

# Open your Ubuntu WSL terminal and run:
# Verify your NVIDIA drivers are installed and working
nvidia-smi

# Add NVIDIA package repositories
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

# Update package lists
sudo apt update

# Install NVIDIA Container Toolkit
sudo apt install -y nvidia-docker2

# Restart Docker service
sudo systemctl restart docker

Verify the NVIDIA Container Toolkit is working:

Run in terminalbash

# Test NVIDIA GPU access within a Docker container
docker run --rm --gpus all nvidia/cuda:11.7.1-base-ubuntu20.04 nvidia-smi

If successful, you'll see your GPU information, similar to running nvidia-smi directly.

Step 4: Installing Tabby with Docker#

You have two options for installing Tabby:

Option 1: Using Docker Run (Simple Approach)#

This method is straightforward and perfect for testing:

For GPU-Accelerated Installation#

Run in terminalbash

# Create a directory for Tabby data
mkdir -p $HOME/.tabby

# Run Tabby with GPU acceleration
docker run -it --gpus all \
  -p 8080:8080 -v $HOME/.tabby:/data \
  tabbyml/tabby \
  serve --model StarCoder-1B --device cuda --chat-model Qwen2-1.5B-Instruct

This downloads the Tabby Docker image, maps port 8080 for web access, creates a persistent volume for data, and uses StarCoder-1B as the code completion model with GPU acceleration.

For CPU-Only Installation#

Run in terminalbash

# Create a directory for Tabby data
mkdir -p $HOME/.tabby

# Run Tabby on CPU
docker run --entrypoint /opt/tabby/bin/tabby-cpu -it \
  -p 8080:8080 -v $HOME/.tabby:/data \
  tabbyml/tabby \
  serve --model StarCoder-1B --chat-model Qwen2-1.5B-Instruct

The CPU version will be significantly slower but works without an NVIDIA GPU.

Option 2: Using Docker Compose with Traefik Integration#

For a more production-ready setup with proper networking and security:

Run in terminalbash

# Create required directories
mkdir -p ~/docker_data/tabby/data

# Create docker-compose.yml file
cat > ~/docker_data/tabby/docker-compose.yml << 'EOF'
version: '3'

services:
  tabby:
    image: tabbyml/tabby
    command: serve --model StarCoder-1B --device cuda --chat-model Qwen2-1.5B-Instruct
    restart: unless-stopped
    volumes:
      - ./data:/data
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.tabby.rule=Host(`tabby.yourdomain.com`)"
      - "traefik.http.routers.tabby.entrypoints=websecure"
      - "traefik.http.routers.tabby.tls.certresolver=letsencrypt"
      - "traefik.http.services.tabby.loadbalancer.server.port=8080"
    networks:
      - traefik_network

networks:
  traefik_network:
    external: true
EOF

# Start Tabby
cd ~/docker_data/tabby && docker compose up -d

This sets up Tabby with Traefik as a reverse proxy for HTTPS support. Replace tabby.yourdomain.com with your actual domain.

Step 5: Setting Up Traefik for Secure Access#

Traefik provides HTTPS and authentication for secure access:

Run in terminalbash

# Create directories for Traefik
mkdir -p ~/docker_data/traefik/config ~/docker_data/traefik/letsencrypt

# Create Traefik's docker-compose.yml
cat > ~/docker_data/traefik/docker-compose.yml << 'EOF'
version: '3'

services:
  traefik:
    image: traefik:v2.10
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./config:/etc/traefik
      - ./letsencrypt:/letsencrypt
    networks:
      - traefik_network

networks:
  traefik_network:
    name: traefik_network
EOF

# Create Traefik configuration file
cat > ~/docker_data/traefik/config/traefik.yml << 'EOF'
entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: ":443"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: traefik_network

certificatesResolvers:
  letsencrypt:
    acme:
      email: your-email@example.com
      storage: /letsencrypt/acme.json
      httpChallenge:
        entryPoint: web

api:
  dashboard: true
  insecure: false
EOF

# Start Traefik
cd ~/docker_data/traefik && docker compose up -d

Replace your-email@example.com with your actual email for Let's Encrypt notifications.

Step 6: Backup and Monitoring#

Set up regular backups for your Tabby data:

Run in terminalbash

# Create a backup script
cat > ~/backup-tabby.sh << 'EOF'
#!/bin/bash
BACKUP_DIR=~/tabby-backups
mkdir -p $BACKUP_DIR
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
tar -czf $BACKUP_DIR/tabby-data-$TIMESTAMP.tar.gz ~/docker_data/tabby/data
find $BACKUP_DIR -type f -mtime +30 -delete  # Keep 30 days of backups
EOF

chmod +x ~/backup-tabby.sh

# Add to crontab to run daily
(crontab -l 2>/dev/null; echo "0 2 * * * ~/backup-tabby.sh") | crontab -

Create a monitoring script to ensure Tabby stays running:

Run in terminalbash

# Create a monitoring script
cat > ~/monitor-tabby.sh << 'EOF'
#!/bin/bash
if ! docker ps | grep -q tabbyml/tabby; then
  cd ~/docker_data/tabby && docker compose up -d
  echo "Tabby was down, restarting at $(date)" >> ~/tabby-monitor.log
fi
EOF

chmod +x ~/monitor-tabby.sh

# Add to crontab to run every 5 minutes
(crontab -l 2>/dev/null; echo "*/5 * * * * ~/monitor-tabby.sh") | crontab -

Step 7: Using Different Models#

Tabby supports various models with different performance characteristics:

Run in terminalbash

# Example using a different code completion model
docker run -it --gpus all \
  -p 8080:8080 -v $HOME/.tabby:/data \
  tabbyml/tabby \
  serve --model TabbyML/CodeLlama-7B --device cuda

Common models include:

StarCoder-1B (default, balanced performance)
TabbyML/J-350M (smaller, faster on CPU)
TabbyML/CodeLlama-7B (larger, better suggestions but requires more VRAM)

Choose based on your hardware capabilities and performance needs.

Step 8: VS Code Integration#

To use Tabby with VS Code:

Install the Tabby extension from the VS Code marketplace
Configure the server URL:
- Local: http://localhost:8080
- Domain: https://tabby.yourdomain.com (if using Traefik)

Security Considerations for Production Use#

For production deployments, implement these additional security measures:

Run in terminalbash

# 1. Never expose Tabby directly to the internet; always use a reverse proxy like Traefik

# 2. Implement proper firewall rules in WSL2
sudo ufw allow ssh
sudo ufw allow 8080/tcp
sudo ufw enable

# 3. Keep all software updated
sudo apt update && sudo apt upgrade -y

# 4. Use a non-root user in Docker
# Add this to your docker-compose.yml:
user: "1000:1000"

# 5. Implement automated backup solutions as shown earlier

Troubleshooting Common Issues#

NVIDIA GPU Not Detected#

Run in terminalbash

# Verify virtualization is enabled in BIOS
# Verify NVIDIA drivers match in both Windows and WSL2
nvidia-smi

Docker Permission Issues#

Run in terminalbash

# Add your user to the docker group
sudo usermod -aG docker $USER
# Log out and back in for changes to take effect

WSL2 Memory Usage Too High#

Edit or create %UserProfile%\.wslconfig in Windows with:

File contentbash

[wsl2]
memory=8GB
processors=4

Restart WSL2 with wsl --shutdown in PowerShell.

Stopping or Removing Tabby#

If you need to stop or completely remove Tabby:

Run in terminalbash

# For docker run method
docker stop $(docker ps -q --filter ancestor=tabbyml/tabby)
docker rm $(docker ps -aq --filter ancestor=tabbyml/tabby)

# For docker-compose method
cd ~/docker_data/tabby && docker compose down
docker image rm tabbyml/tabby traefik:v2.10

Conclusion#

You now have a comprehensive setup for Tabby AI Coding Assistant on Windows using WSL2 and Docker. This configuration provides:

A self-hosted AI coding assistant that keeps your code private
GPU acceleration for better performance (if available)
Secure access via HTTPS with Traefik
Regular backups and monitoring
Flexibility to use different AI models

Whether you're using Tabby for personal projects or in a team environment, this setup provides a robust foundation that can be customized to your specific needs.

Comprehensive Guide for Installing and Securing Meetily AI Meeting Note-Taker with Docker

Mon, 10 Mar 2025 04:04:38 GMT

Introduction#

Meetily is a powerful, privacy-first AI meeting assistant that captures, transcribes, and summarizes meetings in real-time. Unlike cloud-based alternatives such as Otter.ai ($8.33/month) or Granola.ai ($18/month), Meetily processes your meeting data locally, ensuring complete control over your sensitive information.

Key Benefits of Self-Hosting Meetily#

Complete data privacy - All processing performed locally, ensuring sensitive meeting information never leaves your infrastructure
Cost-effective - Freedom from subscription costs associated with cloud-based alternatives
Fully customizable - Open-source nature allows you to modify or extend functionality as needed
Cross-platform compatibility - Works with Google Meet, Zoom, Microsoft Teams, and other video conferencing platforms

Instruction layout/flow

Prerequisites#

Before you begin, ensure your system meets these requirements:

Hardware Requirements#

CPU: At least 4 cores (more cores recommended for optimal transcription performance)
RAM: Minimum 8GB (16GB+ recommended, especially if using local LLMs)
Storage: At least 10GB of free space (more if storing many meeting recordings)
GPU: Optional but recommended for faster transcription processing

Software Requirements#

Operating System: Ubuntu 20.04 LTS or newer
Permissions: sudo privileges on your system
Internet Connection: Required for downloading Docker images

Network Requirements#

Ports 80 and 443: Required for web access (if exposing Meetily externally)
Port 8000: For Meetily backend API
Port 3000: For Meetily frontend (if not using a reverse proxy)

Docker Installation on Ubuntu 20.04+#

If you already have Docker and Docker Compose installed, you can skip to the next section. Otherwise, follow these steps to install Docker on Ubuntu:

Step 1: Update Package Index and Install Dependencies#

Run in terminalbash

sudo apt update
sudo apt install -y apt-transport-https ca-certificates curl gnupg lsb-release

This updates your system's package database and installs necessary packages required for adding Docker's repository.

Step 2: Add Docker's Official GPG Key#

Run in terminalbash

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

These commands download Docker's official GPG key and set appropriate permissions.

Step 3: Set Up the Docker Repository#

Run in terminalbash

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

This adds the Docker repository to your system's sources list.

Step 4: Install Docker Engine and Docker Compose Plugin#

Run in terminalbash

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

This installs Docker Engine, CLI, containerd runtime, and the Docker Compose plugin.

Step 5: Verify Docker Installation#

Run in terminalbash

sudo docker run hello-world

If successful, you'll see a message confirming that Docker is properly installed.

Step 6: Configure Docker to Run Without Sudo (Optional but Recommended)#

Run in terminalbash

sudo groupadd docker
sudo usermod -aG docker $USER

Log out and log back in for the changes to take effect. This allows you to run Docker commands without prefixing them with sudo.

Installing Meetily with Docker#

Now that Docker is set up, let's install Meetily using Docker Compose for easy deployment and management.

Step 1: Create Directory Structure for Meetily#

Run in terminalbash

mkdir -p ~/docker_data/meetily/{data,config,transcripts}
cd ~/docker_data/meetily

This creates a dedicated directory structure for Meetily with separate folders for different types of data.

Step 2: Create Docker Compose Configuration#

Create a docker-compose.yml file for Meetily:

Run in terminalbash

nano ~/docker_data/meetily/docker-compose.yml

Paste the following content:

File contentyaml

version: '3'

services:
  backend:
    image: zackriya/meetily-backend:latest
    container_name: meetily-backend
    restart: unless-stopped
    volumes:
      - ./data:/app/data
      - ./config:/app/config
      - ./transcripts:/app/transcripts
    ports:
      - "8000:8000"
    environment:
      - NODE_ENV=production
    networks:
      - meetily-network
      - proxy

  frontend:
    image: zackriya/meetily-frontend:latest
    container_name: meetily-frontend
    restart: unless-stopped
    depends_on:
      - backend
    ports:
      - "3000:3000"
    environment:
      - REACT_APP_API_URL=http://localhost:8000
    networks:
      - meetily-network
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.meetily.rule=Host(`meetily.yourdomain.com`)"
      - "traefik.http.routers.meetily.entrypoints=https"
      - "traefik.http.routers.meetily.tls=true"
      - "traefik.http.services.meetily.loadbalancer.server.port=3000"

networks:
  meetily-network:
  proxy:
    external: true

Save and exit the text editor.

Step 3: Set Up Configuration Files#

Create the transcription configuration file:

Run in terminalbash

nano ~/docker_data/meetily/config/transcription.json

Paste the following content:

File contentbash

{
  "model": "base",
  "language": "en",
  "translate": false,
  "cpuThreads": 4,
  "batchSize": 512
}

Save and exit the text editor.

Create the LLM configuration file:

Run in terminalbash

nano ~/docker_data/meetily/config/llm.json

Paste the following content:

File contentbash

{
  "provider": "ollama",
  "model": "llama2",
  "host": "host.docker.internal",
  "port": 11434,
  "timeout": 120
}

Save and exit the text editor.

Step 4: Start Meetily Services#

Run in terminalbash

cd ~/docker_data/meetily
docker compose up -d

This starts the Meetily containers in detached mode (running in the background).

Step 5: Verify Installation#

Run in terminalbash

docker ps | grep meetily

This command should show both meetily-backend and meetily-frontend containers running.

Securing Your Meetily Installation#

Set Up a Secure Backup System#

First, create a backup script:

Run in terminalbash

nano ~/docker_data/meetily/backup.sh

Paste the following:

File contentbash

#!/bin/bash
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
BACKUP_DIR=~/meetily_backups
mkdir -p $BACKUP_DIR
tar -czf $BACKUP_DIR/meetily_backup_$TIMESTAMP.tar.gz -C ~/ docker_data/meetily
find $BACKUP_DIR -name "meetily_backup_*.tar.gz" -mtime +30 -delete

Make the script executable:

Run in terminalbash

chmod +x ~/docker_data/meetily/backup.sh

Set up a daily backup schedule:

Run in terminalbash

(crontab -l 2>/dev/null; echo "0 2 * * * ~/docker_data/meetily/backup.sh") | crontab -

This schedules the backup to run every day at 2 AM.

Configure Proper File Permissions#

Run in terminalbash

chmod -R 750 ~/docker_data/meetily/config
chmod -R 750 ~/docker_data/meetily/data
chmod -R 750 ~/docker_data/meetily/transcripts

This restricts access to configuration and data directories, protecting sensitive information.

Restore from Backup#

If you need to restore from a backup:

Run in terminalbash

# Stop Meetily containers
cd ~/docker_data/meetily
docker compose down

# Restore from backup (replace the timestamp with your actual backup file)
tar -xzf ~/meetily_backups/meetily_backup_YYYYMMDD_HHMMSS.tar.gz -C ~/

# Restart Meetily
cd ~/docker_data/meetily
docker compose up -d

Common Problems and Solutions#

Here are solutions to common issues you might encounter when running Meetily.

Problem: Transcription Not Working#

Diagnostic command:

Run in terminalbash

docker logs meetily-backend | grep "transcription"

Solution:

Check if the transcription model is properly configured:

Run in terminalbash

cat ~/docker_data/meetily/config/transcription.json

Adjust settings for your hardware:

Run in terminalbash

nano ~/docker_data/meetily/config/transcription.json

File contentbash

{
  "model": "tiny",
  "language": "en",
  "translate": false,
  "cpuThreads": 2,
  "batchSize": 256
}

Restart the backend service:

Run in terminalbash

docker compose restart meetily-backend

Problem: Connection to Ollama (LLM) Not Working#

Diagnostic command:

Run in terminalbash

docker logs meetily-backend | grep "llm"

Solution:

Ensure Ollama is running:

Run in terminalbash

docker ps | grep ollama

Update the LLM configuration:

Run in terminalbash

nano ~/docker_data/meetily/config/llm.json

Modify as needed:

File contentbash

{
  "provider": "ollama",
  "model": "llama2",
  "host": "host.docker.internal",
  "port": 11434,
  "timeout": 120
}

Restart the backend service:

Run in terminalbash

docker compose restart meetily-backend

Maintenance Procedures#

Updating Meetily#

To update Meetily to the latest version:

Run in terminalbash

cd ~/docker_data/meetily
docker compose down
docker compose pull
docker compose up -d

Monitoring System Resources#

Set up basic monitoring of your Meetily containers:

Run in terminalbash

docker stats meetily-backend meetily-frontend

Regular Maintenance Tasks#

Clean up old meeting data:

Run in terminalbash

find ~/docker_data/meetily/transcripts -type f -mtime +90 -delete

This removes transcripts older than 90 days.

Check container logs for issues:

Run in terminalbash

# View recent logs
docker logs --tail=100 meetily-backend
docker logs --tail=100 meetily-frontend

# Follow logs in real-time
docker logs -f meetily-backend

Set up automatic weekly container restart (helps prevent memory leaks):

Run in terminalbash

(crontab -l 2>/dev/null; echo "0 3 * * 0 cd ~/docker_data/meetily && docker compose restart") | crontab -

This schedules a restart every Sunday at 3 AM.

Conclusion#

You have successfully set up Meetily, a privacy-focused AI meeting assistant, using Docker on Ubuntu. This configuration provides a secure, self-hosted solution for transcribing and summarizing your meetings while maintaining complete control over your data.

By following the security practices outlined in this guide, you've ensured that your meeting data remains private and protected. The backup procedures will help safeguard against data loss, and the maintenance tasks will keep your system running smoothly.

Remember to periodically check for updates to both Meetily and your Docker environment to benefit from the latest features and security improvements. If you encounter issues not covered in this guide, consult the official Meetily documentation or reach out to the community for support.

Complete Guide to Installing and Securing Tabby AI Coding Assistant on Ubuntu 20.04+

Mon, 10 Mar 2025 03:24:30 GMT

Are you looking for a privacy-focused alternative to GitHub Copilot? Tabby is an open-source, self-hosted AI coding assistant that lets you maintain control over your code while enjoying advanced AI assistance. This comprehensive guide will walk you through installing Tabby on Ubuntu 20.04 or newer using Docker.

What is Tabby AI Coding Assistant?#

Tabby is a powerful self-contained AI coding assistant that helps developers write code more efficiently. Unlike proprietary alternatives, Tabby operates entirely on your local infrastructure, ensuring your code remains private and secure.

Key Features:

Self-contained operation with no external databases or cloud services
OpenAPI interface for integration with existing systems
Support for NVIDIA GPUs for faster performance (also works on CPU)
Compatible with various coding models including StarCoder, Code Llama, and others
Integrates with popular code editors (especially VS Code)
Provides intelligent code completion and answers to coding queries

instruction layout/flow

Prerequisites#

Before we begin, make sure your system meets these requirements:

System Requirements#

Ubuntu 20.04 LTS or newer
Minimum 8GB RAM (16GB+ recommended)
At least 30GB disk space for model data
For GPU acceleration: NVIDIA GPU with 4GB+ VRAM

Update Your System#

First, let's update your Ubuntu system:

Run in terminalbash

sudo apt update && sudo apt upgrade -y

This ensures all packages are current before installation.

Install Basic Dependencies#

Run in terminalbash

sudo apt install -y curl wget git software-properties-common apt-transport-https ca-certificates gnupg lsb-release unzip

These are essential utilities needed for the installation process.

Install Docker#

Docker is required to run Tabby. Here's how to install it:

Run in terminalbash

# Remove any old Docker installations
sudo apt remove docker docker-engine docker.io containerd runc

This removes any conflicting previous Docker installations.

Run in terminalbash

# Add Docker's official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

This adds Docker's GPG key for package verification.

Run in terminalbash

# Add Docker repository
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

This adds the Docker repository to your system.

Run in terminalbash

# Update apt and install Docker
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io

This installs Docker from the official repository.

Run in terminalbash

# Add your user to the docker group
sudo usermod -aG docker $USER

This allows you to run Docker without sudo.

Run in terminalbash

# Apply the new group membership
newgrp docker

This activates the new group membership without requiring logout.

Verify your Docker installation:

Run in terminalbash

docker --version

You should see the Docker version if installation was successful.

Set Up NVIDIA Container Toolkit (For GPU Support)#

If you plan to use GPU acceleration (recommended for better performance):

Run in terminalbash

# Verify your NVIDIA driver is installed
nvidia-smi

This shows your GPU information if drivers are properly installed.

Run in terminalbash

# Install NVIDIA Container Toolkit
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt update
sudo apt install -y nvidia-docker2

This installs the NVIDIA Container Toolkit for Docker.

Run in terminalbash

# Restart Docker service
sudo systemctl restart docker

This applies the new NVIDIA configuration.

Verify the NVIDIA Container Toolkit installation:

Run in terminalbash

sudo docker run --rm --gpus all nvidia/cuda:11.7.1-base-ubuntu20.04 nvidia-smi

This should display your GPU information, confirming Docker can access your GPU.

Installation Options#

Option 1: Using Docker Run (Simplest Approach)#

GPU Installation (NVIDIA)#

For systems with NVIDIA GPUs:

Run in terminalbash

mkdir -p $HOME/.tabby
docker run -it --gpus all \
  -p 8080:8080 -v $HOME/.tabby:/data \
  tabbyml/tabby \
  serve --model StarCoder-1B --device cuda --chat-model Qwen2-1.5B-Instruct

This command:

Creates a directory for Tabby data
Runs Tabby with GPU acceleration
Maps port 8080 for web access
Mounts a local directory for data persistence
Uses StarCoder-1B for code completion and Qwen2-1.5B-Instruct for chat

CPU-Only Installation#

If you don't have a GPU:

Run in terminalbash

mkdir -p $HOME/.tabby
docker run --entrypoint /opt/tabby/bin/tabby-cpu -it \
  -p 8080:8080 -v $HOME/.tabby:/data \
  tabbyml/tabby \
  serve --model StarCoder-1B

This uses the CPU-optimized entry point. Note that performance will be significantly slower than with GPU acceleration.

Option 2: Using Docker Compose (Better for Production)#

Create necessary directories and files:

Run in terminalbash

mkdir -p ~/docker_data/tabby/data

This creates a directory structure for your Tabby deployment.

Access Tabby at http://localhost:8080 or your configured domain.

Verification and Testing#

Verify that Tabby is running correctly:

Run in terminalbash

# Check if the container is running
docker ps | grep tabby

This shows if the Tabby container is active.

Run in terminalbash

# Check the logs for any errors
docker logs tabby

This displays container logs to help identify issues.

IDE Integration#

Tabby works best with code editors. For VS Code:

Open VS Code
Go to Extensions (Ctrl+Shift+X)
Search for "Tabby"
Install the TabbyML extension
Open VS Code settings (Ctrl+,)
Search for "Tabby"
Set "Tabby: Server Url" to your Tabby server (e.g., http://localhost:8080)
Restart VS Code

You should now see Tabby suggestions as you code.

Common Problems and Solutions#

Problem: Docker Container Fails to Start with GPU Support#

Symptoms: Error message containing: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

Solution: Reinstall NVIDIA Container Toolkit:

Run in terminalbash

# Reinstall NVIDIA Container Toolkit
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt update
sudo apt install -y nvidia-docker2

# Restart Docker
sudo systemctl restart docker

This reinstalls and reconfigures the NVIDIA Container Toolkit.

If NVIDIA driver is missing:

Run in terminalbash

sudo apt install nvidia-driver-525  # Use the latest recommended version
sudo reboot

This installs NVIDIA drivers needed for GPU support.

Problem: Model Download Failures#

Symptoms: Container starts but fails during model downloading.

Solution: Check for network issues or insufficient disk space:

Run in terminalbash

# Check available disk space
df -h

This shows available disk space on your system.

Run in terminalbash

# If disk space is low, clear Docker unused resources
docker system prune -a

This removes unused Docker resources to free up space.

Run in terminalbash

# Ensure the data directory has correct permissions
chmod -R 777 $HOME/.tabby  # Temporarily for troubleshooting

This temporarily sets permissive permissions for debugging.

Problem: Slow Performance with CPU-Only Mode#

Symptoms: Tabby is running but code completions are very slow.

Solution: CPU-only mode is significantly slower than GPU mode. Consider:

Use a smaller model for better CPU performance:

Run in terminalbash

docker run --entrypoint /opt/tabby/bin/tabby-cpu -it \
  -p 8080:8080 -v $HOME/.tabby:/data \
  tabbyml/tabby \
  serve --model TabbyML/J-350M  # Smaller model better for CPU

Smaller models run faster on CPU but may provide less accurate suggestions.

Maintenance Procedures#

Updating Tabby#

To update Tabby to the latest version:

Run in terminalbash

# For Docker run method
docker pull tabbyml/tabby
# Then restart your container with the same run command

# For Docker Compose method
cd ~/docker_data/tabby
docker compose pull
docker compose up -d

This pulls the latest Tabby image and restarts your container.

Backing Up Data#

Create a backup script:

Run in terminalbash

cat > ~/backup-tabby.sh << 'EOF'
#!/bin/bash
BACKUP_DIR=~/tabby-backups
mkdir -p $BACKUP_DIR
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
tar -czf $BACKUP_DIR/tabby-data-$TIMESTAMP.tar.gz ~/docker_data/tabby/data
# Keep only the latest 5 backups
ls -t $BACKUP_DIR/tabby-data-*.tar.gz | tail -n +6 | xargs rm -f
EOF

chmod +x ~/backup-tabby.sh

This creates a script that makes compressed backups of your Tabby data.

Add a scheduled backup task:

Run in terminalbash

(crontab -l 2>/dev/null; echo "0 2 * * * ~/backup-tabby.sh") | crontab -

This sets up daily backups at 2:00 AM.

Restoring from Backup#

Run in terminalbash

# Stop the container
cd ~/docker_data/tabby && docker compose down

# Restore from backup
tar -xzf ~/tabby-backups/tabby-data-20250310-120000.tar.gz -C /

# Restart the container
cd ~/docker_data/tabby && docker compose up -d

This restores your Tabby data from a backup archive.

Advanced Configuration#

For advanced users, Tabby offers additional options:

Using Different Models#

Tabby supports various models with different capabilities:

Run in terminalbash

# Example using a different code completion model
docker run -it --gpus all \
  -p 8080:8080 -v $HOME/.tabby:/data \
  tabbyml/tabby \
  serve --model TabbyML/CodeLlama-7B --device cuda

This uses the larger CodeLlama-7B model for potentially better suggestions (requires more VRAM).

Common models include:

StarCoder-1B (default, balanced performance)
TabbyML/J-350M (smaller, faster on CPU)
TabbyML/CodeLlama-7B (larger, better suggestions but requires more VRAM)

Conclusion#

You now have a fully functional Tabby AI Coding Assistant installation on your Ubuntu system! This self-hosted solution gives you complete control over your code and development environment while still providing advanced AI-powered assistance.

For the best experience:

Use GPU acceleration if available
Choose models appropriate for your hardware
Keep your installation updated
Maintain regular backups

Whether you're a solo developer or working in a team, Tabby provides a privacy-focused alternative to cloud-based coding assistants that can significantly boost your productivity.

Installing and Securing Jan: The Offline ChatGPT Alternative on Ubuntu 20.04 with Docker

Mon, 10 Mar 2025 03:02:37 GMT

Introduction#

Jan is a powerful open-source alternative to ChatGPT that runs 100% offline on your computer. This privacy-focused AI assistant ensures your conversations never leave your system, giving you complete control over your data. In this guide, we'll walk through installing, configuring, and securing Jan on Ubuntu 20.04 (and newer) using Docker.

Key Benefits of Jan#

100% Offline Operation: All data and conversations remain on your local device
Multiple AI Engine Support: Compatible with llama.cpp and TensorRT-LLM
OpenAI-Compatible API: Accessible at localhost:1337
Cross-Platform Design: Works on various hardware from personal computers to multi-GPU clusters
Privacy-First Approach: Perfect for handling sensitive information

instruction layout/flow

Prerequisites#

Before we begin, make sure your system meets these requirements:

Hardware Requirements#

At least 2 CPU cores and 2 GB of RAM (more recommended for better performance)
For GPU acceleration (optional but recommended):
- NVIDIA GPU with CUDA Toolkit 11.7 or higher
- NVIDIA driver 470.63.01 or higher

Software Requirements#

Ubuntu 20.04 LTS or newer
glibc 2.27 or higher (check with ldd --version)
gcc 11, g++ 11, cpp 11 or higher
Docker Engine and Docker Compose

Network Requirements#

Local port 1337 available for Jan's API server
Internet access for initial setup and model downloads (after setup, Jan works offline)

Step 1: Install Docker on Ubuntu 20.04#

Docker is required to run Jan in container mode. Let's follow the official method to install Docker Engine.

Remove existing Docker installations#

Run in terminalbash

sudo apt-get remove docker docker-engine docker.io containerd runc

This removes any previous Docker installations that might conflict with the new one.

Set up the Docker repository#

Run in terminalbash

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common

Installs packages needed for apt to use repositories over HTTPS.

Add Docker's official GPG key#

Run in terminalbash

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Adds Docker's GPG key to verify package integrity.

Add the Docker repository#

Run in terminalbash

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

Adds the Docker repository to your package sources. The command automatically detects your Ubuntu version.

Install Docker Engine#

Run in terminalbash

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Updates the package index and installs Docker Engine with related tools.

Verify Docker installation#

Run in terminalbash

sudo docker run hello-world

Verifies Docker is correctly installed by running a test container. You should see a success message if everything works properly.

Step 2: Configure Docker for Enhanced Security (Rootless Mode)#

For improved security, let's run Docker in rootless mode, which reduces potential risks by running the Docker daemon and containers as a non-root user.

Install rootless prerequisites#

Run in terminalbash

sudo apt-get install -y uidmap dbus-user-session

Installs packages needed for rootless Docker operation.

Set up rootless Docker#

Run in terminalbash

sudo systemctl disable --now docker.service docker.socket
sudo apt-get install -y docker-ce-rootless-extras

Disables the system-wide Docker service and installs rootless extras.

Run the rootless setup tool#

Run in terminalbash

dockerd-rootless-setuptool.sh install

Configures Docker to run in rootless mode.

Add environment variables to your shell profile#

Run in terminalbash

echo 'export PATH=/usr/bin:$PATH' >> ~/.bashrc
echo 'export DOCKER_HOST=unix:///run/user/1000/docker.sock' >> ~/.bashrc
source ~/.bashrc

Sets required environment variables for rootless Docker. Note: Replace 1000 with your actual user ID if different.

Enable Docker service at startup#

Run in terminalbash

systemctl --user enable docker
sudo loginctl enable-linger $(whoami)

Configures Docker to start automatically with your system.

Step 3: Create a Docker Compose Configuration for Jan#

Now that Docker is installed and configured securely, let's set up Jan using Docker Compose.

Create a directory structure for Jan#

Run in terminalbash

mkdir -p ~/docker_data/jan/data ~/docker_data/jan/config

Creates directories for Jan's data and configuration files.

Create a Docker Compose file#

Run in terminalbash

cat > ~/docker_data/jan/docker-compose.yml << 'EOF'
version: '3'

services:
  jan:
    container_name: jan
    image: janhq/jan:latest
    restart: unless-stopped
    ports:
      - "1337:1337"
    volumes:
      - ./data:/data
      - ./config:/root/.jan
    environment:
      - JAN_HOST=0.0.0.0
      - JAN_PORT=1337
    # Uncomment the following lines for GPU support
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]
EOF

Creates a Docker Compose configuration file for Jan.

Launch Jan#

Run in terminalbash

cd ~/docker_data/jan
docker compose up -d

Starts the Jan container in detached mode.

Access Jan API#

Run in terminalbash

curl http://localhost:1337/v1/models

Verifies that Jan's API is accessible. You should see a response listing available models.

Step 4: Set Up Automatic Backups#

Let's create a backup script to ensure your data is safe.

Create a backup script#

Run in terminalbash

cat > ~/docker_data/jan/backup-jan.sh << 'EOF'
#!/bin/bash

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR=~/jan_backups

mkdir -p $BACKUP_DIR
cd ~/docker_data/jan

# Ensure Jan is running
docker compose ps | grep jan || docker compose up -d

# Create backup archive
tar -czf $BACKUP_DIR/jan_backup_$TIMESTAMP.tar.gz data config

# Remove backups older than 30 days
find $BACKUP_DIR -name "jan_backup_*.tar.gz" -type f -mtime +30 -delete

echo "Backup completed: $BACKUP_DIR/jan_backup_$TIMESTAMP.tar.gz"
EOF

chmod +x ~/docker_data/jan/backup-jan.sh

Creates a backup script that archives Jan's data and configuration files.

Schedule automatic backups#

Run in terminalbash

(crontab -l 2>/dev/null; echo "0 2 * * * ~/docker_data/jan/backup-jan.sh") | crontab -

Schedules a backup every day at 2 AM.

Common Problems and Troubleshooting#

Here are solutions for issues you might encounter:

Issue: Docker fails to start#

Diagnostic command:

Run in terminalbash

systemctl --user status docker

Solution:

Run in terminalbash

systemctl --user restart docker
sudo loginctl enable-linger $(whoami)

Restarts the Docker service and ensures it's configured to start automatically.

Issue: Jan API is unreachable#

Diagnostic command:

Run in terminalbash

curl -v http://localhost:1337/v1/models

Solution:

Run in terminalbash

cd ~/docker_data/jan && docker compose down
cd ~/docker_data/jan && docker compose up -d

Restarts the Jan container to resolve connectivity issues.

Issue: GPU acceleration not working#

Diagnostic command:

Run in terminalbash

docker exec -it jan nvidia-smi

Solution: If you get an error, ensure NVIDIA Docker support is installed:

Run in terminalbash

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
systemctl --user restart docker

Installs the NVIDIA Container Toolkit and restarts Docker to enable GPU support.

Issue: Model download failures#

Solution:

Run in terminalbash

docker exec -it jan /bin/bash -c "mkdir -p /data/models && chmod 777 /data/models"

Ensures the models directory exists and has proper permissions.

Maintenance Procedures#

Updating Jan#

To update Jan to the latest version:

Run in terminalbash

cd ~/docker_data/jan
docker compose pull
docker compose down
docker compose up -d

Pulls the latest Jan image, stops the current container, and starts a new one with the updated image.

Monitoring Jan#

Set up basic monitoring for Jan:

Run in terminalbash

cat > ~/docker_data/jan/monitor-jan.sh << 'EOF'
#!/bin/bash

API_URL="http://localhost:1337/v1/models"
LOG_FILE=~/jan_monitor.log

echo "$(date) - Checking Jan service..." >> $LOG_FILE

# Check if API is responding
if curl -s -f $API_URL > /dev/null; then
    echo "$(date) - Jan is running correctly." >> $LOG_FILE
else
    echo "$(date) - ERROR: Jan is not responding. Attempting restart..." >> $LOG_FILE
    cd ~/docker_data/jan
    docker compose restart
    
    # Check if restart fixed the issue
    if curl -s -f $API_URL > /dev/null; then
        echo "$(date) - Jan successfully restarted." >> $LOG_FILE
    else
        echo "$(date) - CRITICAL: Jan failed to restart. Manual intervention required." >> $LOG_FILE
        # Optionally send an email alert here
    fi
fi
EOF

chmod +x ~/docker_data/jan/monitor-jan.sh
(crontab -l 2>/dev/null; echo "*/10 * * * * ~/docker_data/jan/monitor-jan.sh") | crontab -

Creates a monitoring script that checks Jan's status every 10 minutes and attempts to restart it if issues are detected.

Conclusion#

By following this guide, you've successfully installed, secured, and configured Jan on Ubuntu 20.04 using Docker. You now have access to a powerful AI assistant that runs completely offline, respecting your privacy and giving you full control over your data.

The Docker-based deployment provides a clean, isolated environment for Jan, while security measures like rootless Docker help protect your system. The automatic backup and monitoring solutions ensure your setup remains reliable over time.

Remember that Jan is actively developed, with new features and improvements being added regularly. Keep an eye on the official documentation and GitHub repository for updates and additional configuration options.

Comprehensive Guide to Hosting and Securing Jellyfin on Ubuntu 20.04+ with Docker

Sun, 09 Mar 2025 23:29:12 GMT

Introduction:#

Jellyfin is a free and open-source media server that allows you to manage and stream your media collection to any device. Unlike proprietary alternatives like Plex or Emby, Jellyfin is completely free, has no premium features locked behind paywalls, and doesn't collect user data. This guide will walk you through every step of setting up a secure, well-configured Jellyfin instance on Ubuntu 20.04 or newer using Docker.

Why Use Docker for Jellyfin?#

Docker deployment offers several key advantages:

Simplified installation process
Consistent environments across different systems
Easy updates without dependency conflicts
Enhanced security through container isolation
Simplified backup and restore procedures

Instruction layout/flow

Prerequisites#

Before proceeding with installation, ensure your system meets these requirements:

System Requirements#

Ubuntu 20.04 LTS or newer
2GB RAM minimum (4GB recommended for transcoding)
Dual-core CPU minimum (quad-core recommended for transcoding)
Sufficient storage for your media library
Network connectivity (port forwarding required for remote access)

You can check your Ubuntu version with:

Run in terminalbash

lsb_release -a

Software Requirements#

Docker Engine
Docker Compose
Git (optional, for cloning configuration files)

Required Permissions#

You'll need:

Sudo access for installation
User added to docker group for non-root Docker access

Check your current user ID and group ID (needed for Docker configuration):

Run in terminalbash

#command is: id
id

Installing Docker and Docker Compose#

Follow these steps to install Docker and Docker Compose on your Ubuntu system:

Step 1: Update System Packages#

Run in terminalbash

sudo apt update && sudo apt upgrade -y

This ensures your system has the latest security updates before proceeding.

Step 2: Install Required Dependencies#

Run in terminalbash

sudo apt install apt-transport-https ca-certificates curl software-properties-common gnupg -y

These packages are necessary for adding Docker's repository and securely downloading packages.

Step 3: Add Docker's Official GPG Key#

Run in terminal

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

This adds Docker's official GPG key to verify package integrity.

Step 4: Add Docker Repository#

Run in terminalbash

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

This adds Docker's official repository to your system's package sources.

Step 5: Update Package Database#

Run in terminalbash

sudo apt update

This refreshes your package database to include Docker packages.

Step 6: Install Docker Engine#

Run in terminalbash

sudo apt install docker-ce docker-ce-cli containerd.io -y

This installs Docker Engine, CLI, and containerd.

Step 7: Enable Docker Service#

Run in terminalbash

sudo systemctl enable --now docker

This starts Docker and configures it to launch at system boot.

Step 8: Add User to Docker Group#

Run in terminalbash

sudo usermod -aG docker $USER

This allows running Docker commands without sudo. Important: You'll need to log out and back in for changes to take effect.

Step 9: Install Docker Compose#

Run in terminalbash

sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

This downloads the latest Docker Compose binary and makes it executable.

Step 10: Verify Installation#

Run in terminalbash

docker --version
docker-compose --version

This confirms that both Docker and Docker Compose are correctly installed.

Setting Up Jellyfin with Docker Compose#

Now that Docker is installed, we'll set up Jellyfin using Docker Compose for easier management of container configurations.

Step 1: Create Directory Structure#

Run in terminalbash

mkdir -p ~/docker_data/jellyfin/{config,cache,media}

This creates directories for Jellyfin's configuration, cache, and media files.

Step 2: Create Docker Compose Configuration#

Create a new file at ~/docker_data/jellyfin/docker-compose.yml with the following content:

Run in terminalbash

cat > ~/docker_data/jellyfin/docker-compose.yml << 'EOF'
version: '3'

services:
  jellyfin:
    image: jellyfin/jellyfin:latest
    container_name: jellyfin
    user: 1000:1000  # Replace with your user ID and group ID from 'id' command
    restart: unless-stopped
    volumes:
      - ./config:/config
      - ./cache:/cache
      - ./media:/media
      # Optional: mount additional media locations
      # - /path/to/media:/media2:ro
    ports:
      - 8096:8096  # HTTP port (remove for Traefik-only access)
    environment:
      - TZ=Etc/UTC  # Set to your timezone
    networks:
      - proxy  # For Traefik integration
    #devices:  # Uncomment for hardware acceleration
    #  - /dev/dri:/dev/dri  # Intel GPU
    #  - /dev/video10:/dev/video10  # VAAPI rendering node
    #  - /dev/video11:/dev/video11  # VAAPI DRM node
    #  - /dev/nvidia0:/dev/nvidia0  # Nvidia GPU
    #  - /dev/nvidiactl:/dev/nvidiactl
    #  - /dev/nvidia-modeset:/dev/nvidia-modeset
    #  - /dev/nvidia-uvm:/dev/nvidia-uvm
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.jellyfin.rule=Host(`jellyfin.your-domain.com`)"
      - "traefik.http.routers.jellyfin.entrypoints=websecure"
      - "traefik.http.routers.jellyfin.tls.certresolver=letsencrypt"
      - "traefik.http.services.jellyfin.loadbalancer.server.port=8096"

networks:
  proxy:
    external: true
EOF

Important: Replace 1000:1000 with your actual user ID and group ID from the id command, and jellyfin.your-domain.com with your actual domain.

Setting Up Traefik as a Reverse Proxy (Optional but Recommended)#

Traefik provides secure access to Jellyfin from the internet with automatic SSL certificate management.

Step 1: Create Traefik Directory Structure#

Run in terminalbash

mkdir -p ~/docker_data/traefik/{config,letsencrypt}
touch ~/docker_data/traefik/acme.json
chmod 600 ~/docker_data/traefik/acme.json

This creates directories for Traefik configuration and Let's Encrypt certificates.

Step 2: Create Traefik Configuration Files#

Create the main configuration file:

Run in terminalbash

cat > ~/docker_data/traefik/config/traefik.toml << 'EOF'
[entryPoints]
  [entryPoints.web]
    address = ":80"
    [entryPoints.web.http.redirections.entryPoint]
      to = "websecure"
      scheme = "https"
  
  [entryPoints.websecure]
    address = ":443"
    [entryPoints.websecure.http.tls]
      certResolver = "letsencrypt"
    [entryPoints.websecure.http.middlewares]
      middlewares = ["securityHeaders@file"]

[providers]
  [providers.docker]
    endpoint = "unix:///var/run/docker.sock"
    exposedByDefault = false
    network = "proxy"
  
  [providers.file]
    filename = "/etc/traefik/dynamic.toml"

[certificatesResolvers]
  [certificatesResolvers.letsencrypt]
    [certificatesResolvers.letsencrypt.acme]
      email = "your-email@example.com"  # Replace with your email
      storage = "/letsencrypt/acme.json"
      [certificatesResolvers.letsencrypt.acme.tlsChallenge]

[api]
  dashboard = true

[log]
  level = "INFO"
EOF

Create dynamic configuration file:

Run in terminalbash

cat > ~/docker_data/traefik/config/dynamic.toml << 'EOF'
[http.middlewares]
  [http.middlewares.securityHeaders.headers]
    browserXssFilter = true
    contentTypeNosniff = true
    frameDeny = true
    sslRedirect = true
    [http.middlewares.securityHeaders.headers.customResponseHeaders]
      X-Robots-Tag = "noindex,nofollow,nosnippet,noarchive,notranslate,noimageindex"
      Server = ""
    [http.middlewares.securityHeaders.headers.referrerPolicy]
      policy = "strict-origin-when-cross-origin"
    [http.middlewares.securityHeaders.headers.contentSecurityPolicy]
      policy = "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data: blob:; font-src 'self' data:; media-src 'self' blob:; connect-src 'self' wss:; frame-src 'self'; base-uri 'self'; form-action 'self'"

[http.routers]
  [http.routers.dashboard]
    rule = "Host(`traefik.your-domain.com`)"
    service = "api@internal"
    entryPoints = ["websecure"]
    middlewares = ["securityHeaders", "traefik-auth"]
    [http.routers.dashboard.tls]
      certResolver = "letsencrypt"

[http.middlewares.traefik-auth.basicAuth]
  # Generate with: echo $(htpasswd -nb username password) | sed -e s/\\$/\\$\\$/g
  users = ["admin:$apr1$randomhash"]  # Replace with your own credentials
EOF

Important: Replace your-email@example.com, traefik.your-domain.com, and the basic auth credentials with your own.

Step 3: Create Docker Compose for Traefik#

Run in terminalbash

cat > ~/docker_data/traefik/docker-compose.yml << 'EOF'
version: '3'

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    ports:
      - 80:80
      - 443:443
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./config/traefik.toml:/etc/traefik/traefik.toml:ro
      - ./config/dynamic.toml:/etc/traefik/dynamic.toml:ro
      - ./letsencrypt:/letsencrypt
    networks:
      - proxy
    labels:
      - "traefik.enable=true"

networks:
  proxy:
    external: true
    name: proxy
EOF

Step 4: Create Docker Network#

Run in terminalbash

docker network create proxy

This creates the network that both Traefik and Jellyfin will use.

Step 5: Start Traefik#

Run in terminalbash

cd ~/docker_data/traefik && docker-compose up -d

This starts Traefik in the background.

Step 6: Start Jellyfin#

Run in terminalbash

cd ~/docker_data/jellyfin && docker-compose up -d

This starts Jellyfin in the background.

Setting Up Fail2ban for Additional Security (Optional)#

Fail2ban can protect against brute force attacks by temporarily banning IP addresses that show malicious behavior.

Step 1: Install Fail2ban#

Run in terminal

sudo apt install fail2ban -y

Step 2: Configure Fail2ban for Traefik#

Run in terminalbash

sudo tee /etc/fail2ban/filter.d/traefik-auth.conf > /dev/null << 'EOF'
[Definition]
failregex = ^.*\s"(GET|POST|HEAD).*401\s.*remote_ip=<HOST>\s.*$
ignoreregex =
EOF

Step 3: Create Jail Configuration#

Run in terminalbash

sudo tee /etc/fail2ban/jail.d/traefik.conf > /dev/null << 'EOF'
[traefik-auth]
enabled = true
port = http,https
filter = traefik-auth
logpath = /var/log/traefik/access.log
maxretry = 3
bantime = 3600
EOF

Step 4: Restart Fail2ban#

Run in terminalbash

sudo systemctl restart fail2ban

Backup and Restore Procedures#

Regular backups are essential for data safety. Here's how to set up automated backups for Jellyfin.

Backup Script#

Create a backup script:

Run in terminalbash

cat > ~/backup-jellyfin.sh << 'EOF'
#!/bin/bash
# Jellyfin backup script

# Set backup directory
BACKUP_DIR="/path/to/backups/jellyfin"
mkdir -p "$BACKUP_DIR"

# Set date format for backup files
DATE=$(date +"%Y-%m-%d")

# Backup Jellyfin configuration
tar -czf "$BACKUP_DIR/jellyfin-config-$DATE.tar.gz" -C ~/docker_data jellyfin/config

# Keep only the last 5 backups
ls -t "$BACKUP_DIR"/*.tar.gz | tail -n +6 | xargs rm -f

echo "Backup completed: $BACKUP_DIR/jellyfin-config-$DATE.tar.gz"
EOF

chmod +x ~/backup-jellyfin.sh

Set Up Automated Weekly Backups#

Run in terminalbash

(crontab -l 2>/dev/null; echo "0 2 * * 0 ~/backup-jellyfin.sh") | crontab -

This schedules the backup script to run every Sunday at 2 AM.

Restore Procedure#

If you need to restore from a backup:

Run in terminalbash

# Stop containers
cd ~/docker_data/jellyfin && docker-compose down

# Restore from backup
tar -xzf /path/to/backups/jellyfin/jellyfin-config-YYYY-MM-DD.tar.gz -C ~/docker_data/jellyfin

# Start containers
cd ~/docker_data/jellyfin && docker-compose up -d

Common Problems and Solutions#

Here are some common issues you might encounter and their solutions.

Transcoding Issues#

Problem: Video transcoding is slow or failing.

Diagnosis:

Run in terminalbash

docker logs jellyfin | grep -i transcode

Solution: Enable hardware acceleration if available.

For Intel GPUs:

Run in terminalbash

# Install required packages on host
sudo apt install intel-media-va-driver-non-free -y

# Update docker-compose.yml
# Uncomment the devices section for /dev/dri

Restart Jellyfin:

Run in terminalbash

cd ~/docker_data/jellyfin && docker-compose down && docker-compose up -d

Connection Problems#

Problem: Cannot access Jellyfin through domain.

Diagnosis:

Run in terminalbash

# Check if Traefik is running
docker ps | grep traefik

# Check Traefik logs
docker logs traefik

Solution:

Run in terminalbash

# Verify DNS resolution
nslookup jellyfin.your-domain.com

# Check if ports are forwarded correctly
curl -I http://your-domain.com

Docker Container Fails to Start#

Problem: Jellyfin container won't start.

Diagnosis:

Run in terminalbash

docker logs jellyfin

Solution:

Run in terminalbash

# Check permissions
sudo chown -R $(id -u):$(id -g) ~/docker_data/jellyfin

# Verify ports are not in use
sudo netstat -tuln | grep 8096

Maintenance Procedures#

Regular maintenance ensures your Jellyfin server remains secure and performs optimally.

Updating Jellyfin#

Run in terminalbash

cd ~/docker_data/jellyfin
docker-compose pull
docker-compose down
docker-compose up -d

This pulls the latest image, stops the current container, and starts a new one with the updated image.

Updating Traefik#

Run in terminalbash

cd ~/docker_data/traefik
docker-compose pull
docker-compose down
docker-compose up -d

System Updates#

Run in terminalbash

sudo apt update && sudo apt upgrade -y

Periodic system updates are essential for security.

Monitoring Disk Space#

Run in terminalbash

df -h

Regularly check disk space to ensure sufficient storage for media and container operations.

Conclusion#

You now have a fully functional, secure Jellyfin media server running on Ubuntu 20.04 using Docker. The configuration provides:

Isolated container environment for enhanced security
Automatic SSL certificate management with Let's Encrypt
Reverse proxy with Traefik for secure internet access
Hardened TLS configuration to protect against common vulnerabilities
Backup and restore procedures to prevent data loss
Maintenance instructions for keeping the system updated and secure

By following this guide, you've implemented industry best practices for securing a self-hosted media server while maintaining convenient access for authorized users. Remember to regularly update all components and monitor for security advisories related to Docker, Ubuntu, Traefik, and Jellyfin to maintain the security of your system.

Comprehensive Guide to Self-Hosting n8n on Ubuntu 20.04+ with Docker

Sun, 09 Mar 2025 19:21:59 GMT

Introduction#

n8n is a powerful, free, and open-source workflow automation tool that connects various services and applications. Docker provides an optimal deployment method by creating a consistent environment for running n8n, effectively isolating the application from the host system. This isolation ensures compatibility across different operating systems and simplifies migration when changing hosts or environments.

This guide will walk you through every step of installing and configuring n8n using Docker on Ubuntu 20.04 or newer, addressing common issues and providing best practices for security and maintenance.

instruction layout/chart

Prerequisites#

Before beginning the installation process, ensure your system meets these requirements:

Ubuntu 20.04 or newer operating system
Minimum 1GB RAM (2GB+ recommended for production environments)
At least 10GB of available storage space
Dual-core processor (more cores beneficial for complex workflows)
Static IP address or domain name pointing to your server
Root or sudo access to execute administrative commands
Port 5678 available (n8n's default port)
Basic knowledge of Linux commands, Docker concepts, and networking principles

If running n8n behind a firewall or router, ensure port 5678 is properly forwarded to your server. For custom domain setup with SSL, configure DNS records to point to your server's IP address.

Docker Installation on Ubuntu 20.04+#

Step 1: Update Package Index#

Run in terminalbash

sudo apt update && sudo apt upgrade -y

This updates the package lists and upgrades existing packages to their latest versions.

Step 2: Install Docker Dependencies#

Run in terminalbash

sudo apt install -y apt-transport-https ca-certificates curl software-properties-common

These packages allow apt to use repositories over HTTPS and are prerequisites for secure Docker installation.

Step 3: Add Docker's Official GPG Key#

Run in terminalbash

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

This adds Docker's GPG key to verify package authenticity during installation.

Step 4: Set Up the Docker Repository#

Run in terminalbash

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

This adds the appropriate Docker repository for your Ubuntu version.

Step 5: Install Docker Engine#

Run in terminalbash

sudo apt update

Updates the package database with Docker packages from the newly added repository.

Run in terminalbash

sudo apt install -y docker-ce docker-ce-cli containerd.io

Installs Docker Engine, CLI, and containerd, the core components needed to run Docker containers.

Step 6: Verify Docker Installation#

Run in terminalbash

sudo docker run hello-world

This downloads a test image and runs it in a container. If Docker is properly installed, you'll see a confirmation message.

Step 7: Enable Non-root User Access to Docker (Optional)#

Run in terminalbash

sudo usermod -aG docker $USER

Adds your user to the Docker group, allowing Docker commands without sudo.

Run in terminalbash

newgrp docker

Applies the new group membership without logging out.

Installing n8n with Docker#

Single Container Setup with Docker#

This is the simplest way to get started with n8n using Docker, using SQLite as the default database.

Step 1: Create a Docker Volume for Data Persistence#

Run in terminalbash

docker volume create n8n_data

Creates a Docker volume that will store all n8n data, including workflows, credentials, and execution history.

Step 2: Run n8n Container#

Run in terminalbash

docker run -d --restart unless-stopped \
  --name n8n \
  -p 5678:5678 \
  -v n8n_data:/home/node/.n8n \
  n8nio/n8n

This command:

Runs n8n in detached mode (-d)
Configures automatic restart (--restart unless-stopped)
Names the container for easy reference (--name n8n)
Maps port 5678 on the host to port 5678 in the container
Mounts the Docker volume to n8n's data directory

Docker Compose Setup#

Docker Compose allows you to define and run multi-container Docker applications, useful when running n8n alongside other services like a database.

Step 1: Create Directory Structure#

Run in terminalbash

mkdir -p ~/docker_data/n8n/data

Creates directories for storing Docker Compose files and n8n data.

Step 2: Create Docker Compose Configuration#

Create a docker-compose.yml file with SQLite (default database):

Run in terminalbash

cat > ~/docker_data/n8n/docker-compose.yml << 'EOF'
version: '3'

services:
  n8n:
    image: n8nio/n8n
    restart: unless-stopped
    ports:
      - "5678:5678"
    volumes:
      - ./data:/home/node/.n8n
    environment:
      - N8N_ENCRYPTION_KEY=your-random-encryption-key
      - N8N_HOST=n8n.yourdomain.com
      - N8N_PROTOCOL=https
      - N8N_PORT=5678
      - NODE_ENV=production
    networks:
      - n8n-network

networks:
  n8n-network:
EOF

This creates a basic Docker Compose configuration for n8n with SQLite as the database.

Step 3: Start n8n with Docker Compose#

Run in terminalbash

cd ~/docker_data/n8n
docker compose up -d

These commands navigate to the directory containing the docker-compose.yml file and start n8n in detached mode.

Advanced Setup with PostgreSQL Database#

For production environments, using PostgreSQL is recommended for better performance and reliability.

Step 1: Create a More Advanced Docker Compose Configuration#

Run in terminalbash

cat > ~/docker_data/n8n/docker-compose.yml << 'EOF'
version: '3'

services:
  n8n:
    image: n8nio/n8n
    restart: unless-stopped
    ports:
      - "5678:5678"
    volumes:
      - ./data:/home/node/.n8n
    environment:
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=n8n_password
      - N8N_ENCRYPTION_KEY=your-random-encryption-key
      - N8N_HOST=n8n.yourdomain.com
      - N8N_PROTOCOL=https
      - N8N_PORT=5678
      - NODE_ENV=production
    networks:
      - n8n-network
    depends_on:
      - postgres

  postgres:
    image: postgres:13
    restart: unless-stopped
    volumes:
      - postgres_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=n8n
      - POSTGRES_USER=n8n
      - POSTGRES_PASSWORD=n8n_password
    networks:
      - n8n-network

volumes:
  postgres_data:

networks:
  n8n-network:
EOF

This configuration sets up n8n with PostgreSQL as the database, creating a more robust and scalable solution.

Step 2: Start n8n with PostgreSQL#

Run in terminalbash

cd ~/docker_data/n8n
docker compose up -d

These commands start both n8n and PostgreSQL containers.

Securing n8n with SSL/TLS#

Method 1: Using Nginx as a Reverse Proxy with Let's Encrypt#

Step 1: Install Nginx and Certbot#

Run in terminalbash

sudo apt update
sudo apt install -y nginx certbot python3-certbot-nginx

Installs Nginx web server and Certbot for obtaining Let's Encrypt SSL certificates.

Step 2: Obtain SSL Certificate#

Run in terminalbash

sudo certbot --nginx -d n8n.yourdomain.com

Obtains and installs an SSL certificate for your domain. Follow the prompts to complete the certificate issuance.

Step 3: Configure Nginx as a Reverse Proxy#

Run in terminalbash

sudo cat > /etc/nginx/sites-available/n8n << 'EOF'
server {
    server_name n8n.yourdomain.com;

    location / {
        proxy_pass http://localhost:5678;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        # WebSocket support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
    }

    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/n8n.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/n8n.yourdomain.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
}

server {
    if ($host = n8n.yourdomain.com) {
        return 301 https://$host$request_uri;
    }
    listen 80;
    server_name n8n.yourdomain.com;
    return 404;
}
EOF

This creates an Nginx configuration that acts as a reverse proxy for n8n and includes WebSocket support.

Step 4: Enable the Configuration and Restart Nginx#

Run in terminalbash

sudo ln -s /etc/nginx/sites-available/n8n /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl restart nginx

These commands enable the Nginx configuration, test it for syntax errors, and restart Nginx to apply the changes.

Method 2: n8n with Traefik Integration#

Traefik is a modern HTTP reverse proxy and load balancer that can automatically manage SSL certificates.

Step 1: Create Docker Network for Traefik#

Run in terminalbash

docker network create proxy

Creates a Docker network for communication between Traefik and n8n.

Step 2: Create Directory Structure#

Run in terminalbash

mkdir -p ~/docker_data/traefik/data
mkdir -p ~/docker_data/traefik/config
mkdir -p ~/docker_data/n8n/data

Creates directories for Traefik and n8n data and configuration files.

Step 3: Create Traefik Configuration#

Run in terminalbash

cat > ~/docker_data/traefik/config/traefik.yml << 'EOF'
api:
  dashboard: true
  debug: true

entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: proxy

certificatesResolvers:
  letsencrypt:
    acme:
      email: your-email@example.com
      storage: /data/acme.json
      httpChallenge:
        entryPoint: http
EOF

This creates a Traefik configuration file with support for automatic SSL certificate generation.

Step 4: Create Traefik Docker Compose File#

Run in terminalbash

cat > ~/docker_data/traefik/docker-compose.yml << 'EOF'
version: '3'

services:
  traefik:
    image: traefik:v2.5
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./config/traefik.yml:/traefik.yml:ro
      - ./data:/data
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=https"
      - "traefik.http.routers.traefik.rule=Host(`traefik.yourdomain.com`)"
      - "traefik.http.routers.traefik.tls.certresolver=letsencrypt"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.middlewares=auth"
      - "traefik.http.middlewares.auth.basicauth.users=admin:$$apr1$$xyz123$$aBcDeFgHiJkLmNoPqRsTuV/" # Change this

networks:
  proxy:
    external: true
EOF

This creates a Docker Compose file for Traefik with automatic SSL certificate management.

Step 5: Create n8n Docker Compose File for Traefik#

Run in terminalbash

cat > ~/docker_data/n8n/docker-compose.yml << 'EOF'
version: '3'

services:
  n8n:
    image: n8nio/n8n
    restart: unless-stopped
    volumes:
      - ./data:/home/node/.n8n
    environment:
      - N8N_ENCRYPTION_KEY=your-random-encryption-key
      - N8N_HOST=n8n.yourdomain.com
      - N8N_PROTOCOL=https
      - N8N_PORT=443
      - NODE_ENV=production
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.n8n.entrypoints=https"
      - "traefik.http.routers.n8n.rule=Host(`n8n.yourdomain.com`)"
      - "traefik.http.routers.n8n.tls.certresolver=letsencrypt"
      - "traefik.http.services.n8n.loadbalancer.server.port=5678"

networks:
  proxy:
    external: true
EOF

This creates a Docker Compose file for n8n that integrates with Traefik for automatic SSL certificate management.

Step 6: Start Traefik and n8n#

Run in terminalbash

cd ~/docker_data/traefik
docker compose up -d

cd ~/docker_data/n8n
docker compose up -d

These commands start both Traefik and n8n containers.

Advanced Security Configuration#

Securing n8n with Authentication#

Enable basic authentication to protect your n8n instance:

Run in terminalbash

cat > ~/docker_data/n8n/docker-compose.yml << 'EOF'
version: '3'

services:
  n8n:
    image: n8nio/n8n
    restart: unless-stopped
    ports:
      - "5678:5678"
    volumes:
      - ./data:/home/node/.n8n
    environment:
      - N8N_ENCRYPTION_KEY=your-random-encryption-key
      - N8N_HOST=n8n.yourdomain.com
      - N8N_PROTOCOL=https
      - N8N_PORT=5678
      - NODE_ENV=production
      - N8N_BASIC_AUTH_ACTIVE=true
      - N8N_BASIC_AUTH_USER=admin
      - N8N_BASIC_AUTH_PASSWORD=secure_password
    networks:
      - n8n-network

networks:
  n8n-network:
EOF

This configuration enables basic authentication for n8n, requiring a username and password to access the interface.

Implementing IP Restrictions#

To restrict access to n8n from specific IP addresses, update your Nginx configuration:

Run in terminalbash

sudo cat > /etc/nginx/sites-available/n8n << 'EOF'
server {
    server_name n8n.yourdomain.com;

    # IP restrictions
    allow 192.168.1.0/24;  # Allow your local network
    allow 203.0.113.0/24;  # Allow your office network
    deny all;             # Deny all other IPs

    location / {
        proxy_pass http://localhost:5678;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        # WebSocket support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
    }

    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/n8n.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/n8n.yourdomain.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
}
EOF

This configuration restricts access to n8n to specific IP ranges.

Setting Resource Limits#

To prevent resource exhaustion, set Docker resource limits:

Run in terminalbash

cat > ~/docker_data/n8n/docker-compose.yml << 'EOF'
version: '3'

services:
  n8n:
    image: n8nio/n8n
    restart: unless-stopped
    ports:
      - "5678:5678"
    volumes:
      - ./data:/home/node/.n8n
    environment:
      - N8N_ENCRYPTION_KEY=your-random-encryption-key
      - N8N_HOST=n8n.yourdomain.com
      - N8N_PROTOCOL=https
      - N8N_PORT=5678
      - NODE_ENV=production
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 1G
        reservations:
          cpus: '0.5'
          memory: 512M
    networks:
      - n8n-network

networks:
  n8n-network:
EOF

This configuration limits the CPU and memory usage of the n8n container.

Data Persistence and Backup#

Creating a Backup Script#

Run in terminalbash

cat > ~/backup-n8n.sh << 'EOF'
#!/bin/bash

TIMESTAMP=$(date +"%Y%m%d%H%M%S")
BACKUP_DIR=~/backups
BACKUP_FILE="$BACKUP_DIR/n8n_backup_$TIMESTAMP.tar.gz"

# Create backup directory if it doesn't exist
mkdir -p $BACKUP_DIR

# Backup n8n data
tar -czvf $BACKUP_FILE -C ~/docker_data/n8n data

# Keep only the last 7 backups
ls -t $BACKUP_DIR/n8n_backup_*.tar.gz | tail -n +8 | xargs -r rm

echo "Backup completed: $BACKUP_FILE"
EOF

Make the script executable:

Run in terminalbash

chmod +x ~/backup-n8n.sh

This script creates a compressed backup of n8n data and retains the most recent 7 backups.

Scheduling Automated Backups#

Run in terminalbash

(crontab -l 2>/dev/null; echo "0 2 * * * /home/$USER/backup-n8n.sh") | crontab -

This command adds a cron job that runs the backup script every day at 2:00 AM.

Restore Procedures#

To restore n8n data from a backup, use this restore script:

Run in terminalbash

cat > ~/restore-n8n.sh << 'EOF'
#!/bin/bash

if [ -z "$1" ]; then
  echo "Usage: $0 <backup_file>"
  exit 1
fi

BACKUP_FILE="$1"

if [ ! -f "$BACKUP_FILE" ]; then
  echo "Backup file not found: $BACKUP_FILE"
  exit 1
fi

# Stop n8n container
docker stop n8n

# Create a backup of the current data (just in case)
TIMESTAMP=$(date +"%Y%m%d%H%M%S")
CURRENT_BACKUP="~/n8n_current_data_$TIMESTAMP.tar.gz"
tar -czvf $CURRENT_BACKUP -C ~/docker_data/n8n data
echo "Current data backed up to: $CURRENT_BACKUP"

# Remove current data
rm -rf ~/docker_data/n8n/data

# Restore from backup
mkdir -p ~/docker_data/n8n
tar -xzvf $BACKUP_FILE -C ~/docker_data/n8n

# Start n8n container
docker start n8n

echo "Restore completed"
EOF

Make the script executable:

Run in terminalbash

chmod +x ~/restore-n8n.sh

To restore from a backup, run:

Run in terminalbash

~/restore-n8n.sh /path/to/backup/file.tar.gz

Replace /path/to/backup/file.tar.gz with the actual path to your backup file.

Troubleshooting Common Issues#

Container Startup Issues#

If the n8n container fails to start, check the logs:

Run in terminalbash

docker logs n8n

Common issues include:

Incorrect environment variables
Port conflicts
Insufficient disk space
Permission problems

If there's a port conflict, verify which process is using port 5678:

Run in terminalbash

sudo lsof -i :5678

To check if n8n is running and listening on the expected port:

Run in terminalbash

docker ps -a

This shows all containers, including stopped ones. If n8n is running, verify it's bound to 0.0.0.0:5678.

SSL Certificate Issues#

If you encounter SSL certificate errors, check:

Certificate file existence and permissions: ls -la /etc/letsencrypt/live/n8n.yourdomain.com/
Certificate renewal capability: sudo certbot renew --dry-run
Nginx configuration correctness: sudo nginx -t

Database Connection Issues#

For PostgreSQL connection problems:

Check if PostgreSQL container is running: docker ps | grep postgres
Check PostgreSQL logs: docker logs postgres
Verify environment variables: docker exec n8n env | grep DB_

WebSocket Connection Failures#

Ensure your Nginx configuration includes WebSocket support:

Run in terminalbash

sudo cat > /etc/nginx/sites-available/n8n << 'EOF'
server {
    server_name n8n.yourdomain.com;

    location / {
        proxy_pass http://localhost:5678;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        # WebSocket support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
    }

    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/n8n.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/n8n.yourdomain.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
}
EOF

Reload Nginx:

Run in terminalbash

sudo systemctl reload nginx

Maintenance Procedures#

Update n8n#

To update n8n to the latest version:

Run in terminalbash

docker pull n8nio/n8n:latest

With Docker Compose:

Run in terminalbash

cd ~/docker_data/n8n
docker compose pull
docker compose up -d

Monitor Logs#

Watch n8n logs for issues:

Run in terminalbash

docker logs -f n8n

Update SSL Certificates#

Check Certbot's auto-renewal status:

Run in terminalbash

sudo systemctl status certbot.timer

Manually trigger certificate renewal:

Run in terminalbash

sudo certbot renew

Update System Packages#

Keep your system updated:

Run in terminalbash

sudo apt update && sudo apt upgrade -y

Conclusion#

Self-hosting n8n on Ubuntu 20.04+ with Docker provides a flexible and powerful workflow automation solution that you have complete control over. The Docker deployment method offers advantages in terms of isolation, consistency, and ease of migration, making it the preferred option for most self-hosting scenarios.

Regular maintenance is essential for keeping your n8n instance secure and up-to-date. Remember to:

Update both n8n and your underlying system regularly
Monitor logs for any issues
Maintain proper backups to protect your workflows and data
Review security settings periodically

With these practices in place, your self-hosted n8n instance will provide a reliable platform for automating workflows and connecting various services.

Comprehensive Guide to Installing and Securing Nginx Proxy Manager on Ubuntu 20.04+

Sun, 09 Mar 2025 16:25:24 GMT

Nginx Proxy Manager offers a streamlined approach to managing your reverse proxy configurations through an intuitive web interface. This powerful tool simplifies the creation of proxy hosts, SSL certificate management, and access control rules without requiring deep knowledge of Nginx configuration syntax. In this guide, we'll walk through the complete process of installing, configuring, and securing Nginx Proxy Manager on Ubuntu 20.04 or newer using Docker.

Introduction to Nginx Proxy Manager#

Nginx Proxy Manager provides a user-friendly web interface for managing Nginx, one of the most powerful and widely used web servers and reverse proxies. It eliminates the need to manually edit complex configuration files by offering an intuitive dashboard where you can:

Set up proxy hosts
Manage SSL certificates
Configure redirections
Implement access controls

Using Docker for deployment offers several advantages, including isolation from the host system, simplified updates, and easy integration with other containerized applications.

The key benefits of using Nginx Proxy Manager include:

Automatic SSL certificate management through Let's Encrypt
Ability to easily manage multiple domains and services through a single interface
Detailed access control options
Streamlined proxy configuration process

All of this is available without sacrificing the robust performance and reliability that Nginx is known for.

Instruction layout/chart

Prerequisites for Installation#

Before proceeding with the installation, ensure that your system meets these requirements:

Ubuntu 20.04 or newer server with at least 1GB of RAM and 10GB of available storage space
Root or sudo access for installing system packages and configuring permissions
Open ports:
- 80 (HTTP)
- 443 (HTTPS)
- 81 (admin interface)
A domain name (highly recommended for securing the admin interface and for Let's Encrypt SSL certificates)
Basic familiarity with Linux command-line operations, networking concepts, and Docker fundamentals

With these prerequisites in place, you're ready to proceed with the installation.

Installing Docker and Docker Compose#

Docker provides the containerization platform that will host our Nginx Proxy Manager installation. Let's start by setting up Docker and Docker Compose on your Ubuntu server.

Update System Packages#

Begin by ensuring your system packages are up to date:

Run in terminalbash

sudo apt update && sudo apt upgrade -y

This command updates your package lists and upgrades all installed packages to their latest versions.

Install Docker Prerequisites#

Next, install the necessary packages that allow apt to use repositories over HTTPS:

Run in terminalbash

sudo apt install apt-transport-https ca-certificates curl software-properties-common gnupg -y

These packages provide the infrastructure needed to securely download and verify Docker installation files.

Add Docker Repository and Install Docker#

Now we'll add Docker's official GPG key and set up the Docker repository:

Run in terminalbash

# Add Docker's official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# Set up the stable repository
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Update apt package index
sudo apt update

# Install Docker Engine
sudo apt install docker-ce docker-ce-cli containerd.io -y

These commands add the Docker repository to your system, update the package index, and install the Docker engine.

Enable and Start Docker Service#

To ensure Docker starts automatically when your system boots:

Run in terminalbash

sudo systemctl start docker
sudo systemctl enable docker

The first command starts Docker immediately, while the second ensures it automatically starts whenever your system boots up.

Verify Docker Installation#

Confirm that Docker was installed correctly by running a test container:

Run in terminalbash

sudo docker run hello-world

If successful, you'll see a message indicating that your Docker installation is working correctly.

Install Docker Compose#

Docker Compose allows us to define and run multi-container Docker applications:

Run in terminalbash

sudo curl -L "https://github.com/docker/compose/releases/download/v2.18.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

The first command downloads the Docker Compose binary, and the second makes it executable.

Setting Up Nginx Proxy Manager with Docker#

Now that Docker and Docker Compose are installed, we can proceed with setting up Nginx Proxy Manager.

Create Directory Structure#

First, create the directory structure for storing Nginx Proxy Manager's data and SSL certificates:

Run in terminalbash

mkdir -p ~/docker_data/nginx-proxy-manager/data ~/docker_data/nginx-proxy-manager/letsencrypt

This command creates two directories: one for the application data (including the database) and another for storing SSL certificates.

Create a Custom Docker Network#

Creating a custom Docker network improves security by isolating containers:

Run in terminalbash

docker network create proxy

This network will be used for communication between Nginx Proxy Manager and other services you may want to proxy.

Create Docker Compose Configuration#

Navigate to the Nginx Proxy Manager directory and create a docker-compose.yml file:

Run in terminalbash

cd ~/docker_data/nginx-proxy-manager
nano docker-compose.yml

Copy and paste the following configuration into the file:

File contentyaml

version: '3'
services:
  app:
    image: 'jc21/nginx-proxy-manager:latest'
    container_name: nginx-proxy-manager
    restart: unless-stopped
    ports:
      - '80:80'
      - '81:81'
      - '443:443'
    environment:
      DB_SQLITE_FILE: "/data/database.sqlite"
      PUID: 1000
      PGID: 1000
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
    networks:
      - proxy

networks:
  proxy:
    external: true

Save and exit the file (press Ctrl+X, then Y, then Enter).

Start Nginx Proxy Manager#

Launch the Nginx Proxy Manager container:

Run in terminalbash

docker compose up -d

The -d flag runs the container in detached mode, allowing it to run in the background. Docker will pull the necessary image and start the container.

Verify the Container is Running#

Check that the container started successfully:

Run in terminalbash

docker ps

You should see the nginx-proxy-manager container in the list of running containers.

Initial Configuration#

Now that Nginx Proxy Manager is running, let's set up the initial configuration.

Access the Web Interface#

Open your web browser and navigate to:

Open in your browserbash

http://your_server_ip:81

Replace your_server_ip with the actual IP address of your server. You should see the Nginx Proxy Manager login page.

For the first-time login, use these default credentials:

Email: admin@example.com
Password: changeme

You'll be prompted to change these credentials immediately after login.

Change Default Credentials#

Enter your own email address and a strong password. These will be your new admin credentials for logging into Nginx Proxy Manager.

Security Best Practices#

Let's implement some security best practices to protect your Nginx Proxy Manager installation.

Securing the Admin Interface#

It's recommended to secure the admin interface (port 81) by creating a proxy host with SSL:

First, create a proxy host for a subdomain (e.g., npm.yourdomain.com) that points to your server's IP address
In Nginx Proxy Manager, go to "Proxy Hosts" → "Add Proxy Host"
Enter the following details:
- Domain Names: npm.yourdomain.com
- Scheme: http
- Forward Hostname/IP: nginx-proxy-manager
- Forward Port: 81
- Enable "Block Common Exploits"
- Enable SSL with Let's Encrypt

This allows you to access the admin interface securely via https://npm.yourdomain.com rather than using the IP address and port 81.

Configure a Firewall#

Set up a basic firewall with UFW to restrict access to your server:

Run in terminalbash

# Install UFW if not already installed
sudo apt install ufw -y

# Allow SSH to prevent being locked out
sudo ufw allow ssh

# Allow Nginx Proxy Manager required ports
sudo ufw allow 80
sudo ufw allow 443
sudo ufw allow 81

# Enable the firewall
sudo ufw enable

Verify the firewall status:

Run in terminalbash

sudo ufw status

You should see that ports 22 (SSH), 80, 81, and 443 are allowed.

Setting Up Backup Procedures#

Regular backups are essential for any production system. Let's set up an automated backup process for Nginx Proxy Manager.

Create a Backup Script#

Create a simple backup script:

Run in terminalbash

nano ~/backup-npm.sh

Copy and paste the following script:

File contentbash

#!/bin/bash
TIMESTAMP=$(date +"%Y%m%d-%H%M%S")
BACKUP_DIR=~/backups/nginx-proxy-manager

# Create backup directory if it doesn't exist
mkdir -p $BACKUP_DIR

# Create a compressed archive of the data
cd ~/docker_data
tar -czf $BACKUP_DIR/npm-backup-$TIMESTAMP.tar.gz nginx-proxy-manager/

# Delete backups older than 7 days
find $BACKUP_DIR -type f -name "npm-backup-*.tar.gz" -mtime +7 -delete

Make the script executable:

Run in terminalbash

chmod +x ~/backup-npm.sh

Schedule Automatic Backups#

Set up a daily backup using cron:

Run in terminalbash

(crontab -l 2>/dev/null; echo "0 0 * * * ~/backup-npm.sh") | crontab -

This schedules the backup script to run every day at midnight.

Restoring from Backups#

If you need to restore from a backup:

Run in terminalbash

# Stop the Nginx Proxy Manager container
cd ~/docker_data/nginx-proxy-manager
docker compose down

# Remove the current data (be careful with this command!)
rm -rf data letsencrypt

# Extract the backup
tar -xzf ~/backups/nginx-proxy-manager/npm-backup-TIMESTAMP.tar.gz -C ~/docker_data/

# Start the container again
docker compose up -d

Replace TIMESTAMP with the actual timestamp of the backup you want to restore.

Common Problems and Solutions#

Even with careful installation and configuration, you may encounter issues with Nginx Proxy Manager. Here are some common problems and their solutions.

Container Fails to Start#

If the Nginx Proxy Manager container fails to start, check the logs for error messages:

Run in terminalbash

docker logs nginx-proxy-manager

Common causes and solutions:

Port conflicts: If ports 80, 81, or 443 are already in use, the container won't start. Check for other services using these ports: sudo lsof -i :80,81,443 Stop or reconfigure any services using these ports.
Permission issues: If the container can't write to the mounted volumes, it may fail. Check permissions on the data directories: ls -la ~/docker_data/nginx-proxy-manager/ Ensure the directories are owned by the user specified in the PUID/PGID environment variables.

SSL Certificate Issues#

If you're having trouble obtaining or renewing SSL certificates:

DNS configuration: Ensure your domain correctly points to your server's IP address: dig +short yourdomain.com
Firewall rules: Let's Encrypt needs to reach your server on ports 80 and 443 to verify domain ownership. Ensure these ports are open.
Rate limiting: Let's Encrypt has rate limits for certificate issuance. If you've requested too many certificates recently, you might need to wait before requesting more.

502 Bad Gateway Errors#

If you're seeing 502 Bad Gateway errors when accessing proxied services:

Service availability: Ensure the proxied service is running and accessible: curl -I http://service-hostname:port
Network connectivity: If using Docker networks, ensure both containers are on the same network.
Timeout settings: For services that take longer to respond, increase the proxy timeout settings in the custom Nginx configuration: proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; send_timeout 600; Add these settings in the "Advanced" tab of your proxy host configuration.

Advanced Configuration Options#

For users who need more advanced features, here are some additional configuration options.

Using MySQL/MariaDB Instead of SQLite#

For larger deployments, using MySQL or MariaDB instead of SQLite can provide better performance:

File contentyaml

version: '3.8'
services:
  app:
    image: 'jc21/nginx-proxy-manager:latest'
    # ... other settings ...
    environment:
      DB_MYSQL_HOST: "db"
      DB_MYSQL_PORT: 3306
      DB_MYSQL_USER: "npm"
      DB_MYSQL_PASSWORD: "secure_password"
      DB_MYSQL_NAME: "npm"
      PUID: 1000
      PGID: 1000
    # ... volumes and networks ...
    depends_on:
      - db

  db:
    image: 'jc21/mariadb-aria:latest'
    container_name: npm-db
    restart: unless-stopped
    environment:
      MYSQL_ROOT_PASSWORD: "secure_root_password"
      MYSQL_DATABASE: "npm"
      MYSQL_USER: "npm"
      MYSQL_PASSWORD: "secure_password"
    volumes:
      - ~/docker_data/nginx-proxy-manager/mysql:/var/lib/mysql
    networks:
      - proxy

Replace "secure_password" and "secure_root_password" with strong, unique passwords.

Custom Nginx Configurations#

Nginx Proxy Manager allows you to add custom Nginx configurations to each proxy host:

Custom headers: add_header X-Frame-Options "SAMEORIGIN"; add_header X-Content-Type-Options "nosniff"; add_header X-XSS-Protection "1; mode=block";
URL rewrites: rewrite ^/old-path$ /new-path permanent;
Advanced authentication: Implement HTTP basic authentication or more complex authentication schemes.

These custom configurations can be added in the "Advanced" tab when editing a proxy host.

Maintenance and Updates#

Regular maintenance and updates are essential for keeping your Nginx Proxy Manager installation secure and functioning correctly.

Updating Nginx Proxy Manager#

To update Nginx Proxy Manager to the latest version:

Run in terminalbash

cd ~/docker_data/nginx-proxy-manager
docker compose pull
docker compose down
docker compose up -d

This process pulls the latest image, stops the current container, and starts a new container with the updated image.

Monitoring Container Health#

Regularly monitor the health of your Nginx Proxy Manager container:

Run in terminalbash

# Check container status
docker ps -a | grep nginx-proxy-manager

# View recent logs
docker logs --tail 100 nginx-proxy-manager

# Monitor resource usage
docker stats nginx-proxy-manager

These commands allow you to verify that the container is running correctly and check for any error messages.

Certificate Renewals#

Let's Encrypt certificates are valid for 90 days, and Nginx Proxy Manager automatically attempts to renew them when they're 30 days from expiration. Monitor certificate renewals:

Run in terminalbash

docker exec -it nginx-proxy-manager certbot certificates

This command lists all certificates managed by Let's Encrypt, along with their expiration dates.

System Updates#

Regularly update your Ubuntu server to ensure you have the latest security patches:

Run in terminalbash

sudo apt update && sudo apt upgrade -y

After significant system updates, especially kernel updates, you might need to reboot your server.

Conclusion#

In this comprehensive guide, we've covered the complete process of installing, configuring, and securing Nginx Proxy Manager on Ubuntu 20.04 or newer using Docker. By following these steps, you've established a robust reverse proxy solution that provides an intuitive web interface for managing your proxy configurations and SSL certificates.

We've explored security best practices, including the use of custom Docker networks, firewall configuration, access controls, and regular backups. These measures help protect your proxy manager and the services it handles from unauthorized access and potential data loss.

Nginx Proxy Manager simplifies what would otherwise be a complex task, making it accessible to users with varying levels of technical expertise. Whether you're managing a single website or dozens of services, this tool provides a streamlined approach to proxy management without sacrificing the power and flexibility of Nginx.

Remember to regularly update both Nginx Proxy Manager and your underlying system, monitor for security issues, and maintain good backup practices. With proper care, your Nginx Proxy Manager installation will serve as a reliable gateway to your services for years to come.

A Complete Guide to Installing and Securing Portainer on Ubuntu 20.04 and Up

Sun, 09 Mar 2025 15:35:40 GMT

Introduction#

Portainer is a powerful, lightweight container management platform that provides an intuitive web-based interface for managing Docker environments. Originally designed solely for Docker, Portainer has evolved to support multiple container environments including Docker Swarm, Kubernetes, and Azure Container Instances. It significantly reduces the learning curve for Docker management by eliminating the need to memorize complex command-line instructions.

Portainer is available in two editions:

Community Edition (CE): Free and open-source, designed primarily for individual use and home labs
Business Edition (BE): Offers extended features like advanced role-based access control and registry management for enterprise environments

This guide focuses on the Community Edition, which provides robust features suitable for most users while requiring minimal system resources.

Instruction layout/chart

Prerequisites#

Hardware Requirements#

Portainer is remarkably lightweight:

CPU: 1 core (minimum)
RAM: 1 GB (minimum)
Storage: 10 GB available space for Docker and Portainer
Network connectivity: Internet access for pulling Docker images

Software Requirements#

To successfully install and run Portainer, you'll need:

Ubuntu 20.04 LTS or newer (this guide is tested on Ubuntu 20.04 and 22.04)
Current system packages (we'll update these during installation)
Terminal access with sudo privileges

Network Requirements#

Portainer requires specific ports to be accessible:

Port 9443: For secure HTTPS access to the Portainer web interface (default)
Port 8000: For the Edge agent communication (only if using Edge compute features)
Port 9000: For HTTP access (optional, mainly for legacy compatibility)

User Permissions#

You'll need a user account with sudo privileges for installation. Following security best practices, we'll avoid using the root account directly.

Step 1: Update Your System#

Before beginning any installation, ensure your system is up-to-date with the latest security patches and package versions.

Run in terminalbash

sudo apt update && sudo apt upgrade -y

This command updates your package lists and upgrades all installed packages to their latest versions. If prompted for a restart after updating, reboot before continuing.

Step 2: Install Docker Engine#

Portainer requires Docker Engine to function. If you already have Docker installed, verify it with docker --version and skip to Step 3.

Remove any conflicting packages#

Run in terminalbash

sudo apt-get remove docker docker-engine docker.io containerd runc -y

This ensures a clean slate by removing any older or conflicting versions.

Install required dependencies#

Run in terminalbash

sudo apt install apt-transport-https ca-certificates curl gnupg lsb-release -y

These packages allow apt to use repositories over HTTPS, work with certificates, and fetch Docker's official GPG keys.

Add Docker's official GPG key#

Run in terminalbash

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

This downloads Docker's GPG key and stores it for package verification, ensuring the packages you install are authentic.

Set up the Docker repository#

Run in terminalbash

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

This adds the official Docker repository to your system's sources list, automatically detecting your system architecture and Ubuntu version.

Update package lists with the new repository#

Run in terminalbash

sudo apt update

This refreshes your package lists with the newly added Docker repository.

Install Docker Engine#

Run in terminalbash

sudo apt install docker-ce docker-ce-cli containerd.io -y

This installs the Docker Engine, CLI, and containerd runtime.

Start and enable Docker service#

Run in terminalbash

sudo systemctl start docker
sudo systemctl enable docker

These commands start the Docker service and configure it to start automatically when the system boots.

Verify Docker installation#

Run in terminalbash

sudo systemctl status docker

This checks that Docker is running properly. You should see output indicating that the service is active (running).

Add your user to the Docker group#

Run in terminalbash

sudo usermod -aG docker $USER

This adds your current user to the Docker group, allowing you to run Docker commands without using sudo. This change will take effect after you log out and log back in, or apply it immediately with:

Run in terminalbash

newgrp docker

Test Docker functionality#

Run in terminalbash

docker run hello-world

This pulls and runs the hello-world image. If successful, you'll see a message indicating that Docker is properly installed and your user has the appropriate permissions.

Step 3: Install Portainer#

Now that Docker is installed and running, we can proceed with installing Portainer. We'll present multiple methods, allowing you to choose the one that best fits your needs.

Method 1: Standard Docker Installation#

This is the simplest method, using basic Docker commands.

Create a persistent volume for Portainer data#

Run in terminalbash

docker volume create portainer_data

This creates a named volume that will store Portainer's data, ensuring it persists even if the container is removed or updated.

Run Portainer container#

Run in terminalbash

docker run -d -p 8000:8000 -p 9443:9443 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest

This command runs the Portainer container with the following configuration:

-d: Runs the container in detached mode (in the background)
-p 8000:8000 -p 9443:9443: Maps container ports to host ports for web UI and Edge agent communication
--name=portainer: Names the container "portainer" for easy reference
--restart=always: Ensures the container automatically restarts if it stops or if Docker restarts
-v /var/run/docker.sock:/var/run/docker.sock: Mounts the Docker socket to give Portainer access to the Docker API
-v portainer_data:/data: Mounts the previously created volume to store Portainer data
portainer/portainer-ce:latest: Specifies the image to use (latest Portainer Community Edition)

Verify Portainer is running#

Run in terminalbash

docker ps | grep portainer

You should see the Portainer container listed with its status as "Up".

Method 2: Docker Compose Installation#

Docker Compose provides a more declarative way to deploy containers, making it easier to understand and maintain your configuration.

Install Docker Compose plugin (if not already installed)#

Run in terminalbash

sudo apt install docker-compose-plugin -y

This installs the Docker Compose plugin, which provides the docker compose command.

Create a directory for Portainer#

Run in terminalbash

mkdir -p ~/docker_data/portainer
cd ~/docker_data/portainer

This creates a directory structure for our Portainer deployment.

Create docker-compose.yml file#

Create a docker-compose.yml file with proper configuration. Note: The exact content of this file was not fully provided in the original document.

Step 4: Access and Initial Setup#

After installing Portainer, you need to access it and complete the initial setup.

Access the Portainer Web Interface#

Open your web browser and navigate to:

Open in your browserbash

https://YOUR_SERVER_IP:9443

Replace YOUR_SERVER_IP with your server's IP address. You may see a security warning because Portainer uses a self-signed certificate by default. This is expected - you can proceed safely for initial setup.

Create the Admin User#

On first access, you'll be prompted to create an admin user:

Enter a username (commonly "admin")
Create a strong password (at least 12 characters with a mix of uppercase, lowercase, numbers, and special characters)
Click "Create User"

Connect to the Local Docker Environment#

After creating the admin user, you'll be prompted to connect to a Docker environment:

Choose "Docker" as the environment type
Select "Connect to local Docker environment" since we're managing the Docker installation on the same machine
Click "Connect"

You're now ready to use Portainer to manage your Docker environment!

Step 5: Enhancing Security#

Security is crucial for any container management system. Here are essential steps to secure your Portainer installation:

Enable HTTPS with a Proper SSL Certificate#

For production environments, replace the self-signed certificate with a proper SSL certificate:

Run in terminalbash

docker stop portainer
docker rm portainer
docker run -d -p 8000:8000 -p 9443:9443 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data -v /path/to/certs:/certs portainer/portainer-ce:latest --ssl --sslcert /certs/portainer.crt --sslkey /certs/portainer.key

Replace /path/to/certs with the actual path to your certificate files.

Configure Firewall Rules#

Restrict access to Portainer's ports using UFW (Uncomplicated Firewall):

Run in terminalbash

sudo ufw allow 22/tcp
sudo ufw allow 9443/tcp
sudo ufw enable

This allows SSH (port 22) and Portainer's web interface (port 9443) while blocking other incoming connections.

Implement User Management#

Once logged into Portainer:

Go to "Settings" > "Users" to create additional users with appropriate permissions
Create teams and assign users to them for more granular access control
Use Role-Based Access Control (RBAC) to limit what different users can do

This follows the principle of least privilege, ensuring users only have access to the resources they need.

Step 6: Data Persistence and Backup#

Proper data management ensures you can recover from failures and maintain continuity of your Portainer installation.

Creating a Backup of Portainer Data#

To create a backup of your Portainer data:

Run in terminalbash

docker run --rm -v portainer_data:/data -v $(pwd):/backup alpine sh -c "cd /data && tar -czf /backup/portainer-backup-$(date +%Y%m%d%H%M%S).tar.gz ."

This command creates a timestamped backup archive of your Portainer data in the current directory.

Scheduling Regular Backups#

For automated backups, create a simple bash script and schedule it with cron:

Run in terminalbash

cat > ~/backup-portainer.sh << 'EOF'
#!/bin/bash
BACKUP_DIR=~/portainer-backups
mkdir -p $BACKUP_DIR
docker run --rm -v portainer_data:/data -v $BACKUP_DIR:/backup alpine sh -c "cd /data && tar -czf /backup/portainer-backup-$(date +%Y%m%d%H%M%S).tar.gz ."
find $BACKUP_DIR -type f -name "portainer-backup-*.tar.gz" -mtime +7 -delete
EOF

chmod +x ~/backup-portainer.sh
(crontab -l 2>/dev/null; echo "0 2 * * * ~/backup-portainer.sh") | crontab -

This sets up a daily backup at 2:00 AM and retains only the seven most recent backups.

Restoring from Backup#

To restore Portainer from a backup:

Stop the Portainer container:

Run in terminalbash

docker stop portainer

Remove the existing volume (caution: this will delete all current Portainer data):

Run in terminalbash

docker volume rm portainer_data
docker volume create portainer_data

Restore data from your backup:

Run in terminalbash

docker run --rm -v portainer_data:/data -v /path/to/backup/directory:/backup alpine sh -c "cd /data && tar -xzf /backup/your-backup-file.tar.gz"

Replace /path/to/backup/directory and your-backup-file.tar.gz with the actual backup directory and file name.

Restart Portainer:

Run in terminalbash

docker start portainer

Step 7: Common Problems and Troubleshooting#

Even with careful installation, issues can arise. Here are solutions to common problems:

Connection Refused Errors#

Problem: Unable to connect to Portainer web interface.

Diagnostic command:

Run in terminalbash

curl -kI https://localhost:9443

Solutions:

Verify Portainer is running: docker ps | grep portainer
Check if the correct ports are being exposed: docker port portainer
Ensure your firewall allows connections to port 9443: sudo ufw status
If using a different port, make sure you're accessing the correct one

Certificate Issues#

Problem: SSL certificate errors when accessing Portainer.

Solutions:

For development/testing: Add a security exception in your browser
For production: Use a proper SSL certificate as described in the security section
If using a custom certificate, verify it's correctly formatted and not expired: openssl x509 -in /path/to/your/certs/portainer.crt -text -noout

Docker Socket Permission Issues#

Problem: Portainer cannot connect to Docker daemon, often showing "Error 500" messages.

Diagnostic command:

Run in terminalbash

docker logs portainer

Look for messages related to permission denied on /var/run/docker.sock.

Solutions:

Ensure the Docker socket is properly mounted: docker inspect portainer | grep docker.sock
Check Docker socket permissions: ls -la /var/run/docker.sock
If necessary, adjust permissions (use with caution as this has security implications): sudo chmod 666 /var/run/docker.sock

Port Conflicts#

Problem: Container fails to start due to port conflicts.

Diagnostic command:

Run in terminalbash

sudo netstat -tulpn | grep -E '9443|8000'

Solutions:

Identify the process using the conflicting port: sudo lsof -i :9443
Stop the conflicting service or use different ports for Portainer: docker run -d -p 9444:9443 -p 8001:8000 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest

Browser Cache Issues#

Problem: UI appears broken or outdated after updates.

Solutions:

Clear your browser cache
Try accessing Portainer in a private/incognito window
Use a different browser

Step 8: Maintenance Procedures#

Regular maintenance ensures your Portainer installation remains secure and up-to-date.

Updating Portainer#

To update Portainer to the latest version:

Run in terminalbash

# For standard Docker installation
docker stop portainer
docker rm portainer
docker pull portainer/portainer-ce:latest
docker run -d -p 8000:8000 -p 9443:9443 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest

Run in terminalbash

# For Docker Compose installation
cd ~/docker_data/portainer
docker compose pull
docker compose up -d

This process stops and removes the current container, pulls the latest image, and creates a new container with the same settings. Your data is preserved because it's stored in the persistent volume.

Monitoring Portainer#

To monitor Portainer's resource usage:

Run in terminalbash

docker stats portainer

For logs:

Run in terminalbash

docker logs -f portainer

The -f flag follows the log output in real-time. Press Ctrl+C to exit.

Cleanup and Optimization#

Regularly clean up unused Docker resources to prevent disk space issues:

Remove unused containers: docker container prune -f
Remove unused images: docker image prune -f
Remove unused volumes (be careful, as this will delete data): docker volume prune -f

Security Auditing#

Periodically review your Portainer security configuration:

Check for outdated versions with known vulnerabilities
Review user accounts and permissions
Verify SSL certificate validity
Check firewall rules
Examine access logs for suspicious activity

Conclusion#

You've now successfully installed, secured, and configured Portainer on your Ubuntu 20.04 or newer system. This powerful container management platform provides an intuitive interface for managing Docker environments, making it easier to deploy and maintain containerized applications. By following the security practices outlined in this guide, you've established a solid foundation for safe container management.

Regular maintenance tasks like backups, updates, and security audits will ensure your Portainer installation remains reliable and secure over time. As your containerized infrastructure grows, Portainer's robust management capabilities will continue to simplify operations and enhance productivity.

Whether you've chosen a simple standalone installation or a more complex setup, you now have the knowledge to effectively manage and secure your Portainer environment. The comprehensive troubleshooting section should help you resolve any issues that might arise, ensuring smooth operation of your container management solution.

The Complete Guide to Installing and Securing Uptime Kuma on Ubuntu 20.04 and Later

Sun, 09 Mar 2025 09:38:23 GMT

Uptime Kuma is a powerful, open-source, self-hosted monitoring tool that allows you to track the availability and performance of your websites, services, and applications. This comprehensive guide covers multiple deployment methods for Ubuntu 20.04 and newer, with a primary focus on Docker as the recommended approach for most users.

Understanding Uptime Kuma#

Uptime Kuma provides a sleek, modern web interface for monitoring the uptime of your digital assets. Unlike proprietary solutions, Uptime Kuma gives you complete control over your monitoring data while offering features such as:

Status pages
Comprehensive notification options (including Telegram, Discord, Slack, email)
Detailed metrics about response times and availability

The application is based on Node.js and uses WebSocket for real-time updates, ensuring you receive immediate notifications when a service goes down. It offers monitoring via multiple methods including HTTP/HTTPS, TCP, Ping, DNS, and more, making it versatile for different types of services. The interface is intuitive and responsive, making it accessible even to users with minimal technical experience.

Prerequisites#

Before beginning the installation, ensure your system meets the following requirements:

Hardware Requirements#

CPU: At least 1 vCPU
RAM: Minimum 512 MB (1 GB or more recommended)
Storage: At least 10 GB of available disk space
A stable internet connection

Software Requirements#

Ubuntu 20.04 or newer (instructions also work for Ubuntu 22.04 and later)
SSH access with sudo privileges
Basic familiarity with terminal commands

Network Requirements#

Open port 3001 for initial access (can be changed later)
If using a reverse proxy (recommended): open ports 80 and 443

Instruction Layout Graph:

Installation Method Comparison#

There are multiple ways to install Uptime Kuma on Ubuntu, each with its own benefits:

Docker Installation (Recommended)#

Docker provides the simplest installation method with excellent isolation and easier updates. This is the recommended approach for most users as it requires minimal system configuration and offers the most straightforward upgrade path.

Docker Compose Installation#

Similar to Docker but offers better management for multi-container applications. Ideal if you're already using Docker Compose for other services.

Native Installation#

Installing directly on the host system. More resource-efficient but requires more system configuration and dependency management.

For most users, the Docker installation provides the best balance of simplicity, security, and maintainability. We'll focus primarily on this method while also providing instructions for alternatives.

Docker Installation Method#

This section covers installing Uptime Kuma using Docker, which is the recommended method for most users.

Step 1: Update System Packages#

First, update your system packages to ensure you have the latest security updates and package information:

Run in terminalbash

sudo apt update && sudo apt upgrade -y

This updates your package lists and upgrades all installed packages to their latest versions.

Step 2: Install Docker#

If Docker is not already installed on your system, install it using the following commands:

Run in terminalbash

sudo apt install apt-transport-https ca-certificates curl software-properties-common -y

This installs prerequisite packages needed for the Docker installation.

Run in terminalbash

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

This adds Docker's official GPG key. If you get a warning about apt-key being deprecated, you can ignore it for now.

Run in terminalbash

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

This adds the Docker repository for your specific Ubuntu version.

Run in terminalbash

sudo apt update

This updates package information from the newly added repository.

Run in terminalbash

sudo apt install docker-ce docker-ce-cli containerd.io -y

This installs Docker Community Edition, the CLI, and containerd.

Run in terminalbash

sudo systemctl status docker

This verifies Docker is running properly. You should see "active (running)" in the output. Press Q to exit this view.

Step 3: Set Up Docker for Non-Root Use (Optional but Recommended)#

To avoid having to use sudo for every Docker command, add your user to the Docker group:

Run in terminalbash

sudo usermod -aG docker $USER

This adds your current user to the Docker group. You'll need to log out and back in for this change to take effect.

Step 4: Create a Docker Volume for Uptime Kuma#

Create a Docker volume to store Uptime Kuma's data persistently:

Run in terminalbash

docker volume create uptime-kuma

This creates a named volume that will persist even if the container is removed.

Step 5: Run the Uptime Kuma Container#

Now, deploy the Uptime Kuma container:

Run in terminalbash

docker run -d --restart=always -p 3001:3001 -v uptime-kuma:/app/data --name uptime-kuma louislam/uptime-kuma:1

This command:

Runs the container in detached mode (-d)
Sets it to always restart if it fails or after a system reboot (--restart=always)
Maps port 3001 on your host to port 3001 in the container (-p 3001:3001)
Mounts the volume we created to store data (-v uptime-kuma:/app/data)
Names the container "uptime-kuma" for easy reference (--name uptime-kuma)
Uses the "louislam/uptime-kuma:1" image, which is the stable version (louislam/uptime-kuma:1)

Step 6: Verify the Container is Running#

Check if the container is running correctly:

Run in terminalbash

docker ps | grep uptime-kuma

You should see the uptime-kuma container in the list. If not, check for errors using:

Run in terminalbash

docker logs uptime-kuma

This shows the container logs, which can help identify any startup issues.

Step 7: Access the Uptime Kuma Web Interface#

Open a web browser and navigate to:

Open in your browserbash

http://YOUR_SERVER_IP:3001

Replace YOUR_SERVER_IP with your server's actual IP address.

You'll be presented with the initial setup screen where you can:

Select your preferred language
Create an administrator username and password
Complete the setup

Docker Compose Installation Method#

If you prefer using Docker Compose for container management, follow these steps instead of the Docker method above.

Step 1: Update System Packages#

Run in terminalbash

sudo apt update && sudo apt upgrade -y

This ensures your system is up to date.

Step 2: Install Docker and Docker Compose#

If not already installed:

Run in terminalbash

sudo apt install apt-transport-https ca-certificates curl software-properties-common -y

Run in terminalbash

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Run in terminalbash

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

Run in terminalbash

sudo apt update

Run in terminalbash

sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y

Step 3: Create a Directory for Uptime Kuma#

Run in terminalbash

mkdir -p ~/uptime-kuma
cd ~/uptime-kuma

This creates a directory to store your Docker Compose configuration.

Step 4: Create the Docker Compose File#

Create a docker-compose.yml file:

Run in terminalbash

nano docker-compose.yml

Paste the following content:

File contentbash

version: '3'
services:
  uptime-kuma:
    image: louislam/uptime-kuma:1
    container_name: uptime-kuma
    restart: always
    ports:
      - "3001:3001"
    volumes:
      - uptime-kuma:/app/data
    security_opt:
      - no-new-privileges:true
volumes:
  uptime-kuma:

Save and exit by pressing Ctrl+X, then Y, then Enter.

Step 5: Start Uptime Kuma with Docker Compose#

Run in terminalbash

docker compose up -d

This starts the container in detached mode. Note: in newer versions of Docker, the command is docker compose (without a hyphen) rather than docker-compose.

Step 6: Verify Uptime Kuma is Running#

Run in terminalbash

docker ps | grep uptime-kuma

Step 7: Access the Web Interface#

Access Uptime Kuma at http://YOUR_SERVER_IP:3001

Securing Uptime Kuma with a Reverse Proxy#

Running Uptime Kuma behind a reverse proxy with SSL encryption is highly recommended for production environments. This section covers setting up Caddy as a reverse proxy, which automatically handles SSL certificate provisioning.

Step 1: Install Caddy#

Run in terminalbash

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https

Run in terminalbash

curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg

Run in terminalbash

curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list

Run in terminalbash

sudo apt update

Run in terminalbash

sudo apt install caddy

Step 2: Configure Caddy#

Edit the Caddy configuration file:

Run in terminalbash

sudo nano /etc/caddy/Caddyfile

Replace the content with:

File contentbash

your-domain.com {
    reverse_proxy localhost:3001
}

Replace your-domain.com with your actual domain name that points to your server.

Save and exit by pressing Ctrl+X, then Y, then Enter.

Step 3: Restart Caddy#

Run in terminalbash

sudo systemctl restart caddy

Step 4: Configure Firewall (if enabled)#

If you're using UFW (Ubuntu's default firewall):

Run in terminalbash

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

This allows HTTP and HTTPS traffic.

Step 5: Access Uptime Kuma Securely#

Now you can access Uptime Kuma via HTTPS:

Open in your browserbash

https://your-domain.com

Caddy automatically provisions and renews SSL certificates from Let's Encrypt.

Configuration and Usage#

After successfully installing Uptime Kuma, you'll need to configure it to monitor your services.

Initial Setup#

Upon first access, select your preferred language
Create an administrator account with a strong password
Click "Create" to finalize the setup

Adding Monitors#

From the dashboard, click "Add New Monitor"
Enter a name for your monitor
Select the type of monitoring (HTTP(s), Ping, TCP, etc.)
Enter the URL or IP address to monitor
Configure the check interval (how often to check)
Configure notification settings if desired
Click "Save" to create the monitor

Setting Up Notifications#

Uptime Kuma supports various notification methods:

Click "Settings" in the sidebar
Go to "Notification" tab
Click "Setup Notification"
Select your preferred notification method
Fill in the required information
Test the notification
Save the configuration

Data Persistence and Backups#

Understanding Docker Volume Persistence#

The Docker installation uses a volume named uptime-kuma to store data. This volume persists even if the container is removed, ensuring your configuration and monitoring history remain intact during updates or container recreation.

Creating Backups#

To back up your Uptime Kuma data:

Run in terminalbash

# Find the volume's mount point
docker volume inspect uptime-kuma

Look for the "Mountpoint" value in the output.

Run in terminalbash

# Create a backup (run as root or with sudo)
sudo tar -czvf uptime-kuma-backup-$(date +%Y%m%d).tar.gz -C /var/lib/docker/volumes/uptime-kuma/_data .

This creates a compressed backup of your data with the current date in the filename.

Restoring from Backup#

To restore from a backup:

Run in terminalbash

# Stop the container
docker stop uptime-kuma

# Remove the container (data volume remains intact)
docker rm uptime-kuma

# Extract backup to the volume (run as root or with sudo)
sudo tar -xzvf uptime-kuma-backup-YYYYMMDD.tar.gz -C /var/lib/docker/volumes/uptime-kuma/_data

# Start a new container using the same volume
docker run -d --restart=always -p 3001:3001 -v uptime-kuma:/app/data --name uptime-kuma louislam/uptime-kuma:1

Replace YYYYMMDD with the date of your backup file.

Updating Uptime Kuma#

Updating Docker Installation#

Run in terminalbash

# Pull the latest image
docker pull louislam/uptime-kuma:1

# Stop and remove the current container
docker stop uptime-kuma
docker rm uptime-kuma

# Create a new container with the updated image
docker run -d --restart=always -p 3001:3001 -v uptime-kuma:/app/data --name uptime-kuma louislam/uptime-kuma:1

Updating Docker Compose Installation#

Run in terminalbash

# Navigate to your compose directory
cd ~/uptime-kuma

# Pull the latest image
docker compose pull

# Restart the container with the new image
docker compose up -d

Common Issues and Troubleshooting#

Container Fails to Start#

If the container doesn't start:

Run in terminalbash

docker logs uptime-kuma

Look for error messages in the output. Common issues include:

Port conflicts: If port 3001 is already in use, change the port mapping in your docker run command, e.g., -p 3002:3001
Insufficient permissions: Ensure your user has the correct Docker permissions

Cannot Access Web Interface#

If you cannot access the web interface:

Verify the container is running: docker ps | grep uptime-kuma
Check if the port is accessible: curl -v localhost:3001
Verify firewall settings: sudo ufw status Ensure port 3001 is allowed (or 80/443 if using a reverse proxy)

Notifications Not Working#

If notifications aren't being sent:

Test notification settings from the Uptime Kuma interface
Check outbound connectivity from your server
Verify service-specific configuration (e.g., correct API keys, tokens)

Monitor Shows False Positives#

If monitors are incorrectly reporting services as down:

Check the monitor settings for appropriate timeouts
Verify connectivity between your Uptime Kuma server and the monitored service
Consider adding retry settings to monitors to reduce false positives

Known Issues and Workarounds#

Docker Alpine Version Issues#

The Alpine-based Docker image has known DNS resolution issues. Use the Debian-based image instead by specifying "louislam/uptime-kuma:1" or "louislam/uptime-kuma:1-debian".

WebSocket Connection Issues with Reverse Proxy#

When using a reverse proxy, WebSocket connections may fail if not properly configured. Ensure your reverse proxy configuration includes the appropriate WebSocket headers:

For Nginx:

File contentbash

proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

For Apache:

Codebash

RewriteEngine On
RewriteCond %{HTTP:Upgrade} =websocket [NC]
RewriteRule /(.*) ws://localhost:3001/$1 [P,L]

Caddy handles WebSocket correctly by default with the simple configuration provided earlier.

Security Best Practices#

Use a Strong Admin Password#

Create a secure, unique password for your Uptime Kuma admin account.

Implement a Reverse Proxy with SSL#

As described in the Caddy setup section, always use HTTPS in production environments.

Limit Access with Firewall Rules#

Only open the necessary ports on your server:

Port 3001 if accessing Uptime Kuma directly (development only)
Ports 80 and 443 if using a reverse proxy (recommended)

Consider IP Restrictions#

If using Nginx or Apache as a reverse proxy, consider implementing IP restrictions to limit administrative access.

Maintenance Tasks#

Regular Updates#

Check for updates regularly and follow the update procedure provided earlier. Uptime Kuma development is active, and updates often include security fixes and new features.

Log Monitoring#

Check container logs periodically for warnings or errors:

Run in terminalbash

docker logs uptime-kuma

Database Maintenance#

Uptime Kuma uses SQLite for data storage. No regular database maintenance is typically required, but creating periodic backups is recommended.

Conclusion#

You've successfully installed, secured, and configured Uptime Kuma on Ubuntu 20.04 or newer. By following the Docker or Docker Compose installation method and implementing a reverse proxy with SSL, you now have a robust monitoring solution for your websites and services.

With proper maintenance, backup procedures, and security practices, your Uptime Kuma installation will provide reliable uptime monitoring for your infrastructure. Remember to check for updates regularly and implement the additional security measures recommended in this guide to ensure your monitoring system remains secure and effective.

Complete Guide to Installing and Securing Mastodon with Docker on Ubuntu

Sun, 09 Mar 2025 08:56:14 GMT

Introduction#

Mastodon is an open-source, decentralized social network that gives users control over their data and content. Unlike centralized platforms such as Twitter, Mastodon operates on a federated model where independent servers (instances) connect to form a larger network.

Docker provides an excellent deployment method for Mastodon because it offers:

Isolation between components
Consistency across environments
Simplified management and maintenance
Easier upgrades and backups
Enhanced security through containerization

This comprehensive guide will walk you through setting up your own Mastodon instance using Docker on Ubuntu 20.04 or newer.

Instruction Layout

Prerequisites#

Before beginning installation, ensure you have:

A server running Ubuntu 20.04 or newer
At least 2GB of RAM and 2 CPU cores (4GB+ recommended for production)
At least 20GB of free storage space (more for a growing instance)
A registered domain name pointing to your server
Basic familiarity with Linux command line
Root or sudo access to your server
An SMTP provider for sending emails (required for account verification)
Ports 80 and 443 open on your firewall for HTTP/HTTPS traffic

Hardware Recommendations#

For optimal performance, consider these resource guidelines:

Minimum: 2GB RAM, 2 CPU cores, 20GB storage
Recommended: 4GB RAM, 2-4 CPU cores, 40GB+ storage
For larger instances: 8GB+ RAM, 4+ CPU cores, 100GB+ storage with separate database server

Networking Requirements#

Your server must have these ports accessible:

Port 22: SSH access (restrict to your IP if possible)
Port 80: HTTP (for initial setup and redirecting to HTTPS)
Port 443: HTTPS (for secure access to your instance)

System Preparation#

Let's start by updating your system and installing essential utilities.

Run in terminalbash

sudo apt update && sudo apt upgrade -y

This updates your package lists and upgrades installed packages to their latest versions.

Run in terminalbash

sudo apt install -y ca-certificates curl gnupg lsb-release apt-transport-https software-properties-common

This installs necessary utilities for adding repositories and downloading packages securely.

Setting Up Firewall (UFW)#

Configure a basic firewall to protect your server:

Run in terminalbash

sudo ufw status

Checks if the firewall is active. If the output shows "inactive," proceed with the following steps.

Run in terminalbash

sudo ufw allow OpenSSH

Allows SSH connections so you don't get locked out of your server.

Run in terminalbash

sudo ufw allow 80/tcp

Opens HTTP port for initial setup and Let's Encrypt verification.

Run in terminalbash

sudo ufw allow 443/tcp

Opens HTTPS port for secure web traffic.

Run in terminalbash

sudo ufw enable

Activates the firewall. You'll be prompted to confirm with 'y' - this might disconnect your SSH session if you haven't allowed OpenSSH first.

Run in terminalbash

sudo ufw status

Verifies the firewall is active and shows the allowed services.

Installing Docker and Docker Compose#

Docker allows us to run Mastodon and its dependencies in isolated containers.

Install Docker#

Run in terminalbash

sudo apt remove docker docker-engine docker.io containerd runc

Removes any old Docker versions that might conflict with the installation.

Run in terminalbash

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Downloads and installs Docker's official GPG key for package verification.

Run in terminalbash

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Adds Docker's official repository to your system.

Run in terminalbash

sudo apt update

Updates package lists to include Docker packages.

Run in terminalbash

sudo apt install -y docker-ce docker-ce-cli containerd.io

Installs Docker Engine and related packages.

Run in terminalbash

sudo systemctl status docker

Verifies Docker is running. You should see "active (running)" in the output.

Install Docker Compose Plugin#

Run in terminalbash

sudo apt install -y docker-compose-plugin

Installs the Docker Compose plugin through apt.

Run in terminalbash

docker compose version

Verifies Docker Compose is installed properly. You should see version information.

Add Your User to Docker Group (Optional)#

This step allows you to run Docker commands without sudo.

Run in terminalbash

sudo usermod -aG docker $USER

Adds your current user to the Docker group. You'll need to log out and back in for this to take effect.

Setting Up Mastodon#

Now let's create the directory structure and download Mastodon.

Run in terminalbash

sudo mkdir -p /opt/mastodon

Creates a directory for Mastodon files.

Run in terminalbash

sudo chown $USER:$USER /opt/mastodon

Gives your user ownership of the Mastodon directory.

Run in terminalbash

cd /opt/mastodon

Changes to the Mastodon directory.

Create Directory Structure#

Run in terminalbash

mkdir -p public/system

Creates a directory for user uploads and other files.

Run in terminalbash

mkdir -p postgres

Creates a directory to store PostgreSQL data.

Run in terminalbash

mkdir -p redis

Creates a directory to store Redis data.

Download Mastodon Docker Configuration#

Run in terminalbash

curl -L https://raw.githubusercontent.com/mastodon/mastodon/main/docker-compose.yml -o docker-compose.yml

Downloads the official Mastodon docker-compose.yml file.

Run in terminalbash

curl -L https://raw.githubusercontent.com/mastodon/mastodon/main/.env.production.sample -o .env.production

Downloads the sample environment configuration file.

Configure Docker Compose#

Now we need to modify the docker-compose.yml file to use prebuilt images instead of building locally.

Run in terminalbash

sed -i 's/^  build: .$/  # build: /' docker-compose.yml

Comments out the build directive for each service.

Run in terminalbash

sed -i 's/^  image: tootsuite\/mastodon$/  image: tootsuite\/mastodon:latest/' docker-compose.yml

Changes the image reference to use the latest prebuilt image. For production, you might want to specify a specific version.

Setting File Permissions#

Run in terminalbash

sudo chown -R 991:991 public

Sets the correct permissions for the public directory. 991 is the default user ID for Mastodon in the Docker container.

Configuring Mastodon#

Now we'll set up the configuration file for Mastodon.

Generate Secrets#

Run in terminalbash

docker compose run --rm web rake secret

Generates a random string for the SECRET_KEY_BASE. Copy this output.

Run in terminalbash

docker compose run --rm web rake secret

Run this command again to generate a random string for OTP_SECRET. Copy this output.

Run in terminalbash

docker compose run --rm web bundle exec rake mastodon:webpush:generate_vapid_key

Generates VAPID keys for Web Push notifications. Copy both the private and public keys.

Edit the Environment Configuration#

Now you need to edit the .env.production file to configure your Mastodon instance:

Run in terminalbash

nano .env.production

Opens the environment file for editing. You'll need to update the following settings:

LOCAL_DOMAIN: Set to your domain name
DB_HOST: Set to "db"
DB_USER: Set to "postgres"
DB_NAME: Set to "postgres"
DB_PASS: Leave empty for this setup
REDIS_HOST: Set to "redis"
SECRET_KEY_BASE: Paste the first secret generated earlier
OTP_SECRET: Paste the second secret generated earlier
VAPID_PRIVATE_KEY and VAPID_PUBLIC_KEY: Paste the VAPID keys generated earlier
SMTP settings: Fill in with your email provider's details

SMTP Configuration Example#

Here's an example SMTP configuration using Gmail (you should use an application-specific password):

File contentbash

SMTP_SERVER=smtp.gmail.com
SMTP_PORT=587
SMTP_LOGIN=your_email@gmail.com
SMTP_PASSWORD=your_app_password
SMTP_FROM_ADDRESS=Mastodon 
SMTP_AUTH_METHOD=plain
SMTP_OPENSSL_VERIFY_MODE=none
SMTP_ENABLE_STARTTLS=auto

Initial Setup#

Now that we have the configuration in place, let's set up the database and create an admin account.

Run in terminalbash

docker compose run --rm web bundle exec rake mastodon:setup

This interactive command will verify your configuration, set up the database, and prompt you to create an admin user.

When prompted:

Confirm your configuration settings
Choose to create an admin user when asked
Enter your desired username, email, and password for the admin account
For other prompts, accept the default values by pressing Enter

Starting Mastodon#

Now we're ready to start Mastodon:

Run in terminalbash

docker compose up -d

Starts all Mastodon services in detached mode. This may take a few minutes the first time.

Run in terminalbash

docker compose ps

Checks that all services are running. All containers should show "Up" status.

Setting Up Nginx as a Reverse Proxy#

To expose Mastodon securely to the internet, we'll set up Nginx as a reverse proxy with HTTPS.

Install Nginx and Certbot#

Run in terminalbash

sudo apt install -y nginx certbot python3-certbot-nginx

Installs Nginx web server and Certbot for obtaining SSL certificates.

Create Nginx Configuration#

Run in terminalbash

sudo nano /etc/nginx/sites-available/mastodon

Creates a new Nginx configuration file for Mastodon.

Paste the following configuration, replacing yourdomain.com with your actual domain:

File contentbash

server {
    listen 80;
    listen [::]:80;
    server_name yourdomain.com;
    
    location /.well-known/acme-challenge/ {
        root /var/www/html;
    }
    
    location / {
        return 301 https://$host$request_uri;
    }
}

Save and exit (Ctrl+X, then Y, then Enter).

Run in terminalbash

sudo ln -s /etc/nginx/sites-available/mastodon /etc/nginx/sites-enabled/

Enables the Nginx configuration by creating a symbolic link.

Run in terminalbash

sudo nginx -t

Tests the Nginx configuration for syntax errors.

Run in terminalbash

sudo systemctl reload nginx

Reloads Nginx to apply the new configuration.

Obtain SSL Certificate#

Run in terminalbash

sudo certbot --nginx -d yourdomain.com

Runs Certbot to obtain an SSL certificate for your domain. Follow the prompts to complete the process.

After Certbot completes, it will update your Nginx configuration with SSL settings. Verify that the configuration was updated correctly:

Run in terminalbash

sudo nano /etc/nginx/sites-available/mastodon

The file should now include SSL settings and a server block for HTTPS. If needed, adjust the configuration to include proper proxy settings:

File contentbash

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name yourdomain.com;
    
    # SSL settings added by Certbot
    
    # Mastodon proxy settings
    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header Proxy "";
        
        proxy_buffering off;
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        
        client_max_body_size 16m;
    }
    
    location /api/v1/streaming {
        proxy_pass http://127.0.0.1:4000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header Proxy "";
        
        proxy_buffering off;
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

Save and exit.

Run in terminalbash

sudo nginx -t

Tests the updated Nginx configuration.

Run in terminalbash

sudo systemctl reload nginx

Reloads Nginx to apply the changes.

Security Hardening#

Let's add additional security measures to protect your Mastodon instance.

Configure Docker Container Security#

Edit the docker-compose.yml file to improve container security:

Run in terminalbash

nano docker-compose.yml

Add the following security options to each service:

File contentbash

security_opt:
  - no-new-privileges:true

This prevents processes in the container from gaining additional privileges.

Set Up Fail2ban to Protect Against Brute Force Attacks#

Run in terminalbash

sudo apt install -y fail2ban

Installs Fail2ban, which helps protect against brute force login attempts.

Run in terminalbash

sudo nano /etc/fail2ban/jail.local

Creates a configuration file for Fail2ban.

Add the following content:

File contentbash

[sshd]
enabled = true
maxretry = 3
findtime = 1h
bantime = 24h

[nginx-http-auth]
enabled = true

Save and exit.

Run in terminalbash

sudo systemctl restart fail2ban

Restarts Fail2ban to apply the configuration.

Data Persistence and Backups#

Ensuring data persistence and regular backups is crucial for any production service.

Create a Backup Script#

Run in terminalbash

nano /opt/mastodon/backup.sh

Creates a backup script.

Add the following content:

File contentbash

#!/bin/bash
# Mastodon backup script
BACKUP_DIR="/opt/mastodon/backups"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="mastodon_backup_$DATE.tar.gz"

# Create backup directory if it doesn't exist
mkdir -p $BACKUP_DIR

# Go to Mastodon directory
cd /opt/mastodon

# Create database backup
docker compose exec -T db pg_dump -U postgres postgres > $BACKUP_DIR/mastodon_db_$DATE.sql

# Backup configuration files
cp .env.production $BACKUP_DIR/.env.production_$DATE

# Backup public uploads
tar -czf $BACKUP_DIR/$BACKUP_FILE public/system

# Keep only the last 7 backups
find $BACKUP_DIR -name "mastodon_backup_*" -type f -mtime +7 -delete
find $BACKUP_DIR -name "mastodon_db_*" -type f -mtime +7 -delete
find $BACKUP_DIR -name ".env.production_*" -type f -mtime +7 -delete

echo "Backup completed: $BACKUP_FILE"

Save and exit.

Run in terminalbash

chmod +x /opt/mastodon/backup.sh

Makes the backup script executable.

Set Up a Cron Job for Regular Backups#

Run in terminalbash

sudo crontab -e

Opens the root crontab file.

Add the following line to run backups daily at 2 AM:

File contentbash

0 2 * * * /opt/mastodon/backup.sh > /var/log/mastodon_backup.log 2>&1

Save and exit.

Common Issues and Troubleshooting#

Here are the most frequent issues users encounter and their solutions.

Checking Logs#

To check logs for troubleshooting:

Run in terminalbash

docker compose logs

Shows logs from all services.

Run in terminalbash

docker compose logs web

Shows logs from the web service only.

Run in terminalbash

docker compose logs -f

Shows logs and follows them in real-time (press Ctrl+C to exit).

Common Issues and Solutions#

Database Connection Issues#

Issue: Mastodon cannot connect to the database.

Diagnosis:

Run in terminalbash

docker compose logs web | grep "PG::ConnectionBad"

Solution: Check DB_HOST, DB_USER, and DB_PASS in .env.production. Ensure they match the configuration in docker-compose.yml.

Redis Connection Issues#

Issue: Mastodon cannot connect to Redis.

Diagnosis:

Run in terminalbash

docker compose logs web | grep "Redis::CannotConnectError"

Solution: Verify REDIS_HOST in .env.production is set to "redis".

Email Not Working#

Issue: Registration emails and notifications aren't being sent.

Diagnosis:

Run in terminalbash

docker compose logs sidekiq | grep "SMTP"

Solution: Check your SMTP configuration in .env.production. Some providers may require enabling less secure apps or generating app-specific passwords.

Insufficient Disk Space#

Issue: Docker operations fail due to low disk space.

Diagnosis:

Run in terminalbash

df -h

Solution: Clean unused Docker resources:

Run in terminalbash

docker system prune -a

Web Push Not Working#

Issue: Push notifications don't work.

Diagnosis: Check if VAPID keys are properly configured:

Run in terminalbash

grep VAPID .env.production

Solution: If keys are missing or invalid, regenerate them:

Run in terminalbash

docker compose run --rm web bundle exec rake mastodon:webpush:generate_vapid_key

Update the values in .env.production and restart Mastodon.

Maintenance Tasks#

Regular maintenance ensures your Mastodon instance stays healthy and secure.

Updating Mastodon#

Run in terminalbash

cd /opt/mastodon

Changes to the Mastodon directory.

Run in terminalbash

docker compose pull

Pulls the latest container images.

Run in terminalbash

docker compose down

Stops all containers.

Run in terminalbash

docker compose up -d

Starts containers with updated images.

Run in terminalbash

docker compose run --rm web rails db:migrate

Runs database migrations after updating.

Run in terminalbash

docker compose run --rm web rails assets:precompile

Recompiles static assets after updating.

Monitoring Server Health#

Run in terminalbash

htop

Monitors system resource usage in real-time. (Install with sudo apt install htop if not available)

Run in terminalbash

docker stats

Shows resource usage of Docker containers.

Run in terminalbash

df -h

Checks disk space usage.

User Management#

To create a new administrator account:

Run in terminalbash

docker compose run --rm web bin/tootctl accounts create username --email=email@example.com --confirmed --role=admin

Creates a new admin user with the specified username and email.

To moderate content:

Run in terminalbash

docker compose run --rm web bin/tootctl media remove

Removes remote media that's no longer referenced.

Known Issues and Workarounds#

High Memory Usage#

Issue: Mastodon services, especially Sidekiq, can use a lot of memory on low-spec servers.

Workaround: Adjust the concurrency in docker-compose.yml for the Sidekiq service:

Run in terminalbash

nano docker-compose.yml

Find the Sidekiq section and add an environment variable:

File contentbash

environment:
  - SIDEKIQ_CONCURRENCY=5

Save and restart:

Run in terminalbash

docker compose restart sidekiq

Slow Initial Load Times#

Issue: Initial page loads can be slow.

Workaround: Enable caching by adding these to .env.production:

File contentbash

CACHE_REDIS_HOST=redis
CACHE_REDIS_PORT=6379
CACHE_REDIS_DB=0

File Upload Size Limitations#

Issue: Users cannot upload large files.

Workaround: Adjust Nginx client_max_body_size:

Run in terminalbash

sudo nano /etc/nginx/sites-available/mastodon

Find and modify the client_max_body_size directive:

File contentbash

client_max_body_size 40m;

Save and reload Nginx:

Run in terminalbash

sudo systemctl reload nginx

Conclusion#

Congratulations! You now have a fully functioning Mastodon instance running on Docker with security best practices in place. This setup provides a solid foundation for running your own social media server while maintaining control over your data and user experience.

Remember to:

Regularly update your system and Mastodon
Monitor server resources
Keep backups of your data
Stay informed about security advisories

By following these practices, you'll ensure a stable, secure, and reliable Mastodon experience for you and your users.

The Complete Guide to Installing and Running Ghost CMS with Docker on Ubuntu 20.04 and Newer

Sun, 09 Mar 2025 06:52:06 GMT

Ghost is an open-source, professional publishing platform designed to create blogs, magazines, and news sites. Originally launched in 2013 as a streamlined alternative to WordPress, Ghost focuses on content creation with a clean, distraction-free interface. This comprehensive guide walks you through installing and securing Ghost CMS using Docker on Ubuntu 20.04 or newer versions. Docker provides an isolated, consistent environment that makes deployment straightforward and maintenance simpler. By containerizing Ghost, you'll gain the benefits of easy updates, consistent environments, and simplified backups while maintaining robust security practices.

Understanding Ghost CMS and Docker#

Ghost CMS stands out as a modern, JavaScript-based publishing platform powered by Node.js. It offers a markdown-focused editor, built-in SEO tools, and native subscription capabilities, making it ideal for professional bloggers, journalists, and content creators. Ghost emphasizes performance and simplicity, providing all essential publishing features without the complexity found in some other content management systems.

Docker revolutionizes application deployment by packaging software and dependencies into standardized containers. This approach eliminates environment inconsistencies and simplifies management. For Ghost specifically, Docker eliminates the need to manually configure Node.js, database systems, and other dependencies. You can deploy, update, and migrate your Ghost blog with minimal technical overhead.

When combined, Ghost and Docker create a powerful, flexible publishing platform that's easy to maintain and scale. This guide uses Docker Compose V2 (the docker compose command) rather than the deprecated Docker Compose V1 (docker-compose), ensuring you're following current best practices as of March 2025.

Instruction layout

Prerequisites and System Requirements#

Before beginning the installation process, ensure your system meets the following requirements:

Hardware Requirements#

CPU: At least 1 dedicated core (2+ cores recommended for production)
RAM: Minimum 1GB (2GB or more recommended for production use)
Storage: At least 10GB free space for Docker images, database, and content

Software Requirements#

Ubuntu 20.04 LTS or newer (22.04 LTS or 24.04 LTS)
A non-root user with sudo privileges
Updated system packages

Network Requirements#

A domain name pointing to your server (for production use)
Ability to open ports 80 and 443 for HTTP/HTTPS traffic
Unrestricted outbound internet access to pull Docker images

Required Knowledge#

Basic familiarity with terminal commands
Understanding of web hosting concepts
Ability to follow step-by-step instructions

Initial System Setup#

Start by ensuring your Ubuntu system is fully updated. This prevents package conflicts and security vulnerabilities during installation.

Run in terminalbash

sudo apt update

This command refreshes your package lists, ensuring you have access to the latest software versions. If this fails, check your internet connection or try changing your Ubuntu mirror.

Run in terminalbash

sudo apt upgrade -y

This upgrades all installed packages to their latest versions. The -y flag automatically answers "yes" to the upgrade prompt. This process may take several minutes depending on your internet speed and how many packages need updating.

Next, install essential tools that will be needed throughout this process:

Run in terminalbash

sudo apt install -y ca-certificates curl gnupg lsb-release

This installs certificate authority certificates for secure connections, curl for downloading files, gnupg for verifying package signatures, and lsb-release for detecting your Ubuntu version. These are prerequisites for Docker installation.

Configuring UFW Firewall#

Ubuntu comes with Uncomplicated Firewall (UFW), which provides a user-friendly interface for managing firewall rules. Start by checking if the firewall is active:

Run in terminalbash

sudo ufw status

If the firewall is inactive, you'll see "Status: inactive". If it's already active, you'll see a list of allowed connections.

Allow SSH connections to prevent locking yourself out when enabling the firewall:

Run in terminalbash

sudo ufw allow OpenSSH

This ensures you maintain SSH access after enabling the firewall. Without this rule, you might lose connection to your server and require console access to regain control.

Next, allow HTTP and HTTPS traffic, which will be necessary for accessing your Ghost blog:

Run in terminalbash

sudo ufw allow 80

This opens port 80 for HTTP traffic. This port is necessary for initial Let's Encrypt certificate verification and for redirecting users to the secure HTTPS version of your site.

Run in terminalbash

sudo ufw allow 443

This opens port 443 for HTTPS traffic, which will be used for encrypted connections to your Ghost blog. All production traffic will go through this port.

Now, enable the firewall:

Run in terminalbash

sudo ufw enable

You'll receive a warning that enabling the firewall might disrupt existing SSH connections. Type 'y' and press Enter to continue. Since you've allowed OpenSSH, your current connection shouldn't be affected.

Verify that the firewall is properly configured:

Run in terminalbash

sudo ufw status

This should display a list of allowed services and ports, including OpenSSH, port 80, and port 443. If any are missing, add them using the sudo ufw allow command.

Installing Docker and Docker Compose#

Docker will manage the containers that run your Ghost CMS and supporting services. First, uninstall any older versions of Docker that might cause conflicts:

Run in terminalbash

sudo apt remove docker docker-engine docker.io containerd runc

This removes any previously installed Docker packages. If none are installed, you'll see a message that these packages weren't found, which is fine.

Create a directory for Docker's GPG key:

Run in terminalbash

sudo mkdir -p /etc/apt/keyrings

This creates a directory to store repository keys. If the directory already exists, this command will have no effect.

Download and add Docker's official GPG key:

Run in terminalbash

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

This downloads Docker's verification key and adds it to your system, enabling you to securely download Docker packages. If you encounter a GPG error, try running sudo apt install gnupg first.

Add the Docker repository to your system:

Run in terminalbash

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

This adds the appropriate Docker repository for your Ubuntu version and system architecture. If this command fails, check if the lsb-release package is installed.

Update apt package list to include the Docker repository:

Run in terminalbash

sudo apt update

This refreshes your package lists to include Docker packages from the newly added repository.

Install Docker Engine, containerd, and Docker Compose:

Run in terminalbash

sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

This installs Docker Engine, Docker CLI tools, containerd runtime, and the Docker Compose plugin. If this fails with dependency errors, run sudo apt --fix-broken install and try again.

Verify that Docker is properly installed:

Run in terminalbash

docker --version

This should display the installed Docker version. If you get a "command not found" error, try logging out and back in or rebooting your system.

Verify that Docker Compose V2 is installed:

Run in terminalbash

docker compose version

This should show the installed Docker Compose version. Note that we're using docker compose (with a space), which is the current Compose V2 syntax, not the older docker-compose command.

Add your user to the Docker group to avoid needing sudo for Docker commands:

Run in terminalbash

sudo usermod -aG docker $USER

This adds your current user to the Docker group, allowing you to run Docker commands without sudo. You'll need to log out and back in for this change to take effect.

Verify Docker is working properly by running a test container:

Run in terminalbash

docker run --rm hello-world

This runs a simple test container that confirms Docker is installed and functioning correctly. If successful, you'll see a message that your installation appears to be working correctly.

Installing Certbot for SSL Certificates#

Secure your Ghost blog with free SSL certificates from Let's Encrypt using Certbot:

Run in terminalbash

sudo snap install core

This ensures the core snap is installed and up-to-date. Snap is Ubuntu's package management system that comes pre-installed on Ubuntu 20.04 and newer.

Run in terminalbash

sudo snap refresh core

This updates the core snap package to the latest version. This step helps avoid conflicts with Certbot installation.

Remove any previously installed Certbot packages:

Run in terminalbash

sudo apt remove certbot

This removes any previous Certbot installations that might conflict with the snap version. If Certbot wasn't previously installed, this command has no effect.

Install Certbot using Snap:

Run in terminalbash

sudo snap install --classic certbot

This installs the latest version of Certbot. The --classic flag gives Certbot the permissions it needs to manage your certificates.

Create a symbolic link to make Certbot easily accessible:

Run in terminalbash

sudo ln -s /snap/bin/certbot /usr/bin/certbot

This makes the Certbot command available system-wide. If this command fails, the link may already exist.

Setting Up Ghost with Docker Compose#

Now you're ready to set up Ghost using Docker Compose. Create a directory for your Ghost installation:

Run in terminalbash

mkdir -p ~/ghost-blog

This creates a directory to store your Ghost configuration and content. The -p flag ensures parent directories are created if they don't exist.

Change to the new directory:

Run in terminalbash

cd ~/ghost-blog

Create a Docker Compose configuration file:

Run in terminalbash

nano docker-compose.yml

This opens the nano text editor to create a new Docker Compose file. If you don't have nano, you can install it with sudo apt install nano or use any other text editor.

Add the following configuration, replacing yourdomain.com with your actual domain name and using strong, unique passwords:

File contentbash

version: '3.1'

services:
  ghost:
    image: ghost:latest
    restart: always
    environment:
      url: https://yourdomain.com
      database__client: mysql
      database__connection__host: db
      database__connection__user: ghost
      database__connection__password: YourStrongPasswordHere
      database__connection__database: ghost
      # Set Node.js to production mode
      NODE_ENV: production
    volumes:
      - ./ghost-content:/var/lib/ghost/content
    depends_on:
      - db
    networks:
      - ghost_network

  db:
    image: mysql:8.0
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: YourMySQLRootPasswordHere
      MYSQL_DATABASE: ghost
      MYSQL_USER: ghost
      MYSQL_PASSWORD: YourStrongPasswordHere
    volumes:
      - ./mysql-data:/var/lib/mysql
    networks:
      - ghost_network

networks:
  ghost_network:
    driver: bridge

Save the file by pressing Ctrl+X, then Y, then Enter. This configuration creates two containers: one for Ghost and one for MySQL, with persistent storage for both.

Create directories for persistent storage:

Run in terminalbash

mkdir -p ghost-content mysql-data

This creates directories that will be mounted as volumes to store Ghost content and MySQL data. If these directories exist, this command has no effect.

Set appropriate permissions for these directories:

Run in terminalbash

sudo chown -R 1000:1000 ghost-content

This sets the correct ownership for the Ghost content directory. Ghost runs as user ID 1000 inside the container, so this ensures proper file access.

Start the Ghost application:

Run in terminalbash

docker compose up -d

This starts the Ghost and MySQL containers in detached mode. The -d flag runs containers in the background. The initial startup might take a few minutes as Docker downloads the required images.

Check if the containers are running:

Run in terminalbash

docker compose ps

This shows the status of your containers. Both should show "Up" under the Status column. If any container shows an error or keeps restarting, check the logs with docker compose logs.

Obtaining SSL Certificates#

Before configuring your web server, you need to obtain SSL certificates. Ensure your domain is pointed to your server's IP address before continuing.

Temporarily stop any running web servers to free port 80:

Run in terminalbash

sudo systemctl stop nginx

This stops the Nginx service if it's running. If you don't have Nginx installed yet, this command will show an error which you can ignore.

Obtain SSL certificates for your domain:

Run in terminalbash

sudo certbot certonly --standalone -d yourdomain.com -d www.yourdomain.com

Replace yourdomain.com with your actual domain. This command runs a temporary web server to verify your domain ownership and obtain certificates. You'll be prompted to enter your email address and agree to the terms of service.

Verify that certificates were successfully obtained:

Run in terminalbash

sudo ls -la /etc/letsencrypt/live/yourdomain.com/

This lists the certificate files. You should see files including fullchain.pem and privkey.pem. If these files don't exist, the certificate acquisition failed, and you should check Certbot's output for error messages.

Installing and Configuring Nginx#

Nginx will act as a reverse proxy for your Ghost blog, handling SSL termination and serving static content efficiently:

Run in terminalbash

sudo apt install -y nginx

This installs the Nginx web server. If the installation fails, try running sudo apt update first and ensure no other process is using port 80.

Create an Nginx configuration file for your Ghost blog:

Run in terminalbash

sudo nano /etc/nginx/sites-available/yourdomain.com

Replace yourdomain.com with your actual domain name. This creates a new Nginx configuration file.

Add the following configuration, replacing yourdomain.com with your actual domain:

File contentbash

server {
    listen 80;
    listen [::]:80;
    server_name yourdomain.com www.yourdomain.com;
    
    # Redirect HTTP to HTTPS
    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name yourdomain.com www.yourdomain.com;
    
    # SSL configuration
    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:10m;
    
    # Security headers
    add_header X-Frame-Options SAMEORIGIN always;
    add_header X-Content-Type-Options nosniff always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;
    
    # Ghost Reverse Proxy
    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://localhost:2368;
        proxy_buffering off;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
    
    # Increase max body size for media uploads
    client_max_body_size 50M;
    
    # Optimize content caching
    location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
        proxy_pass http://localhost:2368;
        expires 30d;
        add_header Cache-Control "public, no-transform";
    }
}

Save the file by pressing Ctrl+X, then Y, then Enter. This configuration redirects HTTP traffic to HTTPS and sets up a reverse proxy to your Ghost container, along with security headers and content caching.

Link the configuration to the enabled sites:

Run in terminalbash

sudo ln -s /etc/nginx/sites-available/yourdomain.com /etc/nginx/sites-enabled/

This activates your Nginx configuration. If this command fails, the symbolic link may already exist.

Test the Nginx configuration for syntax errors:

Run in terminalbash

sudo nginx -t

This checks your Nginx configuration for errors. If you see "syntax is ok" and "test is successful", proceed to the next step. If there are errors, check your configuration file for typos.

Restart Nginx to apply the configuration:

Run in terminalbash

sudo systemctl restart nginx

This applies your new Nginx configuration. If Nginx fails to restart, check the error logs with sudo journalctl -xe and verify your SSL certificate paths.

Check if Nginx is running:

Run in terminalbash

sudo systemctl status nginx

This shows the status of the Nginx service. It should show "active (running)". If not, check the error output for clues about what went wrong.

Setting Up Automatic Certificate Renewal#

Let's Encrypt certificates expire after 90 days, so automatic renewal is essential:

Run in terminalbash

sudo systemctl status snap.certbot.renew.service

This checks if the automatic renewal service is active. You should see "active (running)" or information about the next scheduled run.

Test the renewal process to ensure it works correctly:

Run in terminalbash

sudo certbot renew --dry-run

This simulates the renewal process without actually replacing certificates. You should see a message indicating that the renewal simulation succeeded. If there are errors, they might relate to Nginx configuration or certificate ownership.

Configuring Ghost for Production#

Now that Ghost is running behind Nginx with SSL, you need to update some Ghost settings for production use:

Stop the Ghost container:

Run in terminalbash

docker compose down

This stops all containers defined in your docker-compose.yml file. Your content and database data remain safe in the persistent volumes.

Edit the Docker Compose file to add email configuration:

Run in terminalbash

nano docker-compose.yml

Add email configuration to the ghost service environment section:

File contentbash

environment:
  # Existing environment variables...
  mail__transport: SMTP
  mail__options__service: Mailgun # or another provider like SendGrid, Postmark, etc.
  mail__options__host: smtp.mailgun.org
  mail__options__port: 587
  mail__options__auth__user: your-smtp-username
  mail__options__auth__pass: your-smtp-password

Replace the SMTP details with your own email service credentials. This configures Ghost to send emails through your SMTP provider, which is essential for user invitations, password resets, and newsletters.

Save the file and restart the containers:

Run in terminalbash

docker compose up -d

This starts your containers with the updated configuration. Check the Ghost logs for any errors related to the new configuration:

Run in terminalbash

docker compose logs ghost

This shows the logs from the Ghost container. Look for any error messages related to the SMTP configuration or other issues.

Creating a Ghost Admin User#

Access your Ghost blog at https://yourdomain.com/ghost to set up your admin account:

Open a web browser and navigate to https://yourdomain.com/ghost
Follow the on-screen instructions to create your admin account
Set a strong, unique password for your admin account

Creating Backup and Restore Scripts#

Create a backup script to regularly save your Ghost data:

Run in terminalbash

nano backup-ghost.sh

Add the following content to the script:

File contentbash

#!/bin/bash

# Create backup directory if it doesn't exist
BACKUP_DIR="./backups"
mkdir -p $BACKUP_DIR

# Set timestamp for backup files
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")

# Backup Ghost content
echo "Backing up Ghost content..."
tar -czf $BACKUP_DIR/ghost_content_$TIMESTAMP.tar.gz -C ./ghost-content .

# Backup MySQL database
echo "Backing up MySQL database..."
docker compose exec db sh -c 'exec mysqldump -uroot -p"$MYSQL_ROOT_PASSWORD" ghost' > $BACKUP_DIR/ghost_db_$TIMESTAMP.sql

# Compress database backup
gzip $BACKUP_DIR/ghost_db_$TIMESTAMP.sql

echo "Backup completed: $BACKUP_DIR/ghost_content_$TIMESTAMP.tar.gz and $BACKUP_DIR/ghost_db_$TIMESTAMP.sql.gz"

# Optional: Remove backups older than 30 days
find $BACKUP_DIR -name "ghost_*" -type f -mtime +30 -delete

Save the file and make it executable:

Run in terminalbash

chmod +x backup-ghost.sh

This allows the script to be executed. You can now run it with ./backup-ghost.sh to create a backup.

Create a restore script:

Run in terminalbash

nano restore-ghost.sh

Add the following content to the script:

File contentbash

#!/bin/bash

if [ $# -ne 2 ]; then
  echo "Usage: $0 content_backup.tar.gz database_backup.sql.gz"
  exit 1
fi

CONTENT_BACKUP=$1
DB_BACKUP=$2

# Stop containers
docker compose down

# Restore content
echo "Restoring content from $CONTENT_BACKUP..."
rm -rf ./ghost-content/*
tar -xzf $CONTENT_BACKUP -C ./ghost-content

# Restore database
echo "Restoring database from $DB_BACKUP..."
gunzip -c $DB_BACKUP > ./restore_db.sql
docker compose up -d db
sleep 10  # Wait for database to start
docker compose exec db sh -c 'exec mysql -uroot -p"$MYSQL_ROOT_PASSWORD" ghost' < ./restore_db.sql

# Start Ghost
docker compose up -d

echo "Restoration completed. Check your Ghost blog to verify."

Make the restoration script executable:

Run in terminalbash

chmod +x restore-ghost.sh

Set up automated backups with cron:

Run in terminalbash

crontab -e

Add the following line to run backups daily at 2:00 AM:

File contentbash

0 2 * * * /home/username/ghost-blog/backup-ghost.sh >> /home/username/ghost-blog/backups/backup.log 2>&1

Replace /home/username/ghost-blog/ with the actual path to your Ghost directory.

Enhancing Security#

Improve your Ghost installation's security with these additional steps:

Secure MySQL by limiting network access:

Edit your docker-compose.yml file:

Run in terminalbash

nano docker-compose.yml

Add the following to the db service to make it only accept connections from the internal network:

File contentbash

db:
  # Existing configuration...
  command: --default-authentication-plugin=mysql_native_password --bind-address=0.0.0.0 --mysqlx=0

This disables the MySQL X Protocol and configures the authentication plugin. Save the file and restart the containers:

Run in terminalbash

docker compose down
docker compose up -d

Set up container resource limits to prevent resource exhaustion:

Edit your docker-compose.yml file again:

Run in terminalbash

nano docker-compose.yml

Add resource limits to both services:

File content

services:
  ghost:
    # Existing configuration...
    deploy:
      resources:
        limits:
          memory: 1G
        reservations:
          memory: 512M
  
  db:
    # Existing configuration...
    deploy:
      resources:
        limits:
          memory: 1G
        reservations:
          memory: 256M

This prevents containers from consuming excessive resources, which could lead to system instability. Save the file and restart the containers:

Run in terminalbash

docker compose down
docker compose up -d

Maintenance Procedures#

Regularly update your Ghost installation to receive security patches and new features:

Check for new Ghost versions:

Run in terminalbash

docker pull ghost:latest

This pulls the latest Ghost image. The output will indicate if a newer version is available.

Create a backup before updating:

Run in terminalbash

./backup-ghost.sh

This creates a backup of your content and database before updating.

Update Ghost and other containers:

Run in terminalbash

docker compose pull
docker compose up -d

This pulls the latest versions of all images in your docker-compose.yml file and recreates the containers.

Monitor container logs for any issues after updating:

Run in terminalbash

docker compose logs -f

This shows real-time logs from all containers. Press Ctrl+C to exit the log view.

Troubleshooting Common Issues#

Ghost Container Won't Start#

Check the container logs:

Run in terminalbash

docker compose logs ghost

Look for error messages that might indicate the problem. Common issues include:

Database connection problems: Verify MySQL is running and credentials are correct
Port conflicts: Make sure nothing else is using port 2368
File permission issues: Run sudo chown -R 1000:1000 ghost-content

Database Container Won't Start#

Check the database container logs:

Run in terminalbash

docker compose logs db

Common MySQL issues include:

Data directory permission problems: Run sudo chown -R 999:999 mysql-data
Corrupted data files: Restore from a backup or initialize with a fresh database
Insufficient system resources: Check your system's available memory and disk space

Nginx Shows 502 Bad Gateway#

This usually means Nginx can't connect to Ghost. Verify Ghost is running:

Run in terminalbash

docker compose ps

Check if Ghost is accessible locally:

Run in terminalbash

curl http://localhost:2368

This tests if Ghost is responding on its default port. If this works but Nginx shows 502, check your Nginx configuration.

Email Delivery Problems#

Test email configuration:

Run in terminalbash

docker compose exec ghost curl -X POST "http://localhost:2368/ghost/api/admin/mail/test/" \
  -H "Origin: http://localhost:2368" \
  -H "Content-Type: application/json" \
  -d '{"mail":{"to":"your-email@example.com"}}'

Replace your-email@example.com with your actual email address. Check Ghost logs for SMTP errors:

Run in terminalbash

docker compose logs ghost | grep -i mail

This searches logs for mail-related entries. Verify your SMTP credentials and server settings.

SSL Certificate Issues#

If your SSL certificate isn't working, check the certificate files:

Run in terminalbash

sudo ls -la /etc/letsencrypt/live/yourdomain.com/

Test certificate renewal:

Run in terminalbash

sudo certbot renew --dry-run

Verify Nginx is using the correct certificate paths in your configuration file.

Known Limitations and Workarounds#

Memory Usage#

Ghost can consume significant memory, especially with many images or a large database. If you experience performance issues:

Increase the container memory limit in docker-compose.yml
Consider using a swapfile for additional virtual memory:

Run in terminalbash

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

To make the swap permanent, add it to fstab:

Run in terminalbash

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Upload Limits#

If you can't upload large files to Ghost, check your Nginx client_max_body_size setting:

Run in terminalbash

sudo nano /etc/nginx/sites-available/yourdomain.com

Increase the value if needed:

File contentbash

client_max_body_size 100M;

Save and restart Nginx:

Run in terminalbash

sudo nginx -t
sudo systemctl restart nginx

Conclusion#

You now have a fully functional Ghost CMS installation running on Docker with proper security measures in place. Regular maintenance, backups, and security updates will ensure your Ghost blog remains reliable and secure. By following this guide, you've created a robust publishing platform that's easy to manage and scale as your needs grow.

Remember to:

Regularly update Ghost and other components
Monitor system resource usage
Maintain regular backups
Check for security advisories related to Ghost, Docker, and Ubuntu

With proper maintenance, your Ghost blog will provide a stable and secure platform for your content for years to come.

Complete Guide to Installing and Self-Hosting Bitwarden on Ubuntu 20.04+ with Docker

Sun, 09 Mar 2025 06:22:22 GMT

Self-hosting your password manager gives you complete control over your sensitive credentials. This comprehensive guide walks you through setting up Bitwarden on Ubuntu using Docker, with security best practices, troubleshooting tips, and maintenance procedures to ensure a robust deployment.

Introduction: Why Self-Host Bitwarden?#

Bitwarden is an open-source password manager that securely stores your sensitive information in an encrypted vault. Self-hosting Bitwarden provides several key advantages:

Complete control over your password data
No reliance on third-party servers for storing sensitive information
Enhanced privacy since you manage where your data resides
Customizable configuration based on your specific needs

Using Docker for deployment simplifies the installation and maintenance processes by containerizing the application and its dependencies, making the entire setup more portable and easier to manage.

instruction layout

Prerequisites#

Before beginning the installation, ensure your system meets the following requirements:

Ubuntu 20.04 LTS or newer (this guide works for Ubuntu 22.04 as well)
A server with at least 2GB of RAM
Minimum 10GB free disk space
Internet connection for downloading packages
Root or sudo privileges
A domain name pointing to your server (for secure access)
Open ports 80 and 443 on your firewall

System Preparation#

First, update your system to ensure you have the latest packages:

Run in terminalbash

sudo apt update && sudo apt upgrade -y

This step minimizes compatibility issues and security vulnerabilities by bringing your system up to date.

Next, install the necessary dependencies for Docker installation:

Run in terminalbash

sudo apt install apt-transport-https ca-certificates curl gnupg-agent software-properties-common -y

These packages enable secure APT repositories, certificate verification, and other requirements for Docker.

If any package fails to install, you might see error messages. In that case, try running the update command again to fix any potential repository issues.

Creating a Dedicated Bitwarden User#

For security reasons, it's best to create a dedicated user for running Bitwarden. This isolates the Bitwarden installation from other applications and follows the principle of least privilege.

Create a directory for Bitwarden:

Run in terminalbash

sudo mkdir -p /opt/bitwarden

Create a dedicated user named 'bitwarden' with the home directory at /opt/bitwarden:

Run in terminalbash

sudo useradd -s /bin/bash -d /opt/bitwarden bitwarden

Set proper directory permissions:

Run in terminalbash

sudo chmod -R 700 /opt/bitwarden
sudo chown -R bitwarden:bitwarden /opt/bitwarden

These permissions ensure that only the bitwarden user can access the installation directory.

Installing Docker and Docker Compose#

Docker is required to run Bitwarden containers. Let's add the official Docker repository and install Docker.

Add Docker's official GPG key:

Run in terminalbash

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Add the Docker repository:

Run in terminalbash

echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Update package information:

Run in terminalbash

sudo apt update

Install Docker:

Run in terminalbash

sudo apt install docker-ce docker-ce-cli containerd.io -y

Verify Docker installation:

Run in terminalbash

sudo systemctl status docker

You should see "active (running)" in the output. If Docker isn't running, start it with:

Run in terminalbash

sudo systemctl start docker && sudo systemctl enable docker

This command both starts Docker and ensures it launches automatically on system boot.

Allow the bitwarden user to run Docker commands:

Run in terminalbash

sudo usermod -aG docker bitwarden

Adding the bitwarden user to the docker group allows it to execute Docker commands without sudo.

Installing Bitwarden#

Now we're ready to install Bitwarden. Switch to the bitwarden user and navigate to its home directory:

Run in terminalbash

sudo su - bitwarden
cd /opt/bitwarden

Download the official Bitwarden installation script:

Run in terminalbash

curl -Lso bitwarden.sh https://go.btwrdn.co/bw-sh

Make the script executable:

Run in terminalbash

chmod +x bitwarden.sh

Before running the installation, you need to obtain a Bitwarden installation ID and key from Bitwarden's website. Visit https://bitwarden.com/host/ and complete the form to receive your installation ID and key by email.

Run the installation script:

Run in terminalbash

./bitwarden.sh install

The installation wizard will prompt you for several pieces of information:

Your domain name (e.g., bitwarden.yourdomain.com)
Whether to use Let's Encrypt for SSL (recommended for production)
Your email address (for Let's Encrypt certificates)
Your installation ID
Your installation key

If any prompt fails to appear or the installation stops unexpectedly, check for error messages in the terminal output. Common issues include network connectivity problems or insufficient disk space.

Starting Bitwarden Server#

After installation completes, start the Bitwarden server:

Run in terminalbash

./bitwarden.sh start

Verify all containers are running:

Run in terminalbash

docker ps

You should see multiple Bitwarden containers running. If any container is missing or shows a status other than "Up", check the logs:

Run in terminalbash

docker logs bitwarden-mssql

Replace "bitwarden-mssql" with the name of the problematic container.

Configuring SMTP for Email#

Email functionality is important for account recovery and notifications. Edit the global environment file:

Run in terminalbash

nano /opt/bitwarden/bwdata/env/global.override.env

Configure the email settings by adding or modifying these lines:

File contentbash

globalSettings__mail__replyToEmail=your_email@domain.com
globalSettings__mail__smtp__host=smtp.your-provider.com
globalSettings__mail__smtp__port=587
globalSettings__mail__smtp__ssl=true
globalSettings__mail__smtp__username=your_email@domain.com
globalSettings__mail__smtp__password=your_password
globalSettings__mail__smtp__useDefaultCredentials=false

After modifying the configuration, restart Bitwarden:

Run in terminalbash

./bitwarden.sh restart

If email sending fails, check your SMTP settings and ensure your email provider allows SMTP access from your server's IP address.

Security Considerations#

Securing your Bitwarden instance is crucial since it contains sensitive information. Here are essential security measures:

Restrict Admin Access#

Edit the global environment file:

Run in terminalbash

nano /opt/bitwarden/bwdata/env/global.override.env

Add the following line to disable new user registrations (recommended for personal or small team use):

File contentbash

globalSettings__disableUserRegistration=true

Restart Bitwarden to apply changes:

Run in terminalbash

./bitwarden.sh restart

Set Up Proper Firewall Rules#

Install and configure UFW (Uncomplicated Firewall):

Run in terminalbash

sudo apt install ufw -y
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

Verify the firewall status:

Run in terminalbash

sudo ufw status

If your SSH port is not the default (22), replace "ssh" with your specific port.

Regular Updates#

Keep your Bitwarden installation updated:

Run in terminalbash

./bitwarden.sh updateself
./bitwarden.sh update

These commands update both the installation script itself and the Bitwarden application.

Backup and Restoration#

Regular backups are essential for data safety. Bitwarden automatically takes nightly backups of the database, but a comprehensive backup strategy includes more elements.

Setting Up Manual Backups#

Create a backup script:

Run in terminalbash

nano /opt/bitwarden/backup.sh

Add the following content:

File contentbash

#!/bin/bash
# Create timestamp
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
BACKUP_DIR="/opt/bitwarden/backups/$TIMESTAMP"

# Create backup directory
mkdir -p $BACKUP_DIR

# Stop Bitwarden to ensure data consistency
/opt/bitwarden/bitwarden.sh stop

# Backup important directories
cp -r /opt/bitwarden/bwdata/env $BACKUP_DIR/
cp -r /opt/bitwarden/bwdata/core/attachments $BACKUP_DIR/
cp -r /opt/bitwarden/bwdata/mssql/data $BACKUP_DIR/
cp -r /opt/bitwarden/bwdata/core/aspnet-dataprotection $BACKUP_DIR/
cp -r /opt/bitwarden/bwdata/ssl $BACKUP_DIR/

# Restart Bitwarden
/opt/bitwarden/bitwarden.sh start

# Compress backup
tar -czf /opt/bitwarden/backups/bitwarden_backup_$TIMESTAMP.tar.gz -C /opt/bitwarden/backups $TIMESTAMP

# Remove uncompressed backup directory
rm -rf $BACKUP_DIR

echo "Backup completed: /opt/bitwarden/backups/bitwarden_backup_$TIMESTAMP.tar.gz"

Make the script executable:

Run in terminalbash

chmod +x /opt/bitwarden/backup.sh

Create a backup directory:

Run in terminalbash

mkdir -p /opt/bitwarden/backups

To run a manual backup:

Codebash

/opt/bitwarden/backup.sh

For automated backups, set up a cron job. Open the crontab:

Run in terminalbash

crontab -e

Add this line to run the backup weekly at 2 AM on Sunday:

File contentbash

0 2 * * 0 /opt/bitwarden/backup.sh

Restoring from Backup#

To restore from backup:

Run in terminalbash

# Extract backup
tar -xzf /opt/bitwarden/backups/bitwarden_backup_TIMESTAMP.tar.gz -C /opt/bitwarden/backups/

# Stop Bitwarden
/opt/bitwarden/bitwarden.sh stop

# Restore directories
cp -r /opt/bitwarden/backups/TIMESTAMP/env /opt/bitwarden/bwdata/
cp -r /opt/bitwarden/backups/TIMESTAMP/attachments /opt/bitwarden/bwdata/core/
cp -r /opt/bitwarden/backups/TIMESTAMP/data /opt/bitwarden/bwdata/mssql/
cp -r /opt/bitwarden/backups/TIMESTAMP/aspnet-dataprotection /opt/bitwarden/bwdata/core/
cp -r /opt/bitwarden/backups/TIMESTAMP/ssl /opt/bitwarden/bwdata/

# Fix permissions
chown -R bitwarden:bitwarden /opt/bitwarden/bwdata

# Start Bitwarden
/opt/bitwarden/bitwarden.sh start

Replace "TIMESTAMP" with your actual backup timestamp.

Common Problems and Solutions#

Here are some common issues users encounter and their solutions:

Container Startup Failures#

If containers fail to start:

Run in terminalbash

# Check container status
docker ps -a

# View logs of specific container
docker logs bitwarden-mssql

Solution: If the error relates to permissions, ensure proper ownership:

Run in terminalbash

sudo chown -R bitwarden:bitwarden /opt/bitwarden/bwdata

Database Connection Issues#

If you experience database connectivity problems:

Run in terminalbash

# Check database container is running
docker ps | grep mssql

# View database logs
docker logs bitwarden-mssql

Solution: You may need to recreate the database container:

Run in terminalbash

./bitwarden.sh rebuild mssql

SSL Certificate Problems#

If SSL certificates aren't working properly:

Run in terminalbash

# Check certificate status
ls -la /opt/bitwarden/bwdata/ssl

Solution: Reinstall certificates:

Run in terminalbash

./bitwarden.sh updateself
./bitwarden.sh update
./bitwarden.sh rebuild nginx

Email Configuration Issues#

If email notifications aren't working:

Run in terminalbash

# Check mail settings
grep -A 10 "mail" /opt/bitwarden/bwdata/env/global.override.env

Solution: Verify your SMTP settings and test with a simple telnet command:

Run in terminalbash

telnet your-smtp-server 587

If telnet isn't installed:

Run in terminalbash

sudo apt install telnet -y

"Read-only File System" Error#

Some users encounter "mkdir /bw-data: read-only file system" errors:

Run in terminalbash

# Check filesystem status
df -h

# Check for errors
dmesg | grep -i error

Solution: This often indicates filesystem corruption. Try remounting the filesystem:

Run in terminalbash

sudo mount -o remount,rw /

If the issue persists, consider changing the data path in your Docker configuration.

Maintenance Tasks#

Regular maintenance ensures your Bitwarden instance remains secure and performs optimally.

Updating Bitwarden#

Update the Bitwarden installation script:

Run in terminal

./bitwarden.sh updateself

Update Bitwarden:

Run in terminal

./bitwarden.sh update

Rebuild and restart:

Run in terminal

./bitwarden.sh rebuild
./bitwarden.sh restart

Monitoring Disk Space#

Monitor disk space regularly:

Run in terminal

df -h

If space is running low, clean Docker:

Run in terminal

docker system prune -a --volumes

This command removes unused containers, networks, images, and volumes. Be careful, as it can delete useful data.

Log Rotation#

Set up log rotation to prevent logs from consuming excessive disk space:

Run in terminalbash

sudo nano /etc/logrotate.d/bitwarden

Add the following:

File contentbash

/opt/bitwarden/bwdata/logs/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 bitwarden bitwarden
}

Known Issues and Workarounds#

Memory Usage with MSSQL#

The MSSQL container can use significant memory, which might be problematic on smaller servers.

Workaround: Adjust MSSQL memory settings:

Run in terminalbash

nano /opt/bitwarden/bwdata/env/mssql.override.env

Add:

File contentbash

MSSQL_MEMORY_LIMIT_MB=1024

Restart:

Run in terminalbash

./bitwarden.sh restart

Web Vault Connectivity Issues#

Users sometimes encounter "Error fetching network" messages when accessing the web vault.

Workaround: Clear browser cache or try a different browser. If that doesn't help, rebuild the web container:

Run in terminalbash

./bitwarden.sh rebuild web

Database Corruption#

In rare cases, the database may become corrupted.

Workaround: Restore from the latest automatic backup:

Run in terminalbash

# Stop Bitwarden
./bitwarden.sh stop

# Find the latest backup
ls -la /opt/bitwarden/bwdata/mssql/backups

# Restore database (replace DATE with actual date)
cp /opt/bitwarden/bwdata/mssql/backups/vault_DATE.bak /opt/bitwarden/bwdata/mssql/data/

# Start Bitwarden
./bitwarden.sh start

Conclusion#

You now have a secured and properly configured Bitwarden installation on your Ubuntu server. This setup provides a robust, self-hosted password management solution that gives you complete control over your sensitive data. Regular maintenance and backups will ensure your installation remains secure and reliable.

Remember that while self-hosting provides increased privacy and control, it also means you're responsible for security, backups, and maintenance. Following the steps in this guide will help you manage these responsibilities effectively.

For additional support, visit the official Bitwarden community forums or GitHub repository, where many users share their experiences and solutions to common problems.

The Ultimate Guide to Installing and Securing Home Assistant on Ubuntu 20.04+

Sun, 09 Mar 2025 05:49:06 GMT

Home Assistant is a powerful open-source home automation platform that puts local control and privacy first. It allows you to integrate and control all your smart devices from a single dashboard, creating automations that make your home truly smart. This comprehensive guide will walk you through installing Home Assistant on Ubuntu 20.04 and newer versions using multiple methods, ensuring you select the one that best fits your needs and technical comfort level.

Introduction: Understanding Your Installation Options#

Before diving into installation procedures, it's crucial to understand that Home Assistant offers several installation methods, each with different features and complexity levels. We'll cover Docker installation (most flexible), VM installation (most isolated), and direct OS installation (most integrated).

Home Assistant comes in several flavors, each with its own benefits and trade-offs:

Home Assistant OS: A complete, managed operating system designed specifically for Home Assistant. This is the recommended method for dedicated devices.
Home Assistant Container: Running Home Assistant in a Docker container. Great for users who want to run Home Assistant alongside other services on the same machine.
Home Assistant Supervised: A middle ground that allows you to install the Supervisor component on top of a regular operating system.
Home Assistant Core: The most minimal installation, without the Supervisor and add-ons.

This table compares key features across installation methods:

Feature	Home Assistant OS	Home Assistant Container	Home Assistant Core	Home Assistant Supervised
Automations	✓	✓	✓	✓
Dashboards	✓	✓	✓	✓
Integrations	✓	✓	✓	✓
Add-ons	✓	✗	✗	✓
Backups	✓	✗	✗	✓
One-click updates	✓	✗	✗	✓*

*Home Assistant Supervised requires users to maintain their own operating system.

Method 1: Installing Home Assistant Using Docker (Recommended)#

Docker provides an excellent balance of flexibility and isolation, making it ideal for running Home Assistant on an existing Ubuntu system. This method is perfect if you're already running other services on your machine and want to keep Home Assistant contained.

Prerequisites#

Ubuntu 20.04 or newer installed and updated
A user with sudo privileges
Minimum hardware: 2GB RAM, dual-core CPU, 32GB storage
Network: Port 8123 (Home Assistant web interface) should be accessible
Basic familiarity with terminal commands

Step 1: Update Your Ubuntu System#

First, ensure your system is fully updated to prevent any dependency issues:

Run in terminalbash

sudo apt update && sudo apt upgrade -y

This updates your package list and upgrades all installed packages to their latest versions. If the update fails, check your internet connection or try changing your APT mirror.

Step 2: Install Required Dependencies#

Install the necessary packages for Docker installation:

Run in terminalbash

sudo apt install -y apt-transport-https ca-certificates curl software-properties-common gnupg lsb-release

These packages enable secure APT connections and provide tools needed for the Docker repository. If you encounter any "package not found" errors, your APT sources may be incomplete.

Step 3: Add Docker's Official GPG Key#

Add Docker's official GPG key to ensure package authenticity:

Run in terminalbash

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

This command downloads Docker's GPG key and adds it to your system. If it fails, check your internet connection or try again later.

Step 4: Set Up the Docker Repository#

Add the Docker repository to your APT sources:

Run in terminalbash

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

This command adds the appropriate Docker repository for your Ubuntu version. If you see an error about an unsupported distribution, you may need to manually specify a supported Ubuntu codename.

Step 5: Install Docker Engine#

Update the package index and install Docker:

Run in terminalbash

sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

This installs the Docker Engine, CLI, containerd runtime, and Docker Compose plugin. If the installation fails, check the Docker compatibility with your specific Ubuntu version.

Step 6: Add Your User to the Docker Group#

Add your user to the Docker group to run Docker commands without sudo:

Run in terminalbash

sudo usermod -aG docker $USER

This command adds your current user to the Docker group. For the changes to take effect, you'll need to log out and log back in, or run newgrp docker to update your current session.

Step 7: Create a Directory for Home Assistant Configuration#

Create a directory to store your Home Assistant configuration files:

Run in terminalbash

mkdir -p ~/homeassistant/config

This creates a directory structure for Home Assistant's persistent data. If the command fails, check that you have write permissions to your home directory.

Step 8: Run Home Assistant Container#

Now, run the Home Assistant container with the configuration volume mounted:

Run in terminalbash

docker run -d \
  --name=homeassistant \
  --restart=unless-stopped \
  --network=host \
  -v ~/homeassistant/config:/config \
  -v /etc/localtime:/etc/localtime:ro \
  --privileged \
  ghcr.io/home-assistant/home-assistant:stable

This command starts Home Assistant in a Docker container with access to your network and mounts the configuration directory. The --privileged flag is needed for certain integrations, though it has security implications. If the container doesn't start, check the Docker logs with docker logs homeassistant.

Step 9: Verify Home Assistant is Running#

Check that Home Assistant started successfully:

Run in terminalbash

docker ps | grep homeassistant

You should see the running container in the output. If not, check the logs for errors: docker logs homeassistant.

Step 10: Access Home Assistant Web Interface#

Open a web browser and navigate to:

Open in your browserbash

http://YOUR_SERVER_IP:8123

Replace YOUR_SERVER_IP with your server's IP address. The initial setup will take a few minutes as Home Assistant downloads and initializes components. If you can't access the web interface, ensure port 8123 isn't blocked by a firewall.

Method 2: Installing Home Assistant OS in a Virtual Machine#

If you prefer a more isolated installation or want the complete Home Assistant experience with add-ons, installing Home Assistant OS in a virtual machine is an excellent option.

Prerequisites#

Ubuntu 20.04 or newer as host OS
At least 4GB of free RAM and 2 CPU cores
32GB of free disk space
Virtualization enabled in BIOS/UEFI
Administrator privileges

Step 1: Install KVM and Required Packages#

Install KVM and related packages for virtualization:

Run in terminalbash

sudo apt update && sudo apt install -y qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virt-manager ovmf

These packages provide the necessary components for KVM virtualization. If the installation fails, ensure your CPU supports virtualization and that it's enabled in your BIOS/UEFI settings.

Step 2: Add Your User to Virtualization Groups#

Add your user to the libvirt and kvm groups:

Run in terminalbash

sudo adduser $USER libvirt
sudo adduser $USER kvm

This grants your user permission to manage virtual machines. Log out and back in for these changes to take effect.

Step 3: Verify Libvirt is Running#

Check that the libvirt daemon is running:

Run in terminalbash

sudo systemctl status libvirtd

If it's not running, start and enable it:

Run in terminalbash

sudo systemctl enable --now libvirtd

Step 4: Download Home Assistant OS Image#

Create a directory for the VM image and download the latest Home Assistant OS:

Run in terminalbash

mkdir -p ~/hassos-vm && cd ~/hassos-vm
wget https://github.com/home-assistant/operating-system/releases/download/10.0/haos_ova-10.0.qcow2.xz

This downloads the latest QCOW2 image for Home Assistant OS. If the link is outdated, visit the Home Assistant GitHub releases page for the current version.

Step 5: Extract the Image#

Extract the downloaded image:

Run in terminalbash

unxz haos_ova-10.0.qcow2.xz

This extracts the compressed QCOW2 image. If the command fails, ensure you have the xz-utils package installed.

Step 6: Create a Virtual Machine#

Create a virtual machine using the extracted image:

Run in terminalbash

sudo virt-install \
  --name homeassistant \
  --memory 4096 \
  --vcpus 2 \
  --import \
  --disk ~/hassos-vm/haos_ova-10.0.qcow2,format=qcow2 \
  --os-variant generic \
  --network bridge=virbr0 \
  --graphics none \
  --boot uefi

This creates a VM with 4GB RAM and 2 vCPUs. If you encounter errors about missing firmware, ensure the ovmf package is properly installed. Adjust the memory and CPU values based on your system capabilities.

Step 7: Find Your VM's IP Address#

List your virtual machines and find their IP addresses:

Run in terminalbash

sudo virsh list
sudo virsh domifaddr homeassistant

Note the IP address from the output. If no IP address is displayed, wait a few minutes for the VM to fully boot and obtain an IP address.

Step 8: Access Home Assistant#

Open a web browser and navigate to:

Open in your browserbash

http://VM_IP_ADDRESS:8123

Replace VM_IP_ADDRESS with the IP address obtained in the previous step. The initial setup will take a few minutes. If you can't access the web interface, ensure the VM is running and has network connectivity.

Step 9: Enable Automatic Start for the VM#

Configure the VM to start automatically when your system boots:

Run in terminalbash

sudo virsh autostart homeassistant

This ensures Home Assistant will be available after system reboots. If the command fails, check that the VM exists and that you have sufficient privileges.

Method 3: Installing Home Assistant Core on Ubuntu#

For users who want to run Home Assistant directly on their Ubuntu system without containerization or virtualization, Home Assistant Core is a viable option, though it lacks the add-on system available in other methods.

Prerequisites#

Ubuntu 20.04 or newer
Python 3.11 or newer
2GB RAM minimum (4GB recommended)
32GB storage space
Administrator privileges

Step 1: Install Required Dependencies#

Install the necessary packages for Home Assistant Core:

Run in terminalbash

sudo apt update && sudo apt install -y python3-pip python3-dev python3-venv libffi-dev libssl-dev libjpeg-dev zlib1g-dev autoconf build-essential libopenjp2-7 libtiff5 libturbojpeg0-dev tzdata ffmpeg liblapack3 liblapack-dev libatlas-base-dev

These packages provide essential libraries for Home Assistant. If any package installation fails, try running sudo apt update again or check for specific package name changes in your Ubuntu version.

Step 2: Install Python 3.11#

Add the deadsnakes PPA and install Python 3.11:

Run in terminalbash

sudo add-apt-repository ppa:deadsnakes/ppa -y
sudo apt update
sudo apt install -y python3.11 python3.11-dev python3.11-venv

This installs Python 3.11 and related development packages. If you're using Ubuntu 22.04 or newer, Python 3.10+ might already be installed, but 3.11 is recommended for best compatibility with Home Assistant.

Step 3: Create a Home Assistant User#

Create a dedicated user for running Home Assistant:

Run in terminalbash

sudo useradd -rm homeassistant
sudo usermod -aG dialout homeassistant

This creates a system user and adds it to the dialout group for serial device access. If the user already exists, you can skip this step.

Step 4: Create the Home Assistant Directory#

Create a directory for Home Assistant and set the proper ownership:

Run in terminalbash

sudo mkdir -p /srv/homeassistant
sudo chown homeassistant:homeassistant /srv/homeassistant

This creates the directory where Home Assistant will be installed and gives ownership to the homeassistant user. If the directory already exists, adjust the commands accordingly.

Step 5: Create and Activate Virtual Environment#

Switch to the homeassistant user and create a Python virtual environment:

Run in terminalbash

sudo -u homeassistant -H -s
cd /srv/homeassistant
python3.11 -m venv .
source bin/activate

This creates an isolated Python environment for Home Assistant. If you encounter an error about Python 3.11 not being found, ensure it was installed correctly in step 2.

Step 6: Install Home Assistant Core#

With the virtual environment activated, install Home Assistant:

Run in terminalbash

pip3 install wheel
pip3 install homeassistant

This installs the wheel package (required for some dependencies) and then Home Assistant itself. The installation may take several minutes. If you encounter errors about missing C compilers, ensure the build-essential package is installed.

Step 7: Start Home Assistant for the First Time#

Launch Home Assistant to complete the initial setup:

Run in terminalbash

hass

This starts Home Assistant and initializes the configuration directory. Wait until you see "Home Assistant is now available at http://localhost:8123" in the console output. The first startup may take 5-10 minutes as dependencies are downloaded and compiled.

Press Ctrl+C to stop Home Assistant after the initial setup is complete.

Step 8: Create a Systemd Service#

Exit the homeassistant user shell:

Run in terminalbash

exit

Create a systemd service file for Home Assistant:

Run in terminalbash

sudo nano /etc/systemd/system/home-assistant@homeassistant.service

Add the following content to the file:

File contentbash

[Unit]
Description=Home Assistant
After=network-online.target

[Service]
Type=simple
User=%i
WorkingDirectory=/srv/homeassistant
ExecStart=/srv/homeassistant/bin/hass -c "/srv/homeassistant/.homeassistant"
Restart=on-failure
RestartForceExitStatus=100

[Install]
WantedBy=multi-user.target

This creates a systemd service to manage Home Assistant. If you've installed Home Assistant in a different location, adjust the paths accordingly.

Step 9: Enable and Start the Home Assistant Service#

Enable the service to start on boot and start it immediately:

Run in terminalbash

sudo systemctl enable home-assistant@homeassistant.service
sudo systemctl start home-assistant@homeassistant.service

This enables and starts the Home Assistant service. Check its status with:

Run in terminalbash

sudo systemctl status home-assistant@homeassistant.service

If the service fails to start, check the logs for errors:

Run in terminalbash

sudo journalctl -fu home-assistant@homeassistant.service

Step 10: Access Home Assistant#

Open a web browser and navigate to:

Open in your browserbash

http://YOUR_SERVER_IP:8123

Replace YOUR_SERVER_IP with your server's IP address. Complete the initial setup by creating an admin account. If you can't access the web interface, check that port 8123 isn't blocked by a firewall.

Securing Your Home Assistant Installation#

Regardless of which installation method you choose, securing your Home Assistant instance is crucial, especially if you plan to access it remotely.

Set Up HTTPS with Let's Encrypt#

If you're using Docker, you can add a reverse proxy like Nginx Proxy Manager or Traefik alongside Home Assistant:

Option 1: Using Nginx Proxy Manager#

Create a docker-compose.yml file that includes both Home Assistant and Nginx Proxy Manager:

File contentyaml

version: '3'
services:
  homeassistant:
    container_name: homeassistant
    image: ghcr.io/home-assistant/home-assistant:stable
    volumes:
      - ~/homeassistant/config:/config
      - /etc/localtime:/etc/localtime:ro
    restart: unless-stopped
    network_mode: host
    
  nginx-proxy-manager:
    container_name: nginx-proxy-manager
    image: 'jc21/nginx-proxy-manager:latest'
    restart: unless-stopped
    ports:
      - '80:80'
      - '81:81'
      - '443:443'
    volumes:
      - ~/nginx-proxy-manager/data:/data
      - ~/nginx-proxy-manager/letsencrypt:/etc/letsencrypt

Start the services:

Run in terminalbash

docker compose up -d

Access Nginx Proxy Manager at http://YOUR_SERVER_IP:81 and set up a proxy host for your Home Assistant instance, enabling SSL with Let's Encrypt.

Restrict Access with Strong Authentication#

Use a strong, unique password for your Home Assistant admin account
Enable Two-Factor Authentication in Home Assistant
Consider using a separate user account for day-to-day usage

Configure a Firewall#

Install and configure UFW (Uncomplicated Firewall):

Run in terminalbash

sudo apt install -y ufw
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 8123/tcp  # Only if you need direct access to Home Assistant
sudo ufw enable

Verify the firewall status:

Run in terminalbash

sudo ufw status

If you're using a reverse proxy, you only need to allow ports 80 and 443 instead of 8123.

Data Persistence and Backup Procedures#

Protecting your Home Assistant configuration and data is essential for disaster recovery.

Creating Regular Backups#

For Docker Installations#

Install the Auto Backup add-on in Home Assistant (if using Supervisor) or create a script for automated backups:

Run in terminalbash

mkdir -p ~/homeassistant/backups

Create a backup script:

Run in terminalbash

nano ~/backup-homeassistant.sh

Add the following content:

File contentbash

#!/bin/bash
TIMESTAMP=$(date +"%Y%m%d-%H%M%S")
BACKUP_DIR=~/homeassistant/backups
mkdir -p $BACKUP_DIR
tar -czf $BACKUP_DIR/homeassistant-config-$TIMESTAMP.tar.gz -C ~/homeassistant config
echo "Backup created: $BACKUP_DIR/homeassistant-config-$TIMESTAMP.tar.gz"

Make the script executable:

Run in terminalbash

chmod +x ~/backup-homeassistant.sh

Set up a cron job to run backups weekly:

Run in terminalbash

(crontab -l 2>/dev/null; echo "0 2 * * 0 $HOME/backup-homeassistant.sh") | crontab -

This creates a compressed backup of your Home Assistant configuration directory every Sunday at 2 AM. If the backup fails, check that tar is installed and that you have sufficient disk space.

Restoring from Backup#

If you need to restore from a backup:

Run in terminalbash

tar -xzf ~/homeassistant/backups/homeassistant-config-TIMESTAMP.tar.gz -C ~/homeassistant

Replace TIMESTAMP with the actual timestamp of your backup file. If the restore fails, ensure the backup file exists and is not corrupted.

Common Issues and Solutions#

Here are some common issues users face when installing and running Home Assistant on Ubuntu:

Cannot Access Home Assistant Web Interface#

Symptoms: Cannot connect to Home Assistant at http://YOUR_SERVER_IP:8123

Diagnostic Command:

Run in terminalbash

curl -v http://localhost:8123

Solutions:

Check if Home Assistant is running: # For Docker: docker ps | grep homeassistant # For systemd service: sudo systemctl status home-assistant@homeassistant.service
Check firewall settings: sudo ufw status
Ensure the correct ports are open if using Docker: docker run -d \ --name=homeassistant \ --network=host \ # Important for device discovery -v ~/homeassistant/config:/config \ ghcr.io/home-assistant/home-assistant:stable

Home Assistant Container Starts But Crashes#

Symptoms: Container shows as running then stops after a few seconds

Diagnostic Command:

Run in terminalbash

docker logs homeassistant

Solutions:

Check permissions on config directory: sudo chown -R $(id -u):$(id -g) ~/homeassistant/config
Ensure enough memory is available: free -h
Try starting with a clean configuration: mv ~/homeassistant/config ~/homeassistant/config.bak mkdir -p ~/homeassistant/config docker restart homeassistant

Z-Wave or Zigbee Devices Not Detected#

Symptoms: USB devices connected to the host are not detected in Home Assistant

Diagnostic Command:

Run in terminalbash

ls -l /dev/tty*

Solutions:

Add the correct device mapping to Docker: docker stop homeassistant docker rm homeassistant docker run -d \ --name=homeassistant \ --restart=unless-stopped \ --network=host \ -v ~/homeassistant/config:/config \ -v /etc/localtime:/etc/localtime:ro \ --device=/dev/ttyACM0:/dev/ttyACM0 \ # Adjust device path as needed ghcr.io/home-assistant/home-assistant:stable
Add your user to the dialout group: sudo usermod -aG dialout $USER # Log out and back in for this to take effect

Database Errors After Upgrade#

Symptoms: Database-related errors in Home Assistant logs after upgrading

Diagnostic Command:

Run in terminalbash

# For Docker:
docker logs homeassistant | grep -i database

# For direct installation:
sudo journalctl -fu home-assistant@homeassistant.service | grep -i database

Solutions:

Check database file permissions: sudo chown -R homeassistant:homeassistant ~/homeassistant/config/home-assistant_v2.db*
Create a backup before attempting repairs: cp ~/homeassistant/config/home-assistant_v2.db ~/homeassistant/config/home-assistant_v2.db.bak

Maintenance Procedures#

Keeping your Home Assistant installation updated and properly maintained is essential for security and stability.

Updating Home Assistant#

For Docker Installations#

Run in terminalbash

# Pull the latest image
docker pull ghcr.io/home-assistant/home-assistant:stable

# Stop and remove the current container
docker stop homeassistant
docker rm homeassistant

# Start a new container with the latest image
docker run -d \
  --name=homeassistant \
  --restart=unless-stopped \
  --network=host \
  -v ~/homeassistant/config:/config \
  -v /etc/localtime:/etc/localtime:ro \
  ghcr.io/home-assistant/home-assistant:stable

If you're using Docker Compose:

Run in terminalbash

cd ~/
docker compose pull
docker compose up -d

Always create a backup before updating. If the update fails, you can revert to the previous version by specifying the exact image tag.

For Direct Installations#

Run in terminalbash

sudo systemctl stop home-assistant@homeassistant.service
sudo -u homeassistant -H -s
cd /srv/homeassistant
source bin/activate
pip3 install --upgrade homeassistant
exit
sudo systemctl start home-assistant@homeassistant.service

Monitoring Home Assistant#

Install monitoring tools to keep track of your Home Assistant instance:

Run in terminalbash

sudo apt install -y htop iotop

Monitor system resources:

Run in terminalbash

htop

Monitor disk I/O:

Run in terminalbash

sudo iotop

View Home Assistant logs:

Run in terminalbash

# For Docker:
docker logs -f homeassistant

# For direct installation:
sudo journalctl -fu home-assistant@homeassistant.service

Conclusion#

You've now successfully installed and secured Home Assistant on your Ubuntu 20.04+ system. Whether you chose the Docker method for flexibility, the VM approach for isolation, or direct installation for simplicity, you now have a powerful home automation platform at your fingertips.

Remember to keep your system and Home Assistant updated regularly, create backups before making significant changes, and refer to the official Home Assistant documentation for detailed information about specific integrations and features.

With Home Assistant, you're well on your way to creating a smarter, more automated home that respects your privacy and works exactly the way you want it to.

Complete Guide to Installing and Hosting Nextcloud on Ubuntu 20.04+

Sun, 09 Mar 2025 04:28:24 GMT

Introduction: What is Nextcloud and Why Self-Host?#

Nextcloud is a powerful open-source file sharing and collaboration platform that gives you complete control over your data. Unlike third-party cloud services like Dropbox or Google Drive, Nextcloud allows you to host your own private cloud on your server, ensuring data privacy and security while offering similar functionality including file storage, sharing, calendars, contacts, and more through its extensible app system.

Self-hosting Nextcloud provides several key benefits:

Complete data privacy and control
No subscription fees (beyond hosting costs)
Customizable features through the app ecosystem
Scalable storage based on your hardware
Freedom from vendor lock-in

This guide walks you through two installation methods with minimal manual intervention required.

Prerequisites#

Before beginning, ensure you have:

Code

# System requirements
Ubuntu 20.04 LTS or newer
Root or sudo access to the server

Hardware requirements:

2GB RAM (4GB+ recommended for production use)
2 CPU cores (more for multi-user environments)
At least 10GB storage space (plus additional space for your data)

Network requirements:

A domain name pointed to your server (for secure HTTPS access)
Open ports in your firewall: 80/TCP (HTTP)443/TCP (HTTPS)

Installation Methods#

Choose between two installation approaches:

Traditional LAMP Stack Installation - More customizable, better for production environments
Snap Package Installation - Simpler, faster setup with automatic updates

Method 1: Traditional LAMP Stack Installation#

Step 1: Update Your System#

Run in terminalbash

sudo apt update && sudo apt upgrade -y

This updates package lists and upgrades all installed packages to their latest versions.

Step 2: Install LAMP Stack#

Run in terminalbash

sudo apt install apache2 mariadb-server libapache2-mod-php php-gd php-json php-mysql php-curl php-mbstring php-intl php-imagick php-xml php-zip php-mysql php-bcmath php-gmp -y

This installs the web server, database server, and all required PHP extensions for Nextcloud.

Step 3: Configure MariaDB#

Secure your MariaDB installation:

Run in terminalbash

sudo mysql_secure_installation

Follow the prompts to set a root password and secure your database. When asked, answer 'Y' to all security questions.

Create a database and user for Nextcloud:

Run in terminalbash

sudo mysql -u root -p

When the MySQL prompt appears, run:

Codebash

CREATE DATABASE nextcloud;
CREATE USER 'nextclouduser'@'localhost' IDENTIFIED BY 'your-strong-password';
GRANT ALL PRIVILEGES ON nextcloud.* TO 'nextclouduser'@'localhost';
FLUSH PRIVILEGES;
EXIT;

Replace 'your-strong-password' with a secure password. This creates a dedicated database and user for Nextcloud.

Step 4: Configure PHP#

Run in terminalbash

sudo nano /etc/php/7.4/apache2/php.ini

For Ubuntu 22.04, use 8.1 instead of 7.4 in the path.

Find and modify these values:

File contentbash

memory_limit = 512M
upload_max_filesize = 200M
max_execution_time = 360
post_max_size = 200M
date.timezone = Your/Timezone

Replace Your/Timezone with your actual timezone (e.g., America/New_York). These settings optimize PHP for file uploads and Nextcloud operations.

Step 5: Download and Extract Nextcloud#

Run in terminalbash

cd /tmp
wget https://download.nextcloud.com/server/releases/latest.tar.bz2

This downloads the latest stable Nextcloud version.

Extract and move to web directory:

Run in terminalbash

sudo tar -xjf latest.tar.bz2
sudo mv nextcloud /var/www/html/

This extracts Nextcloud and places it in the Apache web root directory.

Step 6: Set Correct Permissions#

Run in terminalbash

sudo chown -R www-data:www-data /var/www/html/nextcloud/
sudo chmod -R 755 /var/www/html/nextcloud/

This ensures the web server can access and modify Nextcloud files. Incorrect permissions will cause installation or operation failures.

Step 7: Configure Apache for Nextcloud#

Run in terminalbash

sudo nano /etc/apache2/sites-available/nextcloud.conf

Add this configuration:

File contentbash

<VirtualHost *:80>
    ServerName your-domain.com
    DocumentRoot /var/www/html/nextcloud/

    <Directory /var/www/html/nextcloud/>
        Options +FollowSymlinks
        AllowOverride All
        Require all granted
        <IfModule mod_dav.c>
            Dav off
        </IfModule>
        SetEnv HOME /var/www/html/nextcloud
        SetEnv HTTP_HOME /var/www/html/nextcloud
    </Directory>

    ErrorLog ${APACHE_LOG_DIR}/nextcloud_error.log
    CustomLog ${APACHE_LOG_DIR}/nextcloud_access.log combined
</VirtualHost>

Replace your-domain.com with your actual domain name.

Enable the site and required Apache modules:

Run in terminalbash

sudo a2ensite nextcloud.conf
sudo a2enmod rewrite headers env dir mime setenvif ssl
sudo systemctl restart apache2

This activates the Nextcloud site and enables necessary Apache modules.

Step 8: Set Up SSL with Let's Encrypt#

Run in terminalbash

sudo apt install certbot python3-certbot-apache -y
sudo certbot --apache -d your-domain.com

Replace your-domain.com with your actual domain. This obtains and installs a free SSL certificate.

Step 9: Complete the Web-Based Installation#

Open your web browser and navigate to https://your-domain.com. You'll see the Nextcloud setup page where you'll enter:

Admin username and password
Data folder: Keep the default /var/www/html/nextcloud/data
Database user: nextclouduser
Database password: The password you set in Step 3
Database name: nextcloud
Database host: localhost

Click "Finish setup" to complete the installation.

Method 2: Snap Package Installation (Simplified Approach)#

Step 1: Install Snapd#

Run in terminalbash

sudo apt update
sudo apt install snapd -y

This installs the snap package manager.

Step 2: Install Nextcloud via Snap#

Run in terminalbash

sudo snap install nextcloud

This downloads and installs Nextcloud with all dependencies.

Step 3: Create Admin Account#

Run in terminalbash

sudo nextcloud.manual-install admin "your-strong-password"

Replace your-strong-password with a secure password. This creates the admin account.

Step 4: Configure Domain#

Run in terminalbash

sudo nextcloud.occ config:system:set trusted_domains 1 --value=your-domain.com

Replace your-domain.com with your actual domain.

Step 5: Enable HTTPS#

Run in terminalbash

sudo nextcloud.enable-https lets-encrypt

Enter your email address and agree to the terms when prompted. This obtains and installs a free SSL certificate.

Security Hardening#

Configure Basic Firewall#

Run in terminalbash

sudo apt install ufw -y
sudo ufw allow 22/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

This installs and configures UFW firewall, allowing only SSH and web traffic.

Protect Against Brute Force Attacks with Fail2ban#

Run in terminalbash

sudo apt install fail2ban -y
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
sudo nano /etc/fail2ban/jail.local

Add to the end of the file:

File contentbash

[nextcloud]
enabled = true
port = 80,443
filter = nextcloud
logpath = /var/log/apache2/nextcloud_access.log
maxretry = 3
bantime = 86400

For snap installations, use /var/snap/nextcloud/current/logs/nextcloud.log as the logpath.

Create a filter:

Run in terminalbash

sudo nano /etc/fail2ban/filter.d/nextcloud.conf

Add:

File contentbash

[Definition]
failregex = ^.*Login failed: '.*' \(Remote IP: ''.*$
ignoreregex =

Restart Fail2ban:

Run in terminalbash

sudo systemctl restart fail2ban

This protects your login page from repeated password attempts.

Data Management and Backups#

Configure External Storage (Traditional Installation)#

Run in terminalbash

sudo mkdir -p /var/nextcloud-data
sudo chown -R www-data:www-data /var/nextcloud-data
sudo chmod -R 750 /var/nextcloud-data

Edit Nextcloud config:

Run in terminalbash

sudo nano /var/www/html/nextcloud/config/config.php

Update the datadirectory parameter:

Codebash

'datadirectory' => '/var/nextcloud-data',

Automated Backup Script (Traditional Installation)#

Run in terminalbash

sudo nano /usr/local/bin/nextcloud-backup.sh

Add this script:

File contentbash

#!/bin/bash
BACKUP_DATE=$(date +"%Y%m%d")
BACKUP_DIR="/var/backups/nextcloud"

# Create backup directory if it doesn't exist
mkdir -p $BACKUP_DIR

# Enable maintenance mode
cd /var/www/html/nextcloud
sudo -u www-data php occ maintenance:mode --on

# Backup database
mysqldump --single-transaction -u nextclouduser -p'your-strong-password' nextcloud > $BACKUP_DIR/nextcloud-db-$BACKUP_DATE.sql

# Backup data directory
tar -czf $BACKUP_DIR/nextcloud-data-$BACKUP_DATE.tar.gz -C /var /var/nextcloud-data

# Backup config directory
tar -czf $BACKUP_DIR/nextcloud-config-$BACKUP_DATE.tar.gz -C /var/www/html/nextcloud config

# Disable maintenance mode
sudo -u www-data php occ maintenance:mode --off

# Remove backups older than 30 days
find $BACKUP_DIR -name "nextcloud-*" -type f -mtime +30 -delete

Make it executable and schedule:

Run in terminalbash

sudo chmod +x /usr/local/bin/nextcloud-backup.sh
sudo crontab -e

Add this line for daily 2AM backups:

File contentbash

0 2 * * * /usr/local/bin/nextcloud-backup.sh

Snap Installation Backup#

Run in terminalbash

sudo nextcloud.export

This creates a complete backup in /var/snap/nextcloud/common/backups/.

Schedule automated backups:

Run in terminalbash

sudo crontab -e

Add:

File contentbash

0 2 * * * /snap/bin/nextcloud.export

Troubleshooting Common Issues#

Fix Permission Denied Errors#

Run in terminalbash

sudo chown -R www-data:www-data /var/www/html/nextcloud/
sudo find /var/www/html/nextcloud/ -type d -exec chmod 750 {} \;
sudo find /var/www/html/nextcloud/ -type f -exec chmod 640 {} \;

This resets ownership and permissions to correct values.

Resolve Database Connection Issues#

Run in terminalbash

sudo mysql -u root -p

Then run:

Codebash

GRANT ALL PRIVILEGES ON nextcloud.* TO 'nextclouduser'@'localhost';
FLUSH PRIVILEGES;
EXIT;

This ensures the database user has proper permissions.

Fix PHP Memory Limit Errors#

Run in terminalbash

sudo nano /etc/php/7.4/apache2/php.ini

For Ubuntu 22.04, use 8.1 instead of 7.4.

Change the memory limit:

File contentbash

memory_limit = 1G

Restart Apache:

Run in terminalbash

sudo systemctl restart apache2

This increases PHP memory to handle larger operations.

Improve Performance with Redis Caching#

Run in terminalbash

sudo apt install redis-server php-redis -y
sudo systemctl enable redis-server
sudo systemctl start redis-server

Edit Nextcloud config:

Run in terminalbash

sudo nano /var/www/html/nextcloud/config/config.php

Add inside the configuration array:

Codebash

'memcache.local' => '\\OC\\Memcache\\Redis',
'memcache.locking' => '\\OC\\Memcache\\Redis',
'redis' => [
     'host' => 'localhost',
     'port' => 6379,
],

Restart Apache:

Run in terminalbash

sudo systemctl restart apache2

This significantly improves performance through caching.

Maintenance Tasks#

Update Nextcloud (Traditional Installation)#

Run in terminalbash

cd /var/www/html/nextcloud
sudo -u www-data php occ maintenance:mode --on

Backup first:

Run in terminalbash

sudo mysqldump --single-transaction -u nextclouduser -p'your-strong-password' nextcloud > /var/backups/nextcloud-db-before-update.sql
sudo tar -czf /var/backups/nextcloud-before-update.tar.gz -C /var/www/html nextcloud

Download and update:

Run in terminalbash

cd /tmp
wget https://download.nextcloud.com/server/releases/latest.tar.bz2
sudo tar -xjf latest.tar.bz2
sudo cp -r nextcloud/. /var/www/html/nextcloud/
sudo chown -R www-data:www-data /var/www/html/nextcloud/

Complete the upgrade:

Run in terminalbash

cd /var/www/html/nextcloud
sudo -u www-data php occ upgrade
sudo -u www-data php occ maintenance:mode --off

Update Snap Installation#

Run in terminalbash

sudo snap refresh nextcloud

This updates Nextcloud to the latest version.

Monitor Your Installation#

Check status:

Run in terminalbash

# For traditional installation
cd /var/www/html/nextcloud
sudo -u www-data php occ status

# For snap installation
sudo nextcloud.occ status

View logs:

Run in terminalbash

# For traditional installation
sudo tail -f /var/log/apache2/nextcloud_error.log

# For snap installation
sudo snap logs nextcloud

These commands help diagnose issues.

Conclusion#

You now have a fully functional, secure Nextcloud installation on your Ubuntu server. Your self-hosted cloud provides a private alternative to commercial services with similar features.

Remember to:

Keep your system updated regularly
Back up your data frequently
Monitor logs for unusual activity
Follow security best practices

For basic usage, you can stop after completing either installation method and securing with SSL. For production environments, consider implementing the security hardening and performance optimization steps.

Complete Guide to Installing and Configuring ComfyUI on Ubuntu

Sat, 08 Mar 2025 06:22:50 GMT

This guide will walk you through installing and configuring ComfyUI on Ubuntu with simple, copy-pastable commands. I've designed this to be as straightforward as possible, explaining each step along the way.

Introduction#

ComfyUI is a powerful tool for creating AI-generated images using Stable Diffusion models. It uses a visual, node-based interface where you connect different components together - kind of like building with blocks. This makes it much easier to use than typing commands.

What you'll get when you're done:

A working ComfyUI installation on your Ubuntu computer
The ability to generate AI images with various models
ComfyUI running as a system service (starting automatically when your computer boots)
Optional: Access to ComfyUI from other devices on your network

Prerequisites#

Before we start, you'll need:

An Ubuntu computer (version 20.04 or newer)
Administrator access (ability to use sudo commands)
At least 8GB of RAM (16GB or more is better)
A graphics card (GPU) with at least 4GB of memory
Internet connection

Don't worry if you're not sure about some of these requirements - we'll check them as we go.

1. System Update and Basic Dependencies#

First, let's make sure your system is up to date and has the basic software we need.

Open a terminal window by pressing Ctrl+Alt+T on your keyboard, then copy and paste these commands:

Run in terminalbash

sudo apt update
sudo apt upgrade -y
sudo apt install -y git python3 python3-pip python3-venv

What this does:

Updates your system's package list
Upgrades any outdated packages
Installs Git (for downloading ComfyUI), Python (the programming language ComfyUI uses), and tools for creating isolated Python environments

2. Graphics Driver Installation#

For ComfyUI to work well, you need proper drivers for your graphics card. We'll cover both NVIDIA and AMD installations. You only need to follow the section that matches your graphics card type.

2.1 Checking Your Graphics Card Type#

Not sure what graphics card you have? Run this command:

Run in terminalbash

lspci | grep -E "VGA|3D|Display"

Look at the output - if you see "NVIDIA" in the results, follow the NVIDIA instructions. If you see "AMD" or "Radeon," follow the AMD instructions.

2.2 For NVIDIA GPUs#

2.2.1 Disable the Default Open-Source Driver#

The default open-source driver (Nouveau) doesn't work well for AI tasks. We need to disable it:

Run in terminalbash

echo 'blacklist nouveau' | sudo tee -a /etc/modprobe.d/blacklist-nouveau.conf
echo 'options nouveau modeset=0' | sudo tee -a /etc/modprobe.d/blacklist-nouveau.conf
sudo update-initramfs -u

What this does: Prevents the basic open-source driver from loading, so we can use NVIDIA's better-performing driver instead.

2.2.2 Install NVIDIA Driver#

Run in terminalbash

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
ubuntu-drivers devices

The last command will show you available drivers. Look for a line that says "recommended" - note that version number.

Now install the recommended driver (replace "550" with your recommended version):

Run in terminalbash

sudo apt install -y nvidia-driver-550

Note: If you already have the NVIDIA driver installed, you'll see a message saying it's already the newest version. That's fine - you can continue to the next step.

2.2.3 Reboot Your System#

Run in terminalbash

sudo reboot

After your computer restarts, open a terminal again and continue.

2.2.4 Verify the Driver Installation#

Run in terminalbash

nvidia-smi

You should see information about your GPU and the installed driver. If you see an error message instead, the driver installation might have failed - try rebooting again or checking for error messages.

2.3 For AMD GPUs#

2.3.1 Install AMD GPU Drivers#

For Ubuntu 22.04 or newer:

Run in terminalbash

sudo mkdir --parents --mode=0755 /etc/apt/keyrings
wget https://repo.radeon.com/rocm/rocm.gpg.key -O - | \
    gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg > /dev/null

echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/amdgpu/6.0.2/ubuntu jammy main" \
    | sudo tee /etc/apt/sources.list.d/amdgpu.list
sudo apt update

sudo apt install -y amdgpu-dkms
sudo usermod -a -G render,video $USER

What this does:

Adds AMD's software repository to your system
Installs the AMD GPU drivers
Adds your user account to the groups needed to use the GPU

2.3.2 Reboot Your System#

Run in terminalbash

sudo reboot

3. ComfyUI Installation#

Now let's install ComfyUI itself.

3.1 Clone the Repository#

This downloads the ComfyUI software to your computer:

Run in terminalbash

cd ~
git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI

3.2 Create and Activate a Virtual Environment#

A virtual environment is like a separate container for Python packages. It keeps ComfyUI's dependencies separate from your system's Python installation:

Run in terminalbash

python3 -m venv venv
source venv/bin/activate

You'll notice your terminal prompt now starts with (venv), showing you're in the virtual environment.

3.3 Install PyTorch#

PyTorch is a machine learning framework that ComfyUI needs. We'll install different versions depending on your graphics card.

For NVIDIA GPUs:

Run in terminalbash

pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121

For AMD GPUs:

Run in terminalbash

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.0

What this does: Installs PyTorch with the right settings for your graphics card type.

3.4 Install Required Dependencies#

Now let's install the rest of the software packages ComfyUI needs:

Run in terminalbash

pip install -r requirements.txt

This might take a few minutes, as it's downloading and installing several packages.

3.5 Test ComfyUI#

Let's make sure ComfyUI works before setting it up permanently:

Run in terminalbash

python main.py

You should see some messages in the terminal, eventually showing something like "Running on local URL: http://127.0.0.1:8188".

Open a web browser and go to: http://127.0.0.1:8188

You should see the ComfyUI interface. If it works, go back to the terminal and press Ctrl+C to stop ComfyUI.

4. Adding Stable Diffusion Models#

ComfyUI needs at least one AI model to generate images.

4.1 Create the Models Directory#

Run in terminalbash

mkdir -p models/checkpoints

4.2 Download a Model#

You'll need to download a Stable Diffusion model file (with a .ckpt or .safetensors extension) and place it in the models/checkpoints directory.

Where to find models:

Hugging Face (https://huggingface.co/models?other=stable-diffusion)
Civitai (https://civitai.com/)

Popular models for beginners:

Stable Diffusion 1.5
Realistic Vision
Dreamshaper

After downloading a model, move it to the checkpoints folder:

Run in terminalbash

# If you downloaded to your Downloads folder:
mv ~/Downloads/your-model-name.safetensors ~/ComfyUI/models/checkpoints/

Replace "your-model-name.safetensors" with your actual downloaded file name.

5. Setting Up ComfyUI as a System Service#

Now we'll set up ComfyUI to run automatically in the background, even after you restart your computer.

5.1 Check if Default Port (8188) is Available#

First, let's make sure the default port (8188) isn't already being used by another program:

Run in terminalbash

sudo netstat -tuln | grep 8188

If this command doesn't return anything, the port is available, and you can continue. If it shows something, that means the port is already in use. In that case, we'll need to use a different port:

Run in terminalbash

# This finds an unused port
port=$(python3 -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()')
echo "Free port: $port"

Make note of the port number it gives you - we'll use it in the next step.

5.2 Create a Service File#

This file tells your system how to run ComfyUI as a background service:

Run in terminalbash

sudo nano /etc/systemd/system/comfyui.service

A text editor will open. Copy and paste the following text, then replace YOUR_USERNAME with your actual username (the name you use to log in to your computer):

File contentbash

[Unit]
Description=ComfyUI Service
After=network.target

[Service]
Type=simple
User=YOUR_USERNAME
Group=YOUR_USERNAME
WorkingDirectory=/home/YOUR_USERNAME/ComfyUI
Environment=PATH=/home/YOUR_USERNAME/ComfyUI/venv/bin:$PATH
ExecStart=/home/YOUR_USERNAME/ComfyUI/venv/bin/python main.py --listen 0.0.0.0 --port 8188
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Important notes:

If you need to use a different port (from step 5.1), change --port 8188 to use your port number
The --listen 0.0.0.0 part allows ComfyUI to be accessed from other devices on your network, not just from your computer

Save the file by pressing Ctrl+X, then Y, then Enter.

5.3 Enable and Start the Service#

Now let's register and start the service:

Run in terminalbash

sudo systemctl daemon-reload
sudo systemctl enable comfyui
sudo systemctl start comfyui

What this does:

Reloads the service configuration
Sets ComfyUI to start automatically when your computer boots
Starts ComfyUI right now

5.4 Check Service Status#

Let's make sure the service is running properly:

Run in terminalbash

sudo systemctl status comfyui

You should see "active (running)" in green text. Press Q to exit this view.

5.5 Access ComfyUI#

Now you can access ComfyUI by opening a web browser and going to:

http://localhost:8188

(If you used a different port, replace 8188 with your port number)

To access from other devices on your network:

Find your computer's IP address:

Run in terminalbash

hostname -I | awk '{print $1}'

On another device on the same network, open a browser and go to: http://YOUR_IP_ADDRESS:8188 (Replace YOUR_IP_ADDRESS with the address from step 1)

6. Optional: Integrating with Nginx Proxy Manager#

Note: This section is completely optional. It's useful if you want to:

Access ComfyUI using a custom domain name instead of an IP address and port
Add password protection
Set up HTTPS for secure access

6.1 Install Docker and Docker Compose (if not already installed)#

First, check if Docker is already installed:

Run in terminalbash

docker --version

If you see version information, Docker is already installed and you can skip to step 6.2.

If not, install Docker:

Run in terminalbash

sudo apt install -y docker.io docker-compose
sudo usermod -aG docker $USER
sudo systemctl enable docker
sudo systemctl start docker

Important: After adding yourself to the docker group, you need to log out and log back in for the change to take effect. You can do this by restarting your computer or logging out and back in.

6.2 Create a Directory for Nginx Proxy Manager#

Run in terminal

mkdir -p ~/docker/nginx-proxy-manager
cd ~/docker/nginx-proxy-manager

6.3 Create Docker Compose File#

Run in terminalbash

nano docker-compose.yml

Copy and paste the following content:

File contentyaml

version: '3'
services:
  app:
    image: 'jc21/nginx-proxy-manager:latest'
    restart: unless-stopped
    ports:
      - '80:80'
      - '81:81'
      - '443:443'
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt

Save the file (Ctrl+X, then Y, then Enter).

6.4 Start Nginx Proxy Manager#

Run in terminalbash

docker-compose up -d

What this does: Downloads and starts Nginx Proxy Manager in a Docker container.

6.5 Access the Nginx Proxy Manager Dashboard#

Open your browser and go to http://YOUR_IP_ADDRESS:81

Replace YOUR_IP_ADDRESS with your computer's IP address (from the hostname -I command earlier).

Default login credentials:

Email: admin@example.com
Password: changeme

You'll be prompted to change these after your first login.

6.6 Create a Proxy Host for ComfyUI#

In the dashboard, go to "Proxy Hosts" and click "Add Proxy Host"
Fill in the form:
- Domain Name: Enter a name you want to use (e.g., comfyui.local)
- Scheme: http
- Forward Hostname/IP: Your computer's IP address
- Forward Port: 8188 (or your custom port)
- Check "Block Common Exploits" for better security
- You can also enable SSL if you want HTTPS access
Save the configuration

7. Optional: Pi-hole Configuration#

Note: This section is only relevant if you're already using Pi-hole for DNS on your network. If you don't know what Pi-hole is, you can skip this section.

7.1 Add a Local DNS Record for ComfyUI#

Access your Pi-hole admin interface (typically http://pi.hole/admin)
Go to "Local DNS" → "DNS Records"
Add a new record:
- Domain: comfyui.local (or your preferred name)
- IP Address: Your computer's IP address
Save the record

This lets you access ComfyUI using the domain name you specified (e.g., http://comfyui.local:8188) from any device on your network.

7.2 Configure Pi-hole with Nginx Proxy Manager (If Using Both)#

If you're using both Pi-hole and Nginx Proxy Manager, they might conflict because both want to use port 80. Here are some options:

Change Pi-hole's web port:
- Edit the lighttpd configuration: sudo nano /etc/lighttpd/lighttpd.conf
- Find the line with server.port = 80 and change it to another port (e.g., server.port = 8080)
- Restart lighttpd: sudo systemctl restart lighttpd
Use Nginx Proxy Manager to proxy Pi-hole as well:
- In Nginx Proxy Manager, add another proxy host for Pi-hole
- Set it to forward to Pi-hole's IP address on port 80
- Access Pi-hole through this proxy host instead of directly

8. Known Issues and Troubleshooting#

8.1 CUDA/GPU Memory Issues#

If ComfyUI crashes with "CUDA out of memory" errors or your images fail to generate:

Run in terminalbash

sudo systemctl edit comfyui.service

Add the following lines between the `````` markers:

File contentbash

[Service]
ExecStart=/home/YOUR_USERNAME/ComfyUI/venv/bin/python main.py --listen 0.0.0.0 --port 8188 --lowvram

Replace YOUR_USERNAME with your actual username.

Save (Ctrl+X, Y, Enter) and then restart the service:

Run in terminalbash

sudo systemctl restart comfyui

What this does: Enables low VRAM mode, which uses less GPU memory but might be slightly slower.

For extremely limited VRAM, you can use --medvram or both flags together: --lowvram --medvram

8.2 AMD GPU Driver Issues#

Some AMD GPUs (especially newer models) might not work properly with the standard settings. If you see errors when running ComfyUI, you might need to override the graphics version:

Run in terminalbash

sudo systemctl edit comfyui.service

Add the following between the `````` markers:

File contentbash

[Service]
Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"

For RDNA3 cards like the 7600, use this instead:

File contentbash

[Service]
Environment="HSA_OVERRIDE_GFX_VERSION=11.0.0"

Save and restart:

Run in terminalbash

sudo systemctl restart comfyui

8.3 Service Not Starting#

If ComfyUI doesn't start, check the logs for errors:

Run in terminalbash

sudo journalctl -u comfyui -e

Common issues include:

Incorrect paths in the service file
Virtual environment not set up correctly
Missing dependencies
GPU driver issues

8.4 Port Already in Use#

If you need to change the port after setting up the service:

Run in terminalbash

sudo systemctl edit comfyui.service

Add:

File contentbash

[Service]
ExecStart=/home/YOUR_USERNAME/ComfyUI/venv/bin/python main.py --listen 0.0.0.0 --port 8189

Replace 8189 with your desired port number. Save and restart:

Run in terminalbash

sudo systemctl restart comfyui

9. Maintenance and Updates#

9.1 Updating ComfyUI#

To update ComfyUI to the latest version:

Run in terminalbash

cd ~/ComfyUI
sudo systemctl stop comfyui
git pull
source venv/bin/activate
pip install -r requirements.txt
sudo systemctl start comfyui

What this does:

Stops the ComfyUI service
Downloads the latest code
Updates any dependencies
Restarts the service

9.2 Updating GPU Drivers#

For both NVIDIA and AMD, the simplest way to update drivers is:

Run in terminalbash

sudo apt update
sudo apt upgrade -y

This will update your drivers along with other system packages.

Conclusion#

You now have a fully functioning ComfyUI installation on your Ubuntu system! You can access it directly through your web browser at http://localhost:8188 (or your custom domain if configured).

If you make any changes to the configuration, remember to restart the service using:

Run in terminalbash

sudo systemctl restart comfyui

Enjoy creating amazing AI-generated images with ComfyUI!

Comprehensive Guide to Deploying Open WebUI with Ollama Using Docker Compose

Sat, 08 Mar 2025 04:31:10 GMT

Introduction#

Open WebUI is an extensible, feature-rich web interface for interacting with large language models that runs entirely offline. It provides a ChatGPT-like experience for models running through Ollama, an open-source framework designed for running LLMs locally. This guide covers multiple deployment configurations using Docker Compose, from basic setups to advanced scenarios with GPU support, enhanced security, and specialized integrations.

Key Benefits#

Complete Privacy: All data remains locally stored with no external requests
Administrative Control: The first account created automatically receives administrator privileges
Model Privacy: All models are private by default unless explicitly shared
Simplified Deployment: Docker Compose handles container orchestration, networking, and volume management

Prerequisites#

Before proceeding with any deployment, ensure your system meets these requirements:

Docker Engine and Docker Compose installed (verify with docker --version and docker-compose --version)
For NVIDIA GPU support:
- Compatible NVIDIA drivers installed
- NVIDIA Container Toolkit configured (verify with nvidia-smi)
For AMD GPU support:
- ROCm drivers installed and configured for container usage

Recommended System Requirements#

Modern CPU with at least 4 cores
16GB or more of RAM
NVIDIA GPU with 8GB+ VRAM (for larger models)
SSD storage for faster model loading

Basic Installation with Docker Compose#

This configuration provides a quick start with minimal setup while delivering a fully functional environment.

Step 1: Create a Project Directory#

Run in terminalbash

mkdir open-webui-ollama
cd open-webui-ollama

Step 2: Create the Docker Compose File#

Create a file named docker-compose.yml with the following content:

File contentyaml

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    volumes:
      - open-webui_data:/app/backend/data
    environment:
      - OLLAMA_API_BASE_URL=http://ollama:11434
    depends_on:
      - ollama
    restart: unless-stopped

volumes:
  ollama_data:
  open-webui_data:

Step 3: Deploy the Services#

Run in terminalbash

docker-compose up -d

Step 4: Access the Interface#

Open your web browser and navigate to http://localhost:3000. Create an account when prompted - the first account will automatically receive administrator privileges.

What Could Go Wrong?#

Port Conflicts: If ports 3000 or 11434 are already in use, you'll see errors. Change the left side of the port mapping (e.g., "3001:8080")
Memory Issues: If your system has limited RAM, larger models may fail to load. Consider using smaller models or increasing your system's memory
Docker Network Issues: If containers can't communicate, check if Docker's bridge network is functioning properly

Advanced Docker Compose Configurations#

Configuration with NVIDIA GPU Support#

For users with NVIDIA GPUs, this configuration enables hardware acceleration for significantly improved performance:

File contentyaml

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
    restart: unless-stopped
    environment:
      - OLLAMA_HOST=0.0.0.0:11434

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    volumes:
      - open-webui_data:/app/backend/data
    environment:
      - OLLAMA_API_BASE_URL=http://ollama:11434
    depends_on:
      - ollama
    restart: unless-stopped
    extra_hosts:
      - "host.docker.internal:host-gateway"

volumes:
  ollama_data:
  open-webui_data:

Configuration with AMD GPU Support#

For AMD GPU users, use this specialized configuration:

File contentyaml

version: '3.8'

services:
  ollama:
    image: ollama/ollama:rocm
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
    restart: unless-stopped
    environment:
      - HSA_OVERRIDE_GFX_VERSION=x.x.x  # Replace with your GPU's GFX version
      - OLLAMA_HOST=0.0.0.0:11434

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    volumes:
      - open-webui_data:/app/backend/data
    environment:
      - OLLAMA_API_BASE_URL=http://ollama:11434
    depends_on:
      - ollama
    restart: unless-stopped

volumes:
  ollama_data:
  open-webui_data:

Replace x.x.x with your specific AMD GPU's GFX version. You can find this information by running rocminfo and looking for the "gfx" value.

What Could Go Wrong with GPU Configurations?#

Driver Compatibility: Ensure your GPU drivers are compatible with the container runtime
Missing NVIDIA Container Toolkit: For NVIDIA GPUs, this must be properly installed
ROCm Compatibility: Not all AMD GPUs are supported by ROCm; check compatibility before deployment
Resource Allocation: If multiple applications are using the GPU, performance may be affected

Specialized Network Configurations#

Configuration for Ollama Running on Host (Outside Docker)#

If Ollama is installed directly on your system rather than in Docker:

File contentyaml

version: '3.8'

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    volumes:
      - open-webui_data:/app/backend/data
    environment:
      - OLLAMA_API_BASE_URL=http://host.docker.internal:11434
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped

volumes:
  open-webui_data:

Configuration for Ollama on a Remote Server#

If Ollama is running on a different machine:

File contentyaml

version: '3.8'

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    volumes:
      - open-webui_data:/app/backend/data
    environment:
      - OLLAMA_API_BASE_URL=https://your-ollama-server:11434
    restart: unless-stopped

volumes:
  open-webui_data:

Replace https://your-ollama-server:11434 with the actual URL of your remote Ollama server.

Configuration with Bundled Ollama#

Open WebUI offers a version with Ollama bundled into the same image for simplified deployment:

File contentyaml

version: '3.8'

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:ollama
    container_name: open-webui
    ports:
      - "3000:8080"
    volumes:
      - ollama:/root/.ollama
      - open-webui:/app/backend/data
    restart: unless-stopped

volumes:
  ollama:
  open-webui:

For GPU support with this bundled version, add the deploy section as shown in the GPU configurations.

What Could Go Wrong with Network Configurations?#

Firewall Issues: Ensure the Ollama server allows incoming connections on port 11434
SSL Certificate Problems: If using HTTPS, ensure certificates are valid
DNS Resolution: The host.docker.internal hostname might not work in all Docker environments
API Version Mismatch: Ensure Open WebUI and Ollama versions are compatible

Enhanced Security Configurations#

Configuration with Enhanced Security#

Add security-related environment variables for better protection:

File contentyaml

version: '3.8'

services:
  ollama:
    # Ollama configuration as in basic setup

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    volumes:
      - open-webui_data:/app/backend/data
    environment:
      - OLLAMA_API_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=your_secret_key  # Generate with: openssl rand -hex 32
      - WEBUI_JWT_SECRET_KEY=your_jwt_secret  # Generate with: openssl rand -hex 32
      - WEBUI_AUTH_ENABLED=true  # Enforce authentication
    depends_on:
      - ollama
    restart: unless-stopped

volumes:
  ollama_data:
  open-webui_data:

Generate secure keys using:

Run in terminalyaml

openssl rand -hex 32

Configuration with SearXNG Integration for RAG#

For users interested in Retrieval-Augmented Generation (RAG) capabilities:

File contentyaml

version: '3.8'

services:
  ollama:
    # Ollama configuration as in basic setup

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    volumes:
      - open-webui_data:/app/backend/data
    environment:
      - OLLAMA_API_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=your_secret_key
      - SEARXNG_BASE_URL=http://searxng:8080
    depends_on:
      - ollama
      - searxng
    restart: unless-stopped

  searxng:
    image: searxng/searxng:latest
    container_name: searxng
    ports:
      - "8080:8080"
    environment:
      - SEARXNG_SECRET=your_searxng_secret  # Generate with: openssl rand -hex 32
    restart: unless-stopped

volumes:
  ollama_data:
  open-webui_data:

Configuration with OpenAI API Integration#

If you want to use Open WebUI with OpenAI's API:

File contentyaml

version: '3.8'

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    volumes:
      - open-webui_data:/app/backend/data
    environment:
      - OPENAI_API_KEY=your_openai_api_key
    restart: unless-stopped

volumes:
  open-webui_data:

Replace your_openai_api_key with your actual OpenAI API key.

What Could Go Wrong with Enhanced Security?#

Secret Key Issues: If the secret key changes, existing sessions will be invalidated
JWT Authentication: If misconfigured, users may be unable to log in
SearXNG Integration: If SearXNG is unreachable, RAG functionality will not work
API Key Exposure: Storing API keys in Docker Compose files poses a security risk; consider using Docker secrets or environment variables from a secure source

Production Deployment Configurations#

Configuration with Traefik Reverse Proxy#

For production deployments with domain name support and SSL certificates:

File contentyaml

version: '3.8'

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik/traefik.yml:/etc/traefik/traefik.yml:ro
      - ./traefik/acme.json:/acme.json
    networks:
      - traefik-network
    restart: unless-stopped

  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    volumes:
      - ollama_data:/root/.ollama
    networks:
      - ollama-network
      - traefik-network
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    volumes:
      - open-webui_data:/app/backend/data
    networks:
      - ollama-network
      - traefik-network
    environment:
      - OLLAMA_API_BASE_URL=http://ollama:11434
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.openwebui.rule=Host(`your-domain.com`)"
      - "traefik.http.routers.openwebui.entrypoints=websecure"
      - "traefik.http.routers.openwebui.tls.certresolver=letsencrypt"
      - "traefik.http.services.openwebui.loadbalancer.server.port=8080"
    depends_on:
      - ollama
    restart: unless-stopped

networks:
  traefik-network:
    external: true
  ollama-network:
    external: true

volumes:
  ollama_data:
  open-webui_data:

Before deploying, create the necessary networks:

Run in terminalyaml

docker network create traefik-network
docker network create ollama-network

And create a basic Traefik configuration file at ./traefik/traefik.yml:

File contentyaml

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https

  websecure:
    address: ":443"

certificatesResolvers:
  letsencrypt:
    acme:
      email: your-email@example.com
      storage: acme.json
      httpChallenge:
        entryPoint: web

providers:
  docker:
    exposedByDefault: false
    network: traefik-network

Also, create an empty acme.json file and set proper permissions:

Run in terminalbash

touch ./traefik/acme.json
chmod 600 ./traefik/acme.json

Configuration with Cloudflare Tunnel#

For secure remote access without exposing your server directly to the internet:

File contentyaml

version: '3.8'

services:
  ollama:
    # Ollama configuration as in basic or GPU setup

  open-webui:
    # Open WebUI configuration as above

  tunnel:
    image: cloudflare/cloudflared:latest
    container_name: cloudflare-tunnel
    command: tunnel --no-autoupdate run
    environment:
      - TUNNEL_TOKEN=your_cloudflare_tunnel_token
    restart: unless-stopped
    depends_on:
      - open-webui

Replace your_cloudflare_tunnel_token with the token generated from your Cloudflare account.

What Could Go Wrong with Production Configurations?#

Certificate Issuance: Let's Encrypt has rate limits that may be hit during testing
Domain Configuration: DNS must be properly configured to point to your server
Traefik Configuration: Syntax errors in traefik.yml may prevent the service from starting
Network Isolation: Ensure services are on the correct networks to communicate properly
Cloudflare Token: If expired or revoked, the tunnel will stop working

Service Management and Maintenance#

Starting and Stopping Services#

To start your services:

Run in terminalbash

docker-compose up -d

To stop services while preserving volumes:

Run in terminalbash

docker-compose down

To completely remove everything including volumes:

Run in terminalbash

docker-compose down -v

Viewing Logs#

To view logs of all services:

Run in terminalbash

docker-compose logs -f

For a specific service:

Run in terminalbash

docker-compose logs -f open-webui

Updating Services#

Pull the latest images and restart:

Run in terminalbash

docker-compose pull
docker-compose up -d

Backing Up Data#

Back up the Docker volumes:

Run in terminalbash

docker run --rm -v ollama_data:/data -v $(pwd):/backup alpine tar czf /backup/ollama_backup.tar.gz -C /data .
docker run --rm -v open-webui_data:/data -v $(pwd):/backup alpine tar czf /backup/open-webui_backup.tar.gz -C /data .

To restore from backups:

Run in terminalbash

docker run --rm -v ollama_data:/data -v $(pwd):/backup alpine sh -c "rm -rf /data/* && tar xzf /backup/ollama_backup.tar.gz -C /data"
docker run --rm -v open-webui_data:/data -v $(pwd):/backup alpine sh -c "rm -rf /data/* && tar xzf /backup/open-webui_backup.tar.gz -C /data"

What Could Go Wrong with Maintenance?#

Disk Space: Ensuring sufficient disk space before updates and backups is crucial
Inconsistent State: Stopping services unexpectedly may lead to data corruption
Version Incompatibility: New versions might have breaking changes requiring configuration updates
Backup Failures: Always verify backups are complete and restorable

Troubleshooting Common Issues#

GPU Not Detected#

Verify GPU Drivers: # For NVIDIA nvidia-smi # For AMD rocm-smi
Check Container Toolkit: # For NVIDIA sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
Verify Docker Compose Configuration: Ensure the deploy section is correctly formatted.

Connection Issues Between Open WebUI and Ollama#

Check Network Configuration: Ensure the OLLAMA_API_BASE_URL is correctly set.
Test Connection: docker exec open-webui curl -s http://ollama:11434/api/version
Check Firewall Rules: If using different servers, ensure ports are open.

Performance Issues#

Check System Resources: # CPU and memory usage htop # GPU usage nvidia-smi -l 1
Use a Smaller Model: If resources are limited, try models that require less memory.
Adjust Concurrent Requests: In Open WebUI settings, lower the number of concurrent requests.

Model Loading Failures#

Check Disk Space: df -h
Manually Pull Models: docker exec -it ollama ollama pull model_name
Check Ollama Logs: docker-compose logs -f ollama

Environmental Variables Reference#

Open WebUI Environment Variables#

Variable	Description	Default
`OLLAMA_API_BASE_URL`	URL to the Ollama API	`http://ollama:11434`
`WEBUI_SECRET_KEY`	Secret key for encryption	Random generated
`WEBUI_JWT_SECRET_KEY`	Secret for JWT tokens	Random generated
`WEBUI_AUTH_ENABLED`	Enable authentication	`true`
`WEBUI_ALLOW_REGISTRATION`	Allow new user registration	`true`
`SEARXNG_BASE_URL`	URL to SearXNG for RAG	None
`OPENAI_API_KEY`	OpenAI API key	None
`WEBUI_SERVE_PORT`	Port to serve the UI	`8080`
`WEBUI_MAX_UPLOAD_SIZE`	Max file upload size	`100` (MB)

Ollama Environment Variables#

Variable	Description	Default
`OLLAMA_HOST`	Host and port to bind Ollama	`0.0.0.0:11434`
`OLLAMA_MODELS`	Models to pull at startup	None
`OLLAMA_DEBUG`	Enable debug logging	None
`OLLAMA_NUM_GPU`	Number of GPUs to use	All available

Advanced Configuration Examples#

Configuration with Multiple Models Loaded at Startup#

File contentyaml

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    environment:
      - OLLAMA_HOST=0.0.0.0:11434
      - OLLAMA_MODELS=llama3,phi3,gemma
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
    restart: unless-stopped

  open-webui:
    # Standard Open WebUI configuration

volumes:
  ollama_data:
  open-webui_data:

Configuration with Resource Limits#

File contentyaml

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        limits:
          cpus: '4'
          memory: 16G
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
              count: 1  # Use only one GPU if multiple are available
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    volumes:
      - open-webui_data:/app/backend/data
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 2G
    environment:
      - OLLAMA_API_BASE_URL=http://ollama:11434
    depends_on:
      - ollama
    restart: unless-stopped

volumes:
  ollama_data:
  open-webui_data:

Configuration with Persistent Cache Directory#

This helps speed up repeated model loading:

File contentyaml

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
      - ollama_cache:/tmp/ollama_cache
    environment:
      - OLLAMA_HOST=0.0.0.0:11434
      - XDG_CACHE_HOME=/tmp/ollama_cache
    restart: unless-stopped

  open-webui:
    # Standard Open WebUI configuration

volumes:
  ollama_data:
  open-webui_data:
  ollama_cache:

Conclusion#

This comprehensive guide covers most deployment scenarios for Open WebUI with Ollama using Docker Compose. By following these configurations and best practices, you can create a robust and secure deployment tailored to your specific requirements.

Remember that the performance of local language models depends significantly on your hardware capabilities. Regular maintenance, including updates and backups, ensures your deployment remains secure and benefits from the latest improvements to both Open WebUI and Ollama.

As these technologies continue to evolve, these configurations may need adjustments to accommodate new features and capabilities. Stay informed about updates to both projects and adapt your configurations accordingly.

Complete Guide to Installing and Using Ollama

Sat, 08 Mar 2025 03:28:21 GMT

Introduction#

Ollama is a lightweight tool that lets you run large language models (LLMs) locally on your own machine. This means you can access powerful AI capabilities without sending your data to third-party services. Ollama makes it easy to download, manage, and run various open-source models like Llama, Mistral, and many others directly on your computer.

Running models locally with Ollama offers several benefits:

Privacy: Your data stays on your machine
No internet required after initial model download
Free to use (though you need sufficient hardware)
Customizable experience

Prerequisites#

Before installing Ollama, make sure your system meets these requirements:

Operating System: macOS, Windows, or Linux
RAM: At least 8GB (16GB+ recommended for better performance)
Storage: At least 10GB of free space (more for multiple models)
CPU: 64-bit processor (some models may require AVX2 instructions)
GPU: Optional but recommended for better performance

Installation Process#

macOS Installation#

Run in terminalbash

curl -fsSL https://ollama.com/install.sh | sh

This downloads and runs the Ollama installation script. Ollama will be installed as an application.

If the command fails, check your internet connection or try downloading the installer directly from the Ollama website.

Windows Installation#

Run in terminalbash

winget install Ollama.Ollama

This uses Windows Package Manager to install Ollama. Alternatively, you can download the installer from the Ollama website.

If you get a "not found" error, make sure you have winget installed or download the .msi installer from the Ollama website.

Linux Installation#

Run in terminalbash

curl -fsSL https://ollama.com/install.sh | sh

This downloads and runs the Ollama installation script, installing Ollama on your Linux system.

If you encounter permission issues, you might need to run the command with sudo.

Running Ollama#

After installation, Ollama starts automatically as a service. You can interact with it through the command line.

Starting Ollama (if not already running)#

Run in terminalbash

ollama serve

This starts the Ollama service if it's not already running. Keep this terminal window open while using Ollama.

If you see "address already in use" errors, Ollama is likely already running.

Default Model Directories#

Ollama stores models in the following default locations:

macOS: ~/.ollama/models
Linux: ~/.ollama/models
Windows: C:\Users\<username>\.ollama\models

These directories are created automatically during installation.

Managing Models with Ollama#

Pulling Your First Model#

Run in terminalbash

ollama pull mistral

This downloads the Mistral model, a good general-purpose model that balances performance and resource usage.

Depending on your internet connection, this might take several minutes. The command will show download progress.

Pulling Latest Models#

Run in terminalbash

ollama pull llama3.2:8b

This pulls the 8 billion parameter version of Llama 3.2, one of the newer models available.

Newer models like Llama 3.2 often provide improved capabilities but may require more resources.

Listing Available Models#

Run in terminalbash

ollama list

This shows all models currently downloaded on your system.

If no models appear but you've downloaded some, check if the Ollama service is running.

Running a Model#

Run in terminalbash

ollama run mistral

This starts a chat session with the Mistral model in your terminal.

If you see errors about the model not being found, make sure you've pulled it first.

GPU Configuration#

Setting Up NVIDIA GPU Support#

For NVIDIA GPUs, ensure you have the appropriate drivers and CUDA installed:

Run in terminalbash

# Check CUDA installation
nvidia-smi

This displays your GPU information and CUDA version. Ollama works best with CUDA 11.4+.

If the command isn't found, install NVIDIA drivers and CUDA toolkit first.

Configuring GPU Memory Allocation#

Run in terminalbash

CUDA_VISIBLE_DEVICES=0 ollama run llama3:8b --gpu-layers 35

This runs the model on GPU 0 and specifies loading 35 layers onto the GPU.

Adjust the number of GPU layers based on your available VRAM. More layers mean better performance but require more memory.

Setting Up AMD GPU Support#

For AMD GPUs, ensure ROCm is installed:

Run in terminalbash

# Check ROCm installation
rocminfo

This verifies ROCm is properly installed. AMD support is more limited than NVIDIA.

If your AMD GPU isn't detected, ensure you have compatible drivers and ROCm version installed.

Intel GPU Configuration#

For Intel Arc GPUs:

Run in terminalbash

ONEAPI_DEVICE_SELECTOR=level_zero:0 ollama run mistral --gpu-layers 32

This enables Intel Arc GPU acceleration through OneAPI.

Intel GPU support is newer and may have limitations compared to NVIDIA support.

Organizing Models#

Creating a Custom Model Directory#

By default, Ollama stores models in its data directory. To use a custom location:

Run in terminalbash

mkdir -p ~/ollama-models

This creates a directory for organizing your models.

The directory won't be used by Ollama automatically; we'll need to configure it.

Configuring Custom Model Directory#

For Linux/macOS:

Run in terminalbash

export OLLAMA_MODELS=~/ollama-models

For Windows (PowerShell):

Codebash

$env:OLLAMA_MODELS="C:\path\to\ollama-models"

This tells Ollama where to store and look for models.

Note that this setting only applies to the current terminal session. Add it to your shell profile for persistence.

Security Considerations#

Restricting Network Access#

By default, Ollama only listens on localhost. To restrict access further:

Run in terminalbash

OLLAMA_HOST=127.0.0.1:11434 ollama serve

This ensures Ollama only accepts connections from the local machine.

Never expose Ollama to the public internet without proper security measures.

Configuring Firewall Rules#

For Linux (using ufw):

Run in terminalbash

sudo ufw allow from 192.168.1.0/24 to any port 11434

This allows only your local network to access Ollama.

Adjust the IP range to match your specific network configuration.

Setting Up Basic Authentication#

While Ollama doesn't have built-in authentication, you can use a reverse proxy:

Run in terminalbash

# Using nginx as a reverse proxy with basic auth
sudo apt install nginx apache2-utils
sudo htpasswd -c /etc/nginx/.htpasswd username

Then configure nginx to proxy requests to Ollama with authentication.

This adds a layer of protection when exposing Ollama on a network.

Advanced Model Management#

Removing a Model#

Run in terminalbash

ollama rm mistral

This removes the Mistral model from your system, freeing up disk space.

Make sure you're not currently using the model when removing it.

Pulling a Specific Model Version#

Run in terminalbash

ollama pull llama2:7b

This pulls a specific version of a model (in this case, the 7 billion parameter version of Llama 2).

Larger models provide better quality but require more resources.

Pulling Models from Hugging Face#

Ollama supports importing models from Hugging Face using the Modelfile approach:

Run in terminalbash

echo "FROM hf://huggingface/mistralai/Mistral-7B-v0.1" > Modelfile

This creates a Modelfile that references a Hugging Face model.

Make sure you have proper permissions to access the model if it's not public.

Creating a Model from the Modelfile#

Run in terminalbash

ollama create mistral-hf -f Modelfile

This creates a new model in Ollama based on the Hugging Face definition.

This process may take time as it downloads and converts the model.

Using Ollama With Different Interfaces#

Using the Web UI#

While Ollama doesn't include a web UI by default, you can install a compatible one:

Run in terminalbash

git clone https://github.com/ollama-webui/ollama-webui.git
cd ollama-webui
docker compose up -d

This sets up a user-friendly web interface for Ollama on http://localhost:3000.

Make sure Docker is installed on your system first.

Using Ollama API#

Ollama also provides a REST API:

Run in terminalbash

curl -X POST http://localhost:11434/api/generate -d '{
  "model": "mistral",
  "prompt": "Why is the sky blue?"
}'

This sends a prompt to Ollama's API and returns the model's response.

If you get connection errors, make sure the Ollama service is running.

Troubleshooting Common Issues#

Model Won't Download#

If a model fails to download:

Run in terminalbash

ollama pull mistral --insecure

This attempts to download using non-secure connections, which can help with certain network issues.

Only use this option if you trust the source.

Out of Memory Errors#

If you encounter out of memory errors:

Run in terminalbash

ollama run mistral --compute=cpu

This forces the model to run on CPU only, which uses less memory but runs slower.

Alternatively, try using smaller models like tinyllama or orca-mini.

Permission Denied Errors#

For Linux users encountering permission issues:

Run in terminalbash

sudo chown -R $(whoami) ~/.ollama

This ensures you have proper ownership of the Ollama directories.

Be careful with sudo commands; only use when necessary.

GPU Not Detected#

If your GPU is not being detected:

Run in terminalbash

# For NVIDIA
ollama run mistral --verbose

This shows detailed logs that can help identify GPU-related issues.

Check driver compatibility and ensure your GPU has enough VRAM for the model.

Known Bugs and Workarounds#

GPU Memory Leaks#

Some users report GPU memory not being released properly after using Ollama:

Run in terminalbash

ollama serve --compute=cpu # then restart with GPU

This runs Ollama in CPU-only mode temporarily, which can help reset GPU memory.

Restart Ollama completely if issues persist.

CLI Freezing on Windows#

Some Windows users report the CLI freezing:

Run in terminalbash

ollama run mistral --verbose

Running with verbose output can help identify where issues occur.

The Windows version of Ollama is newer and may have more bugs than macOS/Linux versions.

Model Loading Timeout#

For very large models that time out during loading:

Run in terminalbash

OLLAMA_TIMEOUT=600 ollama run llama3:70b

This increases the timeout to 10 minutes (from the default 3 minutes).

Adjust the timeout value based on your system's performance.

Maintenance#

Updating Ollama#

Run in terminalbash

ollama pull ollama

This updates the Ollama software to the latest version.

It's recommended to update regularly for bug fixes and new features.

Backing Up Your Models#

Run in terminalbash

tar -czf ollama-backup.tar.gz ~/.ollama

For Linux/macOS, this creates a compressed backup of your Ollama data directory.

For Windows, consider using a backup tool like 7-Zip to compress the Ollama directory.

Updating Models#

Run in terminalbash

ollama pull mistral:latest

This updates a model to its latest version.

Note that this downloads the model again, so make sure you have enough disk space.

Advanced Usage#

Creating Custom Models#

You can create custom models by extending existing ones:

Run in terminalbash

echo "FROM mistral
PARAMETER temperature 0.7
SYSTEM You are a helpful coding assistant." > CustomModelfile
ollama create coder -f CustomModelfile

This creates a custom model with specific parameters and system instructions.

Customizing models can help tailor them to specific tasks.

Running Models with Custom Parameters#

Run in terminalbash

ollama run mistral --temperature 0.2

This runs the model with a lower temperature setting (more focused, less creative responses).

Different parameters can significantly change how the model responds.

Exporting Models#

Run in terminalbash

ollama export mistral > mistral-exported.ollama

This exports a model to a file that can be transferred to another machine.

The exported file contains the complete model and can be large.

Importing Models#

Run in terminalbash

ollama import < mistral-exported.ollama

This imports a previously exported model.

Make sure you have enough disk space before importing.

Additional Resources#

Official Documentation and Support#

Official Documentation: Ollama Documentation
GitHub Repository: Ollama on GitHub
Community Discord: Ollama Discord Server
Issue Tracker: GitHub Issues

Model Libraries and Resources#

Ollama Library: Official Model Library
Hugging Face: Model Repository
Community Models: Ollama Community Models

These resources provide additional support, updates, and community-contributed content for Ollama users.

Ollama Tags Reference List#

Model Size Tags#

7b - 7 billion parameter models
8b - 8 billion parameter models
13b - 13 billion parameter models
34b - 34 billion parameter models
70b - 70 billion parameter models
latest - Latest version of a model

Specialization Tags#

instruct - Fine-tuned for following instructions (e.g., mistral:instruct)
chat - Optimized for conversation (e.g., llama2:chat)
code - Specialized for programming (e.g., codellama:7b)
vision - Models with image understanding capabilities
math - Models optimized for mathematical reasoning
medical - Models with medical knowledge focus
tiny - Extra small models for resource-constrained environments
uncensored - Models with fewer content restrictions

Quantization Tags#

q4_0 - 4-bit quantization, method 0 (smallest size, lowest quality)
q4_1 - 4-bit quantization, method 1 (better quality than q4_0)
q5_0 - 5-bit quantization
q5_1 - 5-bit quantization, method 1
q8_0 - 8-bit quantization (better quality, larger size)
f16 - 16-bit float precision (high quality, largest size)

Usage Examples#

llama3:8b - 8B parameter version of Llama 3
codellama:13b-instruct - 13B CodeLlama optimized for instruction following
mistral:7b-q4_0 - 7B Mistral model with 4-bit quantization for smaller size
llama2:70b-chat-q5_0 - 70B Llama 2 chat model with 5-bit quantization
gemma:2b-instruct - 2B Gemma instruction model
phi:2.7b - 2.7B Phi model

Custom Tags#

custom - User-created model variants
finetune - Models that have been fine-tuned
merged - Models created by merging multiple base models
rag - Models configured for retrieval-augmented generation

Conclusion#

Ollama provides a powerful way to run AI language models on your own hardware. With the commands in this guide, you can install, configure, and manage Ollama and its models effectively. As the project is actively developed, check the official documentation for the latest features and best practices.

Remember that running models locally requires substantial resources, especially for larger models. Start with smaller models if you're facing performance issues, and gradually explore larger ones as you become more familiar with the system.

Consider your security needs when deploying Ollama, especially in multi-user or networked environments. The privacy benefits of local AI only apply when the system is properly secured.

he system.

Installing Docker on Ubuntu: A Step-by-Step Guide 🚀(with Debugg)

Sat, 08 Mar 2025 03:13:38 GMT

Docker simplifies application deployment by packaging software into containers

Here's how to install Docker on Ubuntu efficiently:

1. Uninstall Old Versions#

Remove any previous Docker installations to avoid conflicts

Run in terminalbash

sudo apt-get remove docker docker-engine docker.io containerd runc

This ensures a clean environment for the new installation

2. Set Up Docker's Official Repository#

a. Install Prerequisite Packages

Ensure apt can use repositories over HTTPS

Run in terminalbash

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg

b. Add Docker’s GPG Key

Add Docker's official GPG key to verify package authenticity

Run in terminalbash

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo tee /etc/apt/keyrings/docker.asc > /dev/null
sudo chmod a+r /etc/apt/keyrings/docker.asc

c. Set Up the Stable Repository

Add Docker's repository to your sources list

Run in terminalbash

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Then, update the package index

Run in terminalbash

sudo apt-get update

3. Install Docker Engine#

Install Docker Engine and associated components

Run in terminalbash

sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

This command installs:

docker-ce: Docker Engine
docker-ce-cli: Command-line interface for Docker
containerd.io: Container runtime
docker-buildx-plugin: Docker Build
docker-compose-plugin: Docker Compos

4. Verify Installation#

a. Check Docker Version

Confirm the installation by checking the Docker version

Run in terminalbash

docker --version

b. Run a Test Container

Run the "Hello World" container to verify Docker is functioning correctly

Run in terminalbash

sudo docker run hello-world

This command downloads and runs a test image, outputting a confirmation message upon success

5. Manage Docker as a Non-Root User#

To run Docker commands without sudo

a. Create Docker Group

Create the Docker group if it doesn't already exist

Run in terminalbash

sudo groupadd docker

b. Add User to Docker Group

Add your user to the Docker group

Run in terminalbash

sudo usermod -aG docker $USER

c. Apply Group Changes

Log out and log back in to apply the group changes. Alternatively, use

Run in terminalbash

newgrp docker

d. Verify Non-Root Access

Run the "Hello World" container without sudo

Run in terminalbash

docker run hello-world

Code

If successful, Docker is configured for non-root usage

Troubleshooting Common Issues#

a. Permission Denied While Accessing Docker Socket

Error:
Got permission denied while trying to connect to the Docker daemon socket

Solution:

Ensure your user is added to the Docker group
Verify group membership with

Run in terminal

groups $USER

If the issue persists, adjust the ownership of the Docker directory

Run in terminal

sudo chown "$USER":"$USER" /home/"$USER"/.docker -R
sudo chmod g+rwx "$HOME/.docker" -R

b. Docker Daemon Not Running

Error:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Solution:

Start the Docker service

Run in terminalbash

sudo systemctl start docker

Enable Docker to start on boot

Run in terminalbash

sudo systemctl enable docker

c. Old Versions of Docker Installed

Issue:
Conflicts with previous Docker installations

Solution:

Uninstall old Docker versions

Run in terminal

sudo apt-get remove docker docker-engine docker.io containerd runc

Then, proceed with the installation steps above

d. Network Connectivity Issues

Error:
Failed to fetch errors during "apt-get update"

Solution:

Check your internet connection -
Ensure your system's DNS settings are correctly configured

By following these steps, you should have Docker successfully installed on your Ubuntu systemFor more detailed information, refer to the [official Docker documentation]

Reference Link: (https://docs.docker.com/engine/install/ubuntu/

🚀 Setting Up a CI/CD Pipeline with Jenkins on Ubuntu

Sat, 08 Mar 2025 03:13:06 GMT

A CI/CD (Continuous Integration and Continuous Deployment) pipeline automates software development, testing, and deployment. Jenkins is a popular open-source automation server used for CI/CD.

1. Introduction to CI/CD and Jenkins#

Continuous Integration (CI): Automates code integration, testing, and validation.
Continuous Deployment (CD): Automates software release and deployment.
Jenkins: An automation server that facilitates CI/CD pipelines.

2. Prerequisites#

Before starting, ensure you have:

✅ Ubuntu 20.04 or later
✅ sudo (administrator) access
✅ Java (JDK) installed (required for Jenkins)

Install Java (JDK 11):

Run in terminalbash

sudo apt update
sudo apt install -y openjdk-11-jdk

3. Installing Jenkins on Ubuntu#

1. Update the package index:

Run in terminalbash

sudo apt update

2. Add Jenkins repository and GPG key:

Run in terminalbash

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://pkg.jenkins.io/debian/jenkins.io-2023.key | sudo tee /etc/apt/keyrings/jenkins.asc > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/jenkins.asc] https://pkg.jenkins.io/debian-stable binary/" | sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null

3. Install Jenkins:

Run in terminalbash

sudo apt update
sudo apt install -y jenkins

4. Start and enable Jenkins:

Run in terminalbash

sudo systemctl start jenkins
sudo systemctl enable jenkins

4. Access Jenkins Web Interface 🌍#

Open http://localhost:8080 in your browser.
Retrieve the initial admin password:

Run in terminalbash

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Follow the setup wizard to install recommended plugins and create an admin user.

5. Installing Essential Jenkins Plugins#

🔹 Git Plugin: Allows Jenkins to pull code from GitHub/GitLab.
🔹 Pipeline Plugin: Enables declarative pipelines.
🔹 Docker Pipeline Plugin: Required for Docker integration.

6. Creating Your First Jenkins Pipeline#

1. Go to Jenkins Dashboard → "New Item" → Select "Pipeline".

2. Enter a name and click OK.

3. Under the "Pipeline" section, paste this script:

File contentbash

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                echo 'Building the application...'
            }
        }
        stage('Test') {
            steps {
                echo 'Running tests...'
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying the application...'
            }
        }
    }
}

7. Integrating GitHub/GitLab with Jenkins#

Modify your pipeline script to fetch code from a Git repository:

File contentbash

pipeline {
    agent any
    stages {
        stage('Checkout Code') {
            steps {
                git 'https://github.com/your-repo.git'
            }
        }
    }
}

8. Automating Build Triggers ⚙️#

Enable Poll SCM to check for repository changes at intervals.
Use GitHub Webhooks to trigger builds on new commits.

9. Deploying a Sample Web App 🌐#

1. Install Docker:

Run in terminalbash

sudo apt install -y docker.io
sudo systemctl start docker
sudo systemctl enable docker

2. Modify pipeline script to deploy an Nginx container:

File content

pipeline {
    agent any
    stages {
        stage('Deploy') {
            steps {
                sh 'docker run -d -p 8080:80 nginx'
            }
        }
    }
}

After running the pipeline, visit http://localhost:8080 to see Nginx running.

🚀 Troubleshooting Common Issues#

Jenkins not starting?

Run in terminalbash

sudo systemctl restart jenkins

Permission issues?

Run in terminalbash

sudo usermod -aG jenkins $USER

Jenkins build failing?

Run in terminalbash

sudo journalctl -u jenkins --no-pager

🎯 Conclusion#

✅ You now have a fully functional CI/CD pipeline using Jenkins.
✅ Jenkins automates testing, builds, and deployments.
✅ Next Steps: Add Kubernetes, AWS, or Terraform integrations.

📌 Reference Links#

Complete Guide to Hosting Mermaid Charts Locally

Sat, 08 Mar 2025 02:33:37 GMT

Introduction#

Mermaid.js is a powerful JavaScript library designed to create visually appealing diagrams from simple text-based definitions. It supports various diagram types such as flowcharts, sequence diagrams, Gantt charts, class diagrams, and more. Hosting Mermaid locally allows you to quickly visualize documentation, software architectures, or processes without relying on external services, ensuring privacy, security, and offline availability.

This guide covers all available options for hosting Mermaid charts locally, including direct HTML embedding, using local servers (Node.js), Docker setups, and configuration customizations. It also includes security best practices, troubleshooting steps, common issues with solutions, and maintenance procedures.

Prerequisites#

Required Software Versions:#

Node.js v16 or higher
npm or yarn or pnpm (latest stable versions)
Docker (Optional for containerized deployment)

Minimum Hardware Specifications:#

CPU: 1 Core
RAM: 512MB minimum
Storage: 500MB free space minimum

Network Requirements:#

Ports: Default ports 9000 (for local dev server), 3333 (for docs server)

User Permissions:#

Standard user with sudo privileges for installations
Avoid running as root unless necessary

Installation Methods#

You can host Mermaid locally using one of the following methods:

Option 1: Simple HTML File (No server required)#

Create an HTML file (mermaid.html) with the following content:

File contentxml

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>Mermaid Chart Local</title>
</head>
<body>
    <div class="mermaid">
        graph LR
        A[Client] --> B[Server]
    </div>

    <script type="module">
        import mermaid from 'https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.esm.min.mjs';
        mermaid.initialize({ startOnLoad: true });
    </script>
</body>
</html>

Explanation: Loads Mermaid directly from CDN and renders diagrams in-browser.

Diagnostic: Open this file in your browser. If diagrams don't render:

Check browser console (F12) for JavaScript errors.
Ensure internet connection to access CDN.

Option 2: Local Node.js Setup#

Install Mermaid via npm/yarn/pnpm:

Run in terminalbash

npm install mermaid

Run in terminalbash

yarn add mermaid

Run in terminalbash

pnpm add mermaid

Explanation: Installs Mermaid as a local dependency.

Diagnostic: Verify installation by checking node_modules/mermaid.

Create an HTML file referencing local Mermaid:

File contentxml

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>Mermaid Chart Local</title>
</head>
<body>
    <div class="mermaid">
        graph TD;
        A --> B;
        B --> C;
    </div>

    <script type="module">
        import mermaid from './node_modules/mermaid/dist/mermaid.esm.min.mjs';
        mermaid.initialize({ startOnLoad: true });
    </script>
</body>
</html>

Open this file locally in browser. If diagrams fail:

Ensure correct path to mermaid.esm.mjs.
Check browser console for errors.

Option 3: Using Docker (Recommended for consistency)#

Install Docker first if not installed:

Run in terminalbash

sudo apt update && sudo apt install docker.io -y

Explanation: Installs Docker on Ubuntu/Debian systems.

Diagnostic command:

Run in terminalbash

docker --version

Create a Dockerfile:

Codebash

FROM nginx:alpine
COPY . /usr/share/nginx/html
EXPOSE 80

Build and run your container:

Run in terminalbash

docker run -d -p 8080:80 -v $(pwd):/usr/share/nginx/html nginx:latest

Explanation: Serves your current directory via Nginx web server.

Diagnostic: Visit http://localhost:8080 in browser. If page doesn't load:

Run docker logs [container_id] to check errors.
Verify port 8080 is open (sudo lsof -i :8080).

Configuration Options#

Mermaid supports multiple configuration sources:

Global Configuration (Recommended):#

Set global diagram configuration by modifying your initialization script:

File contentxml

mermaid.initialize({
  theme: 'forest',
  securityLevel: 'strict', // options: strict(default), antiscript, loose, sandbox(beta)
  flowchart: {
    useMaxWidth: true,
    htmlLabels: true,
  }
});

Explanation: Customizes global diagram rendering settings.

Diagnostic: If diagrams fail after customization:

Reset configuration by removing custom settings temporarily.

Security Considerations#

Avoid Running as Root: Always run Mermaid under standard user permissions.
Network Access Restriction: Limit access via firewall if hosted publicly.
File Permissions: Set restrictive permissions on diagram files:

Run in terminalbash

chmod 644 /path/to/your/diagrams/*.html

Explanation: Prevents unauthorized file modifications.

Data Persistence & Backup Procedures#

For Docker setups, ensure data persistence by mounting volumes:

Run in terminalbash

docker run -d -p 8080:80 -v /path/to/data:/usr/share/nginx/html nginx:latest

Explanation: Ensures your Mermaid files persist after container restarts.

Backup regularly using:

Run in terminalbash

tar czvf mermaid_backup_$(date +%F).tar.gz /path/to/data/

Restore data easily with:

Run in terminalbash

tar xzvf backup.tar.gz -C /path/to/data/

Common Issues & Solutions#

Issue	Diagnostic	Solution
Diagrams not rendering	Browser console errors	Ensure correct Mermaid import paths; verify CDN availability
Labels out of bounds	Check font loading issues	Initialize Mermaid after DOM load completes
Security restrictions blocking clicks	Check securityLevel setting	Adjust `securityLevel` to "loose" or "antiscript"

Example fix for labels out of bounds issue:

File contentbash

document.addEventListener('DOMContentLoaded', () => {
    mermaid.initialize({ startOnLoad:true });
});

Known Open Bugs & Workarounds#

Cookiebot Interference Bug (#6045): Diagrams not rendering due to Cookiebot scripts interference. Workaround: Delay Mermaid initialization until after Cookiebot consent scripts have completed loading or disable Cookiebot temporarily during debugging.

Maintenance Guide#

Update Installation Safely:#

Update Mermaid regularly using npm/yarn/pnpm commands:

Run in terminalbash

npm update mermaid # or yarn upgrade mermaid or pnpm update mermaid

Explanation: Keeps Mermaid up-to-date with latest fixes/features.

Monitor for Issues:#

Check logs regularly (docker logs) and browser console logs periodically.

Backup Configurations & Data Regularly:#

Copy your HTML and diagram definition files periodically to a secure location:

Run in terminalbash

cp /path/to/your/files/*.html /backup/location/

This comprehensive guide ensures you can host Mermaid charts locally with ease, troubleshoot common issues effectively, maintain security best practices, and keep your data safe through regular backups and updates.

🐳 Docker Container Monitoring with Prometheus and Grafana: A Comprehensive Guide

Wed, 05 Mar 2025 03:49:47 GMT

Monitoring Docker containers is essential for maintaining healthy, high-performing containerized applications. This guide will walk you through setting up a robust monitoring system using Prometheus and Grafana, two powerful open-source tools that work seamlessly together to collect, store, and visualize metrics from your Docker environment.

🔹Understanding Container Monitoring Architecture#

Before diving into the setup, it's important to understand how the monitoring stack works. In a typical Docker monitoring setup with Prometheus and Grafana, the architecture consists of several components working together:

Prometheus serves as the central metrics collection and storage system. It scrapes metrics from various exporters at regular intervals (typically every 15 seconds) and stores them in its time-series database. Grafana connects to Prometheus as a data source and creates visualizations through dashboards. These components work with specialized exporters that collect specific metrics.

📊 Prometheus#

An open-source monitoring system that collects and stores time-series data from configured targets, providing a powerful query language for analyzing metrics.

📈 Grafana#

A visualization platform that transforms the metrics collected by Prometheus into informative and customizable dashboards.

🖥️ Node Exporter#

A Prometheus exporter for collecting hardware and OS-level metrics from the host system, such as CPU usage, memory consumption, disk I/O, and network statistics.

📦 cAdvisor (Container Advisor)#

A monitoring tool that collects, aggregates, and exports resource usage information from running containers, providing detailed insights into container performance.

🔹 Prerequisites#

Before starting the setup process, ensure you have the following installed on your system:

Docker Engine (latest stable version)
Docker Compose
Basic understanding of YAML configuration files
Access to terminal/command line interface

🔹Setting Up Your Docker Monitoring Stack#

1️⃣ Create Directory Structure#

First, establish a proper directory structure to organize your configuration files:

Run in terminalbash

mkdir -p docker-monitoring/prometheus
mkdir -p docker-monitoring/grafana/provisioning/datasources
mkdir -p docker-monitoring/grafana/provisioning/dashboards
cd docker-monitoring

2️⃣ Create Prometheus Configuration#

Create a file named prometheus.yml in the prometheus directory:

File contentyaml

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

This configuration tells Prometheus to scrape metrics from itself, cAdvisor (for container metrics), and Node Exporter (for host metrics) every 15 seconds.

3️⃣ Create Docker Compose File#

Create a docker-compose.yml file in the docker-monitoring directory:

File contentyaml

version: '3.8'

networks:
  monitoring:
    driver: bridge

volumes:
  prometheus_data: {}
  grafana_data: {}

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
    ports:
      - "9090:9090"
    networks:
      - monitoring

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($|/)'
    ports:
      - "9100:9100"
    networks:
      - monitoring

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    restart: unless-stopped
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    ports:
      - "8080:8080"
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
    ports:
      - "3000:3000"
    networks:
      - monitoring
    depends_on:
      - prometheus

This Docker Compose file defines four services: Prometheus, Node Exporter, cAdvisor, and Grafana. Each service is configured with appropriate volumes, ports, and network settings.

4️⃣ Launch the Monitoring Stack#

Start all the services using Docker Compose:

Run in terminalbash

docker-compose up -d

This command launches all four containers in detached mode. You can verify that they're running properly with:

Run in terminalbash

docker-compose ps

🔹 Configuring Grafana#

1️⃣ Access Grafana Dashboard#

Open your web browser and navigate to http://localhost:3000. You'll be greeted with the Grafana login page. Use the following credentials:

Username: admin
Password: admin

You'll be prompted to change the password on your first login for security purposes.

2️⃣ Add Prometheus as a Data Source#

Click on the gear icon (⚙️) in the left sidebar to open the Configuration menu
Select "Data sources"
Click "Add data source"
Select "Prometheus" from the list
Set the URL field to http://prometheus:9090
Click "Save & Test" to verify the connection

If everything is set up correctly, you should see a green "Data source is working" message.

3️⃣ Import Dashboards#

Click the "+" icon in the left sidebar
Select "Import"
Enter one of these dashboard IDs:
- 893 (Docker and system monitoring)
- 10619 (Docker containers)
- 1860 (Node Exporter Full)
Click "Load"
Select your Prometheus data source from the dropdown
Click "Import"

These dashboards provide comprehensive monitoring for your Docker containers and host system metrics.

🔹Monitoring Multiple Hosts#

If you need to monitor Docker containers across multiple hosts, you'll need to install Node Exporter and cAdvisor on each host machine, then update your Prometheus configuration to scrape metrics from these additional targets.

Update Prometheus Configuration#

Modify your prometheus.yml file to include the additional hosts:

File contentbash

scrape_configs:
  # Existing jobs...

  - job_name: 'node-exporter-host2'
    static_configs:
      - targets: ['host2-ip:9100']

  - job_name: 'cadvisor-host2'
    static_configs:
      - targets: ['host2-ip:8080']

After updating the configuration, reload Prometheus:

Run in terminalbash

curl -X POST http://localhost:9090/-/reload

🔹Common Issues and Troubleshooting#

⚠️ Issue 1: Grafana Can't Connect to Prometheus#

If you see a "Bad Gateway" error when trying to connect Grafana to Prometheus, check the following:

Ensure Prometheus container is running: docker ps | grep prometheus
Check Prometheus logs for errors: docker logs prometheus
Verify network connectivity: In your Docker Compose file, make sure both services are on the same network
Try using the container name instead of IP address in the Grafana data source configuration: http://prometheus:9090

⚠️ Issue 2: Prometheus Configuration Permission Denied#

If Prometheus fails to start with a permission error like "permission denied" for the configuration file:

Check the permissions of your prometheus.yml file
Ensure the file is readable by the Prometheus container
Try adding these permissions: chmod a+r prometheus/prometheus.yml

⚠️ Issue 3: cAdvisor Shows as Unhealthy#

If the cAdvisor container shows as unhealthy:

Check the logs: docker logs prometheus-cAdvisor
Verify the volumes are correctly mounted
Ensure the host system allows access to the Docker socket
Try an older or newer version of cAdvisor if the current one has issues

⚠️ Issue 4: No Data Appearing in Grafana Dashboards#

If your dashboards aren't showing any data:

Verify Prometheus is scraping metrics successfully: visit http://localhost:9090/targets
Check the query expression in Grafana's panel edit mode
Adjust the time range in Grafana to ensure it covers when data was collected
Restart the containers if needed: docker-compose restart

🔹 Advanced Tips and Best Practices#

💾 Persistence and Data Retention#

To ensure your metrics data persists across container restarts, use named volumes for Prometheus and Grafana as shown in the Docker Compose file. You can adjust Prometheus's data retention period by adding this to the command section:

File content

command:
  # Existing commands...
  - '--storage.tsdb.retention.time=15d'

This sets data retention to 15 days, but you can adjust based on your storage capacity and requirements.

🔒 Security Considerations#

Change default credentials immediately after setup
Consider using environment variables for sensitive information rather than hardcoding in files
Restrict access to the Prometheus and Grafana interfaces using a reverse proxy with authentication
Use network segmentation to isolate your monitoring stack from production services

⚡ Performance Optimization#

Adjust scrape intervals based on your needs - more frequent scraping provides higher resolution but increases resource usage
Use recording rules in Prometheus for frequently used or complex queries
Consider federation for large-scale deployments
Monitor the resource usage of your monitoring stack itself

🔔 Adding Alerts#

While this guide focuses on monitoring, you can extend your setup to include alerting:

Define alerting rules in Prometheus
Set up Alertmanager alongside Prometheus
Configure notification channels in Grafana
Create alert thresholds for critical metrics

🔹 Conclusion#

You now have a fully functional Docker monitoring stack with Prometheus and Grafana. This setup provides comprehensive visibility into the performance and health of your Docker containers and host system. By following the tips and best practices outlined in this guide, you can ensure your monitoring system remains reliable, secure, and efficient.

Remember that monitoring is an ongoing process. As your containerized applications evolve, you may need to adjust your monitoring strategy by adding more exporters, creating custom dashboards, or implementing automated alerting based on your specific requirements.

🔹 References#

Reddit: Getting started Grafana Prometheus monitoring
Reddit: What grafana dashboard you use to monitor docker
Reddit: How do I add Prometheus as a datasource to Grafana
Reddit: Docker monitoring with grafana and adding second host
Building a Docker-based Monitoring System with Prometheus and Grafana
Reddit: Prometheus not working after configuring docker named volumes
Reddit: Monitoring docker-containers with grafana, prometheus & fluent-bit
Reddit: Problems with Grafana and Prometheus Dashboard
Reddit: The Essential Guide to Grafana Docker Monitoring
Server Fault: Still can't connect from my grafana docker container to prometheus
Kubernetes Training: Prometheus & Grafana: Docker Compose Monitoring Tutorial
Grafana Community: Connection error for data source Prometheus with grafana
Dev.to: How to Set Up Grafana and Prometheus Using Docker
Grafana Community: Tutorial Grafana Fundamentals Docker Prometheus
GitHub: Docker Monitoring Workshop with Grafana Alloy, Prometheus and Loki
YouTube: Install Grafana, Prometheus & Node Exporter with Docker

🚀 An Introduction to Container Orchestration with Kubernetes

Wed, 05 Mar 2025 03:36:03 GMT

In today's cloud-native world, managing containers manually isn't practical. Kubernetes (K8s) is the most popular container orchestration platform that automates the deployment, scaling, and operation of containerized applications.

1. What is Container Orchestration?#

Containerization packages applications into lightweight, portable environments.
Orchestration automates container lifecycle management.
Tools like Kubernetes handle scheduling, scaling, and networking of containers.

2. Why Use Kubernetes?#

Kubernetes provides:

✅ Automated Scaling: Adjusts resources based on demand.
✅ Self-Healing: Automatically restarts failed containers.
✅ Load Balancing: Distributes traffic efficiently.
✅ Service Discovery: Enables easy container communication.
✅ Rolling Updates: Deploys application updates seamlessly.

3. Installing Kubernetes on Ubuntu#

To install Kubernetes, follow these steps:

1. Update System Packages:

2. Install Docker (Container Runtime):

Run in terminalbash

sudo apt install -y docker.io
sudo systemctl enable --now docker

3. Install Kubernetes Tools:

Run in terminalbash

sudo apt install -y apt-transport-https curl
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt install -y kubelet kubeadm kubectl

4. Disable Swap (Kubernetes requirement):

Run in terminalbash

sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab

5. Initialize Kubernetes Cluster:

Run in terminalbash

sudo kubeadm init --pod-network-cidr=192.168.1.0/16

Follow the output instructions to configure kubectl:

Run in terminalbash

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

6. Install a Network Plugin (e.g., Flannel):

Run in terminalbash

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

4. Understanding Kubernetes Components#

Master Node: Controls and manages worker nodes.
Worker Nodes: Run containerized applications.
Pods: The smallest unit in Kubernetes, containing one or more containers.
Services: Expose applications running on Pods.
Deployments: Define how applications should be managed and scaled.

5. Deploying a Sample App on Kubernetes#

Let's deploy an Nginx web server on Kubernetes:

1. Create a Deployment YAML file:

Run in terminalbash

nano nginx-deployment.yml

Add the following configuration:

File contentbash

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

2. Deploy the Application:

Run in terminalbash

kubectl apply -f nginx-deployment.yml

3. Check Deployment Status:

Run in terminalbash

kubectl get pods

4. Expose the Deployment:

Run in terminalbash

kubectl expose deployment nginx-deployment --type=NodePort --port=80

5. Get Access URL:

Run in terminalbash

kubectl get svc

Now, visit http://: to see the Nginx app running.

🚀 Troubleshooting Kubernetes Issues#

1. Kubernetes Pods Not Running?

Run in terminalbash

kubectl get pods -A

Check logs for errors:

Run in terminalbash

kubectl logs

2. Cluster Initialization Failed?

Ensure you have disabled swap.
Check kubeadm init logs for details.

3. Issues with Node Connectivity?

Ensure the network plugin (Flannel) is installed.
Check node status using: kubectl get nodes

🎯 Conclusion#

✅ Kubernetes automates container deployment, scaling, and networking.
✅ Master nodes manage worker nodes and workloads.
✅ Deployments and services make Kubernetes a powerful orchestrator.

📌 Reference Links#

Docker Installation on Ubuntu(Simple)

Wed, 05 Mar 2025 02:20:50 GMT

When installing Docker on Ubuntu, there are multiple options and configurations to consider both during and after installation. Here’s a comprehensive guide:

During Installation#

1. Install from the Official Ubuntu Repository (Basic)#

Run in terminal

sudo apt update
sudo apt install docker.io -y

Pros: Simple, stable.
Cons: Might not have the latest version.

2. Install from Docker’s Official Repository (Recommended)#

Run in terminal

sudo apt update
sudo apt install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo tee /etc/apt/keyrings/docker.asc > /dev/null
sudo chmod a+r /etc/apt/keyrings/docker.asc
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

Pros: Latest stable version, full feature set.
Cons: Slightly longer setup.

3. Install from a DEB Package (Offline Installation)#

Run in terminal

wget https://download.docker.com/linux/ubuntu/dists/$(lsb_release -cs)/pool/stable/amd64/docker-ce_XX.YY.ZZ~3-0~ubuntu_amd64.deb
sudo dpkg -i docker-ce_XX.YY.ZZ~3-0~ubuntu_amd64.deb

Pros: Useful for air-gapped environments.
Cons: Manual updates required.

4. Install Using Snap (Not Recommended)#

Run in terminal

sudo snap install docker

Pros: Easy installation and updates.
Cons: Can have permission issues with volume mounts.

After Installation#

1. Verify Installation#

Run in terminal

docker --version
docker info

2. Start and Enable Docker Service#

Run in terminal

sudo systemctl start docker
sudo systemctl enable docker

3. Run a Test Container#

Run in terminal

sudo docker run hello-world

4. Allow Non-root Users to Run Docker#

Run in terminal

sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker

Pros: Run Docker without sudo.
Cons: May have security risks.

5. Enable Docker to Start on Boot#

Run in terminal

sudo systemctl enable docker

6. Configure Docker Daemon Settings#

Edit the configuration file:

Run in terminal

sudo nano /etc/docker/daemon.json

Example configuration:

File content

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" },
  "storage-driver": "overlay2",
  "insecure-registries": ["myregistry.local:5000"]
}

Restart Docker:

Run in terminal

sudo systemctl restart docker

7. Check Logs#

Run in terminal

sudo journalctl -u docker --no-pager

8. Install Docker Compose#

Run in terminal

sudo apt install docker-compose-plugin
docker compose version

9. Set Up a Private Docker Registry#

Run in terminal

docker run -d -p 5000:5000 --restart=always --name registry registry:2

10. Remove Docker (if needed)#

Run in terminal

sudo apt remove docker-ce docker-ce-cli containerd.io -y
sudo rm -rf /var/lib/docker /etc/docker

Optional Configurations#

- Use Different Storage Drivers#

Run in terminal

docker info | grep "Storage Driver"

Change driver (Example: overlay2):

Run in terminal

sudo systemctl stop docker
sudo rm -rf /var/lib/docker
sudo nano /etc/docker/daemon.json
# Add:
{
  "storage-driver": "overlay2"
}

Run in terminal

sudo systemctl restart docker

- Enable IPv6#

Run in terminal

sudo nano /etc/docker/daemon.json

File content

{
  "ipv6": true,
  "fixed-cidr-v6": "2001:db8:1::/64"
}

Run in terminal

sudo systemctl restart docker

- Configure Remote API#

Run in terminal

sudo nano /lib/systemd/system/docker.service
# Modify ExecStart:
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock

Run in terminal

sudo systemctl daemon-reload
sudo systemctl restart docker

These steps cover all major aspects of Docker installation and configuration on Ubuntu.🚀