Tag Archives: Docker

Docker, Logging and Monitoring

Docker Monitoring

Docker Node Exporter Dashboard

Many of our applications and services run as Docker containers. Our monitoring solution covers two aspects of Docker container performance:

These data collectors enable several Grafana dashboards that help us to manage our Docker cluster.

Monitoring Setup

We run a combination of Node Exporter and cAdvisor on each Docker host VMs. These containers scrape data for our Docker hosts and feed it to the Prometheus instance in our Docker stack.

The following video explains how all of this is set up –

Setting up Docker Server Monitoring – Node Explorer and cAdvisor

Dashboards

We are using several dashboards to implement our Docker monitoring solution.

Docker Node Summary

Docker Host Summary Dashboard

We are using a modified version of the Grafana Dashboard above to monitor the overall performance of our Docker nodes.

Docker Node Details

Docker Host Details

We are using a modified version of the Grafana Dashboard above to monitor and enable drilling into detailed performance metrics for our Docker nodes.

Docker Container Summary

Docker Container Summary Dashboard

We are using a modified version of the Grafana Dashboard above to monitor and enable a summary view of the containers in our Docker cluster.

Docker Container Details

Docker Container Details Dashboard

We are using a modified version of the Grafana Dashboard above to monitor and enable drilling into the detailed performance of containers in our Docker cluster.

Windows VM Dashboard

Windows VM Dashboard
Windows VM Dashboard

We are using a modified version of the Grafana Dashboard above to monitor and enable drilling into the performance of Windows VMs in our Docker cluster.

Docker, Logging and Monitoring, Server

Proxmox Monitoring

Proxmox Cluster Metrics
Proxmox Cluster Metrics

We set up a Grafana Dashboard to monitor our Proxmox Cluster. The main components in this monitoring stack include:

The following sections cover the setup and configuration of our Proxmox monitoring stack.

Set Up and Configuration

The following video explains how to set up a Grafana dashboard for Proxmox. This installation uses the Proxmox monitoring function to feed data to Influx DB.


Monitoring Proxmox with Grafana

And here is a video that explains setting up self-signed certificates –


Configuring Self-Signed Certificates

We are using the Proxmox [Flux] dashboard with our setup.

Docker, Logging and Monitoring

Grafana Logging and Monitoring

We’ve added a Monitoring and Logging system to our Home Lab. The system is based on Grafana, Prometheus, Grafana Loki, Promtail, Telegraf, and InFlux DB.

Installation

The following video covers the installation of our monitoring stack.


Setup Logging and Monitoring in Docker

Configure Loki and Promtail

Grafana Lofi and Promtail work together to scape and store log data. These tools can scrape docker data and accept syslog data as well. The following video explains how to configure Loki and Promtail.


Configure Grafana Loki and Promtail for logs

There are a few details that we needed to do differently than the video:

  • We had to configure a tsdb schema for Loki
  • The links for configuring the Loki Docker driver can be found here and here.
  • Set parameters in the Loki Docker driver via /etc/docker/daemon.json to avoid blocking the Docker.
  • Recreating containers with Portainer does not enable Lofi to access their logs. To make this work, we needed to use docker compose up -d –force-recreate

The contents of /etc/docker/daemon.json are as follows:

{
  "log-driver": "loki",
  "log-opts": {
     "loki-url": "http://localhost:3100/loki/api/v1/push",
      "loki-batch-size": "400",
      "loki-retries": "2",
      "loki-max-backoff": "800ms",
      "loki-timeout": "1s",
      "keep-file": "true",
      "mode": "non-blocking"
    }
}

Syslog

We have configured a combination of Loki and Promtail to accept Syslog events. Promtail does not support Syslog events using the UDP protocol. To solve this problem, we set up rsyslog running under the Ubuntu system, which hosts the Promtail Docker container, to consolidate and forward all Syslog events as a front end to Promtail. Information on configuring rsyslog as a front end to Promtail can be found here.

Monitoring Dashboards

The following video provides some information on configuring dashboards and other monitoring capabilities.


Create and Configure Grafana Dashboards

Docker, Software Development

VS Code Server

We do a variety of software development and Java coding tasks. To make this easier and more accessible from all our computers, we will try VS Code and VS Code Server.

VS Code Server allows editing using a web browser on any computer. The VS Code web interface is hosted from a server running in a Docker container.

Installation and Set Up

The following video explains how to set up VS Code Server and connect it to a GitHub repository.


VS Code Server Installation and Set Up

VS Code Extensions

The following video suggests several useful VS Code plugin extensions.


VS Code Server With Extensions

Docker, Network

Nginx Proxy Manager

Many services and devices in our home lab have web interfaces. Traefik Reverse Proxy provides ingress control and SSL certificates for our docker services.

While Traefik can be used for services outside of Docker, configuring it is complex and requires restarting the Trafik container. As a result, we also run Nginx Proxy Manager in a container to enable SSL certificates and simple reverse proxy configuration of our web-based services outside of Docker.

Installation

Installing Nginx Proxy Manager is easy. The following video explains the process, including using a DNS-01 challenge to obtain SSL certificates via Let’s Encrypt.


Installing Nginx Proxy Manager in Docker

We configured a Docker macVLAN network for the Nginx Proxy Manager container so that the proxy could determine the source IP addresses that access it. This enables IP filtering and other features.

Docker, Network

Pihole with a Cloudflare Tunnel

PiHole in Docker

We are running three PiHole installations, which enable load balancing and high availability for our DNS services. We also use a Cloudflare encrypted tunnel to protect information in external DNS queries via the Internet.

Our PiHole instances are deployed on a combination of Docker host VMs in our Proxmox Cluster and a stand-alone Raspberry Pi Docker host.

Deploy PiHole with a Cloudflare Tunnel

Our software service stack for our dockerPiHole installs Pi includes the following applications:

Our combined stack was created using  information in the following video:


Deploy PiHole with Cloudflare Tunnel in Docker

Ubuntu Port 53 Fix

Unubtu VMs include a DNS caching server on port 53, which prevents Pihole from being deployed. To fix this, run the commands at this link on the host Ubuntu VM before installing the Pihole and Cloudflare Tunnel containers.

Scheduled Block List Updates

We must update our piHole block list by doing a Gravity pull. We do this daily via a cron job. This can be configured on the RPi host using the following commands –

# Edit the user crontab
sudo crontab -u <user-id> -e

# The following to the user crontab
min hr * * * su ubuntu -c /usr/bin/docker exec pihole pihole -g | /usr/bin/mailx -s"RPi Docker - Gravity Pull" [email protected]
Docker

CrowdSec

CrowdSec provides community-based security. It works alongside our Traefik installation to detect and remediate security events.

We combined the information in the following two videos to set up Crowdsec on our Docker infrastructure.


Install CrowdSec in Docker


A more advanced install

We set up a cron job to update CrowdSec. The commands to do this  are as follows:

docker exec crowdsec cscli hub update
docker exec crowdsec cscli hub upgrade

We also set up an online account to view our dashboard.

Docker

Watchtower Container Update

We are running the Watchtower container on all our stand-alone docker hosts to keep our containers up to date. The following video explains how to install and configure Watchtower.


Install and Configure Watchtower on Docker

We have Watchtower configured to detect and notify us about updated container images. We install these manually using Protainer.

Docker, Network

Cloudflare DDNS

We use Cloudflare to host our domains and the associated external DNS records. Cloudflare provides excellent security and scaling features and is free for our use cases.

We do not have a static IP address from either of our ISPs. This, coupled with the potential of a failover from our primary to our secondary ISP, requires us to use DDNS to keep the IPs for our domains up to date in Cloudflare’s DNS.

We run a docker container for each domain that periodically checks to see if our external IP address has changed and updates our DNS records in Cloudflare.  The repository for this container can be found here.

Deploying the DDNS update container is done via a simple docker compose yml –

version: '2'
services:
  cloudflare-ddns:
    image: oznu/cloudflare-ddns:latest
    restart: unless-stopped
    container_name: your-container-name
    environment:
        - API_KEY=YOUR-CF-API-KEY
        - ZONE=yourdomain.com
        - PROXIED=true
        # Runs every 5 minutes
        - CRON=*/5 * * * *

You’ll need a separate container for each DNS Zone you host on Cloudflare.