Category Archives: Logging and Monitoring

Information on Logging and Monitoring tools.

Docker Monitoring

Docker Node Exporter Dashboard

Many of our applications and services run as Docker containers. Our monitoring solution covers two aspects of Docker container performance:

These data collectors enable several Grafana dashboards that help us to manage our Docker cluster.

Monitoring Setup

We run a combination of Node Exporter and cAdvisor on each Docker host VMs. These containers scrape data for our Docker hosts and feed it to the Prometheus instance in our Docker stack.

The following video explains how all of this is set up –

Setting up Docker Server Monitoring – Node Explorer and cAdvisor

Dashboards

We are using several dashboards to implement our Docker monitoring solution.

Docker Node Summary

Docker Host Summary Dashboard

We are using a modified version of the Grafana Dashboard above to monitor the overall performance of our Docker nodes.

Docker Node Details

Docker Host Details

We are using a modified version of the Grafana Dashboard above to monitor and enable drilling into detailed performance metrics for our Docker nodes.

Docker Container Summary

Docker Container Summary Dashboard

We are using a modified version of the Grafana Dashboard above to monitor and enable a summary view of the containers in our Docker cluster.

Docker Container Details

Docker Container Details Dashboard

We are using a modified version of the Grafana Dashboard above to monitor and enable drilling into the detailed performance of containers in our Docker cluster.

Windows VM Dashboard

Windows VM Dashboard
Windows VM Dashboard

We are using a modified version of the Grafana Dashboard above to monitor and enable drilling into the performance of Windows VMs in our Docker cluster.

Proxmox Monitoring

Proxmox Cluster Metrics
Proxmox Cluster Metrics

We set up a Grafana Dashboard to monitor our Proxmox Cluster. The main components in this monitoring stack include:

The following sections cover the setup and configuration of our Proxmox monitoring stack.

Set Up and Configuration

The following video explains how to set up a Grafana dashboard for Proxmox. This installation uses the Proxmox monitoring function to feed data to Influx DB.


Monitoring Proxmox with Grafana

And here is a video that explains setting up self-signed certificates –


Configuring Self-Signed Certificates

We are using the Proxmox [Flux] dashboard with our setup.

Grafana Logging and Monitoring

We’ve added a Monitoring and Logging system to our Home Lab. The system is based on Grafana, Prometheus, Grafana Loki, Promtail, Telegraf, and InFlux DB.

Installation

The following video covers the installation of our monitoring stack.


Setup Logging and Monitoring in Docker

Configure Loki and Promtail

Grafana Lofi and Promtail work together to scape and store log data. These tools can scrape docker data and accept syslog data as well. The following video explains how to configure Loki and Promtail.


Configure Grafana Loki and Promtail for logs

There are a few details that we needed to do differently than the video:

  • We had to configure a tsdb schema for Loki
  • The links for configuring the Loki Docker driver can be found here and here.
  • Set parameters in the Loki Docker driver via /etc/docker/daemon.json to avoid blocking the Docker.
  • Recreating containers with Portainer does not enable Lofi to access their logs. To make this work, we needed to use docker compose up -d –force-recreate

The contents of /etc/docker/daemon.json are as follows:

{
  "log-driver": "loki",
  "log-opts": {
     "loki-url": "http://localhost:3100/loki/api/v1/push",
      "loki-batch-size": "400",
      "loki-retries": "2",
      "loki-max-backoff": "800ms",
      "loki-timeout": "1s",
      "keep-file": "true",
      "mode": "non-blocking"
    }
}

Syslog

We have configured a combination of Loki and Promtail to accept Syslog events. Promtail does not support Syslog events using the UDP protocol. To solve this problem, we set up rsyslog running under the Ubuntu system, which hosts the Promtail Docker container, to consolidate and forward all Syslog events as a front end to Promtail. Information on configuring rsyslog as a front end to Promtail can be found here.

Monitoring Dashboards

The following video provides some information on configuring dashboards and other monitoring capabilities.


Create and Configure Grafana Dashboards