Skip to content

Monitoring

There are plenty of monitoring and observability tools available on the market, both as commercial services and self-hosted open-source projects. However, many of these tools are quite complex and require substantial resources to run. In this chapter, we will create our own monitoring service that is as simple as a single script file. But first, let’s outline the requirements for our monitoring service:

  • Monitor CPU, RAM, and disk usage, and notify the user if usage exceeds specified thresholds.
  • Ensure apps/services are not down. If something goes down, notify the user.
  • Autostart after reboot.
  • Minimal resource usage.
  • Send notifications via email or Telegram.

Monitoring Service

Our monitoring service will consist of just one bash script. Don’t worry if you don’t understand every line of the script—just copy it and replace the specified values.

The first question is how to determine if an app or service is down. There are three options:

  • Send a request to a special endpoint (e.g., turbocloud.dev/health or localhost:6000/health if you are sending the request from the same server where the project is hosted).
  • If using Docker to manage projects on the server, ask Docker if a container is online
  • If using systemd to manage projects on the server, ask systemctl whether the app/service is active.

The first option works best for projects you developed yourself, as you can easily add an endpoint. The Docker and systemd options work well for any project, even if you don’t understand how to add a health check endpoint. We will cover all three options.

How to Start Monitoring

1. Create a Monitoring Script

Choose the option that suits your deployments, copy the corresponding script into any text editor, replace the parameter values with your own, and proceed to the next step.

Monitoring with Health Check Endpoint

Every n seconds, the monitoring service sends a request to a predefined endpoint and checks the response. The request should be as lightweight as possible, which is why a HEAD request is typically used. If the monitoring service receives a 200 response, the app/service is working fine; otherwise, it sends a notification via email, Telegram, or another method. Here’s an example of such a script:

#!/bin/sh

URL="your-health-url"

echo "Checking $URL .."
status_code=$(curl --write-out '%{http_code}' --silent  --output /dev/null "$URL")

if [[ $status_code -ge 200 && $status_code -le 299 ]]; then
    echo -e "\x1B[32m✅ OK - Status code: $status_code for domain $URL  \x1B[0m"
    exit 0
fi

if [[ $status_code -ge 300 && $status_code -le 399 ]]; then
    echo -e "\x1B[33m⚠️ Warning - Redirection - Code: $status_code for URL: $URL \x1B[0m"
    exit 1
fi

echo "Error - Status code: $status_code for URL: $URL" >> invalid_urls.txt
echo -e "\x1B[31m⛔ Error - Status code: $status_code for URL: $URL \x1B[0m"
exit 2

Monitoring with Docker

This option assumes that your projects are deployed as Docker containers and managed by Docker. The monitoring service will check whether the container is running. If not, it will send a notification.

#!/bin/sh

container_name="your-container-name"

# Check if the container exists
if docker inspect "$container_name" > /dev/null 2>&1; then
    echo "The container $container_name exists."

    # Check if the container is running
    if $(docker inspect -f '{{.State.Status}}' "$container_name" | grep -q "running"); then
        echo "The container $container_name is running."
    else
        echo "The container $container_name is not running."

        # Start the container if it is not running
        docker start "$container_name"
    fi
else
    echo "The container $container_name does not exist."

    # Create and start the container if it does not exist
    docker run -d --name "$container_name" your_image_name
fi

Monitoring with systemd

If you manage projects on your server with systemd, this option will check whether a systemd-managed service is active. If the service is not active, it sends a notification.

#!/bin/sh

service_name="your-service-name"

STATUS="$(systemctl is-active $service_name)"
if [ "${STATUS}" = "active" ]; then
    echo "Service $service_name is running."
else 
    echo "Service $service_name is not running."
    exit 1  
fi

It may sound a bit unconventional, but with just one bash script, you can notify yourself of various events happening on your server—whether something is down, resource usage is high, or errors appear in the logs. For most small to medium-sized projects, this will be sufficient.

2. Starting the Monitoring Script with systemd

The next question is how to run the monitoring script at regular intervals. The most reliable option on any Linux system is to use systemd timers, which can start programs or scripts on a schedule (e.g., every minute, daily, etc.). Here’s how to set it up:

  • SSH into the server:
ssh root@ip_address_of_server
  • Create a file for the monitoring script:

Copy your script from the step 1 (with the appropriate values) into a file on the server.

cat > ~/monitoring.sh <<EOL
script_contents
EOL
  • Create a systemd service:
cat > /etc/systemd/system/monitoring.service <<EOL
# Monitoring Service

[Unit]
Description=Monitoring Service
Wants=monitoring.timer

[Service]
Type=oneshot
ExecStart=~/monitoring.sh

[Install]
WantedBy=multi-user.target
EOL
  • Create a timer to start the service based on a schedule (e.g., every 60 seconds):
cat > /etc/systemd/system/monitoring.timer <<EOL
# Monitoring Timer

[Unit]
Description=Monitoring Timer
Requires=monitoring.service

[Timer]
Unit=monitoring.service
OnCalendar=*-*-* *:*:00

[Install]
WantedBy=timers.target
EOL
  • Start the service and timer:
systemctl start monitoring.service
  • Test notifications:

Verify that you receive notifications when events occur, such as an app/service going down. If everything is set up correctly, you should receive notifications via email or Telegram.