Setup server alerts using webhooks

If you’re using self-hosted servers, you might have run into this (or similar) blog which covers most of the things you need to do on your first login to the server.

Over the past weeks, I’ve failed to setup mail alerts on ssh login, sudo, and other events due to the various cloud providers blocking the SMTP ports for security reasons and making it difficult to setup a Mail Transfer Agent (MTA) quickly¹.

Slack alerts seemed to be the next logical step, and it takes considerably less time to setup². We’ll be leveraging Unix systems’ Pluggable Authentication Module (PAM) – which can be configured under /etc/pam.d – to setup slack alert on ssh login and logout events.

The following steps shall guide you to easily setup the same and maybe adapt the process to other services like discord, telegram, or what have you.

• Setup incoming webhook in slack
  • Follow the instructions under the Getting Started section on slack’s webhook documentation for creating an app and tieing it to a #channel under your desired workspace. This should land you with a Webhook URL.
• We’ll use the following script which sends a POST request with the details (IP ADDRESS, HOSTNAME) on either open_session (login) or close_session (logout) event as payload to the WEBHOOK URL.

#!/bin/bash

WEBHOOK_URL="<WEBHOOK_URL>"
CHANNEL="#<CHANNEL_NAME>"
HOST="$(hostname)"

if [ "$PAM_TYPE" == "open_session" ] || [ "$PAM_TYPE" == "close_session" ]; then
    content="\"attachments\": [{ 
        \"mrkdwn_in\": [\"text\", \"fallback\"], 
        \"fallback\": \"Event : $PAM_TYPE to \`$HOST\`\", 
        \"text\": \"SSH: $PAM_TYPE to \`$HOST\`\", 
        \"fields\": [ { 
                \"title\": \"User\", 
                \"value\": \"$PAM_USER\", 
                \"short\": true 
            }, { 
                \"title\": \"IP Address\", 
                \"value\": \"$PAM_RHOST\", 
                \"short\": true 
        } ],
        \"color\": \"#f30c00\" 
    }]"
    curl -X POST --data-urlencode \
        "payload={
                \"channel\": \"$CHANNEL\",
                \"mrkdwn\": true, 
                \"username\": \"SSH Notifications\", 
                $content, 
                \"icon_emoji\": \":warning:\"}" \
        "$WEBHOOK_URL" &
fi
exit

• You can name the script anything you want and place it anywhere; For this example, I’ve placed it in /usr/local/sbin/ssh-slack

• Make the script executable.

$ chmod +x /usr/local/sbin/ssh-slack

• One of the modules of PAM - pam_exec.so helps us trigger the scripts based on various authentication events³. We’ll add the path to our script under /etc/pam.d/sshd which will trigger our script on any ssh authentication-related events.

$ sudo echo "session   optional   pam_exec.so   /usr/local/sbin/ssh-slack" >> /etc/pam.d/sshd

• That’s It! You should have the slack alerts working now.

Note

• The process described isn’t limited to ssh authentication-related events; We can configure it to work with other deamons like fail2ban and others to setup alerts or send logs to services on critical events.