Developing Alert Email Notifications#

The email alert feature of alerting-monitor is designed to work with an external email server. This server is responsible for sending out the alert emails generated by alerting-monitor. Ensure that you have access to a suitable email server and that it is correctly configured to send emails on behalf of alerting-monitor. For development/testing purposes, you can use, for example, Mailpit tool. Keep in mind that Mailpit tool uses self-signed certificates.

Using Self-Signed Certificates with SMTP Servers in Alerting Monitor#

By default, when Alerting Monitor is configured to send alerts via an SMTP server, it will relay this to Prometheus* Alertmanager, which will attempt to verify the identity of the SMTP server using its SSL/TLS certificate. Out of the box, Prometheus Alertmanager trusts CAs that were added by Prometheus Team from Debian Linux distribution ca-certificates. This verification process involves checking that the certificate is signed by a trusted Certificate Authority (CA).

If your SMTP server is using a self-signed certificate, or a certificate signed by a non-public CA, this verification process will fail. This is because self-signed certificates and certificates signed by non-public CAs are not automatically trusted.

To use SMTP server that has a self-signed certificate or a certificate signed by a non-public CA, you need to disable the SMTP server identity verification in your Helm* values.

You can do this by changing the following line in your Helm values file:

smtp:
  initialize: true
  insecureSkipVerify: true

In this example, the insecureSkipVerify: true line disables the SMTP server identity verification.

Also, note that disabling verification makes the connection to the SMTP server less secure, as it allows connections to servers with untrusted certificates. Use this option only if you understand the risks and have no other option.

Required Secrets that need to be present in Kubernetes* cluster#

The address and port to email server (SMTP endpoint) should be stored in a Kubernetes* secret. User login and SMTP ‘From’ header should be stored in a Kubernetes secret called smtp (can be changed in values smtp.configSecret). A separate Kubernetes secret called smtp-auth is also needed with password to email server if helm value smtp.userPassword is set to true.

Below is an example of how to create these secrets:

In the /deployments/alerting-monitor/templates of the Alerting Monitor repository you need to create 2 files: smtp.yaml and smtp-auth.yaml.

Example smtp.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: smtp
  namespace: orch-infra
type: Opaque
stringData:
  smartHost: <smtp-address>
  smartPort: <smtp-port>
  from: <from>
  authUsername: <username>

Variable	Description
smtp-address	Address to SMTP server
smtp-port	Port to SMTP server
from	SMTP header that will be included in email (example <foo@bar.com>)
username	Login to external email server

Example smtp-auth.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: smtp-auth
  namespace: orch-infra
type: kubernetes.io/basic-auth
stringData:
  password: "<password>"

How to section#

AWS* environments#

To configure this functionality on AWS* environments, you must create/adjust the terraform variables.

See Configure Email Notifications for more information about configuring email notifications.

Kind environment#

Enable email notifications for alerts during the install phase

Cluster examples provided in /orch-configs/profiles/alerting-emails.yaml in the Edge Manageability Framework repository enable alert notifications by default. Users should verify this setting before proceeding with further tasks. Verify that the following section is present in the alerting-emails.yaml:
```
argo:
  o11y:
    alertingMonitor:
      smtp:
        initialize: true #alert notifications is enabled by default
        configSecret: smtp
        userPasswordAuth: true
        passwordSecret:
          name: smtp-auth
          key: password
        requireTls: true
```
Mailpit tool is included automatically to deployments that set enableMailpit: true in the orchestratorDeployment section of the cluster configuration file.

Warning

You must only enable email notifications for alerts during the install phase.
Intel does not support email notifications added during runtime.
Add required secrets to alerting-monitor templates

For the deployment to be successful, you need to provide the secrets containing SMTP server data. For more information, see Required Secrets that Need to be Present in Kubernetes* Cluster.
Make these changes in deployments/alerting-monitor/values.yaml of the Alerting Monitor repository to enable MailPit tool.
- Set devMode value to true in values depending on your deployment.
- Set smtp.initialize value to true in values depending on your deployment.
- Set smtp.userPasswordAuth value to false in values depending on your deployment.
- Set smtp.requireTls value to false in values depending on your deployment.
- Set smtp.insecureSkipVerify value to true in values depending on your deployment.
```
devMode: true

smtp:
  initialize: true
  configSecret: smtp
  userPasswordAuth: false
  passwordSecret:
    name: smtp-auth
    key: password
  requireTls: false
  insecureSkipVerify: true
```

In this example, the insecureSkipVerify: true line disables the SMTP server identity verification.

Accessing MailPit Tool#

The following command is used to access the UI of MailPit tool:

kubectl port-forward -n mailpit-dev service/mailpit-svc 1080:8025 --address 0.0.0.0

The UI is available on <HOST_IP>:1080

Turn off email notifications#

To turn off email notifications for alerts, you need to remove /orch-configs/profiles/alerting-emails.yaml from the target cluster definition.