用 Prometheus 实现系统监控报警邮件通知

804 03-03

报警流程

Prometheus 配置监控规则
监控对象触发阈值
阈值超出持续时间
推送告警到 Alertmanager
Alertmanager 处理告警信息 1）分组(group)：类似告警合并为一个通知。2）静默(silences)：不通知，系统升级时使用。3）抑制(inhibition): 只通知一次，相同内容不再通知。
Alertmanager 发送通知到媒体，邮箱，钉钉，企业微信等接收到通知

安装部署 AlertManager

部署 alertermanager

下载二进制文件

wget https://github.com/prometheus/alertmanager/releases/download/v0.24.0/alertmanager-0.24.0.linux-amd64.tar.gz
tar zxvf alertmanager-0.24.0.linux-amd64.tar.gzmv alertmanager-0.24.0.linux-amd64 /apps/alertmanager

创建 alertermanager 服务

vim /etc/systemd/system/alertmanager.service
[Unit]Description=alertmanagerDocumentation=https://prometheus.io/After=network.target
[Service]User=rootType=simple#不能有单引号和双引号ExecStart=/home/prometheus/alertmanager/alertmanager/alertmanager --config.file=/home/prometheus/alertmanager/alertmanager/alertmanager.yml --storage.path=/home/prometheus/alertmanager/alertmanager/data --web.listen-address=:19093 --cluster.listen-address=0.0.0.0:19094 --web.external-url=http://192.168.1.108:19093Restart=on-failure[Install]WantedBy=multi-user.target

启动服务：

systemctl daemon-reloadsystemctl enable --now alertmanagersystemctl status alertmanager

访问 192.168.1.108:19093 为 alertmanager 管理页面：

Alertmanager 配置

配置文件详解，以邮箱告警为例：

vim /home/prometheus/alertmanager/alertmanager/alertmanager.yml#邮件发送者global:  resolve_timeout: 30s  smtp_smarthost: 'smtp.qq.com:465'   smtp_from: '809xxx59@qq.com'   smtp_auth_username: '80xxx4859@qq.com'   smtp_auth_password: 'xxxxxxxxvpobcee'  smtp_hello: '@qq.com'  smtp_require_tls: false
templates:  - '/home/prometheus/alertmanager/alertmanager/tmpl/email.tmpl'  #增加templates配置route:  group_by: ['alertname']  group_wait: 30s  group_interval: 5m  repeat_interval: 5m  receiver: 'email'  routes:  - receiver: dingtalk-webhook    group_wait: 10s  - receiver: email    group_wait: 10sreceivers:  - name: 'email'    email_configs:      - to: 'niuming@dync.cc'        send_resolved: trueinhibit_rules:  - source_match:      severity: 'critical'    target_match:      severity: 'warning'    equal: ['alertname', 'dev', 'instance']

项目 Value

Prometheus 规则

新建规则文件，配置分组信息，告警阈值和时间，告警标签和注释等。

指标表达式采用 PromQL 语句，多数指标单位为 bytes 字节，需要转换成 KMG，例如 2M=210241024。

Prometheus 规则文件，对于邮箱，钉钉或企业微信，该文件通用：

vim /home/prometheus/prometheus/rule/qtalk_auth.yamlgroups: - name: qtalk_auth 程异常退出   rules:   - alert: 应用进程 qtalk_auth 异常退出 # 告警名称     expr: (namedprocess_namegroup_num_procs{groupname="map[:qtalk_auth]"}) == 0     for: 30s # 满足告警条件持续时间多久后，才会发送告警     labels: #标签项        severity: error        ip: 192.168.1.108     annotations: # 解析项，详细解释告警信息         summary: "进程异常报警 Alert {{ $labels.instance }} ，异常停止超过30秒."         description: "{{$labels.ip}} 进程{{$labels.groupname}} 异常停止！请立即查看!"

检验 prometheus 报警规则文件，显示 SUCCESS：

/home/prometheus/prometheus/promtool check rules rule/qtalk_auth.ymlChecking rule/qtalk_auth.yml  SUCCESS: 1 rules found

Prometheus 配置

配置 Prometheus 文件，alertmanagers 服务器的 IP 和端口，prometheus 服务器规则文件的路径：

vim /home/Prometheus/prometheus/prometheus.yml# my global configglobal:  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.  # scrape_timeout is set to the global default (10s).
# Alertmanager configurationalerting:  alertmanagers:    - static_configs:        - targets: ["192.168.1.108:19093"]           #- alertmanager:["192.168.1.108:19093"]
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.rule_files: - "rule/*.yml"  # - "first_rules.yml"  # - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:# Here it's Prometheus itself.scrape_configs:  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.  - job_name: "prometheus"
    # metrics_path defaults to '/metrics'    # scheme defaults to 'http'.
    static_configs:      - targets: ["localhost:9090"]
  - job_name: 'process'    static_configs:      - targets: ['192.168.1.108:9256']