15.Prometheus 2.0: Новый формат Alert rules

Thank you for reading this post, don't forget to subscribe!

Как признаются сами разработчики:

Alerting and recording rules are one of the critical features of Prometheus

С этим функционалом связано несколько узких мест / багов, а именно:

все правила (rules) запускаются с одним и тем же интервалом;
все правила выполняются одновременно;
написание правил требует от пользователя знания еще одного DSL.

Перевод rules-файлов в .YML-формат — часть улучшений, направленных на устранение озвученных проблем.

Рассмотрим изменение синтаксиса на примере файла containers.rules, ранее он выглядел так:

[codesyntax lang="php"]

ALERT high_cpu_usage_on_container
  IF sum(rate(container_cpu_usage_seconds_total{name=~".+"}[1m])) by (name,host) * 100 &gt; 70
  FOR 5m
  LABELS { severity = "warning" }
  ANNOTATIONS {
      summary = "High CPU usage: CONTAINER '{{ $labels.name }}' on '{{ $labels.host }}'",
      description = "{{ $labels.name }} is using a LOT of CPU. CPU usage is {{ humanize $value}}%.",
  }
 
ALERT graylog_eating_memory
  IF sum(container_memory_rss{name=~"graylog"}) by (name) &gt; 1500000000
  FOR 5m
  LABELS { severity = "warning" }
  ANNOTATIONS {
      summary = "High memory usage: CONTAINER '{{ $labels.name }}' on '{{ $labels.instance }}'",
      description = "{{ $labels.name }} is eating up a LOT of memory. Memory consumption of {{ $labels.name }} is at {{ humanize $value}}.",
  }
 
ALERT elasticsearch_eating_memory
  IF sum(container_memory_rss{name=~"elasticsearch"}) by (name) &gt; 1500000000
  FOR 5m
  LABELS { severity = "warning" }
  ANNOTATIONS {
      summary = "High memory usage: CONTAINER '{{ $labels.name }}' on '{{ $labels.instance }}'",
      description = "{{ $labels.name }} is eating up a LOT of memory. Memory consumption of {{ $labels.name }} is at {{ humanize $value}}.",
  }
 
ALERT prometheus_eating_memory
  IF sum(container_memory_rss{name=~"prometheus"}) by (name) &gt; 1200000000
  FOR 5m
  LABELS { severity = "warning" }
  ANNOTATIONS {
      summary = "High memory usage: CONTAINER '{{ $labels.name }}' on '{{ $labels.instance }}'",
      description = "{{ $labels.name }} is eating up a LOT of memory. Memory consumption of {{ $labels.name }} is at {{ humanize $value}}.",
  }

ALERT high_cpu_usage_on_container

IF sum(rate(container_cpu_usage_seconds_total{name=~".+"}[1m])) by (name,host) * 100 > 70

FOR 5m

LABELS { severity = "warning" }

ANNOTATIONS {

summary = "High CPU usage: CONTAINER '{{ $labels.name }}' on '{{ $labels.host }}'",

description = "{{ $labels.name }} is using a LOT of CPU. CPU usage is {{ humanize $value}}%.",

}

ALERT graylog_eating_memory

IF sum(container_memory_rss{name=~"graylog"}) by (name) > 1500000000

FOR 5m

LABELS { severity = "warning" }

ANNOTATIONS {

summary = "High memory usage: CONTAINER '{{ $labels.name }}' on '{{ $labels.instance }}'",

description = "{{ $labels.name }} is eating up a LOT of memory. Memory consumption of {{ $labels.name }} is at {{ humanize $value}}.",

}

ALERT elasticsearch_eating_memory

IF sum(container_memory_rss{name=~"elasticsearch"}) by (name) > 1500000000

FOR 5m

LABELS { severity = "warning" }

ANNOTATIONS {

summary = "High memory usage: CONTAINER '{{ $labels.name }}' on '{{ $labels.instance }}'",

description = "{{ $labels.name }} is eating up a LOT of memory. Memory consumption of {{ $labels.name }} is at {{ humanize $value}}.",

}

ALERT prometheus_eating_memory

IF sum(container_memory_rss{name=~"prometheus"}) by (name) > 1200000000

FOR 5m

LABELS { severity = "warning" }

ANNOTATIONS {

summary = "High memory usage: CONTAINER '{{ $labels.name }}' on '{{ $labels.instance }}'",

description = "{{ $labels.name }} is eating up a LOT of memory. Memory consumption of {{ $labels.name }} is at {{ humanize $value}}.",

}

[/codesyntax]

В новом варианте файл должен называться containers.rules.yml, а содержимое его будет следующим:

[codesyntax lang="php"]

groups:
- name: /var/www/monitor/prometheus/containers.rules
  rules:
  - alert: high_cpu_usage_on_container
    expr: sum(rate(container_cpu_usage_seconds_total{name=~".+"}[1m])) BY (name, host)
      * 100 &gt; 70
    for: 5m
    labels:
      severity: warning
    annotations:
      description: '{{ $labels.name }} is using a LOT of CPU. CPU usage is {{ humanize
        $value}}%.'
      summary: 'High CPU usage: CONTAINER ''{{ $labels.name }}'' on ''{{ $labels.host
        }}'''
  - alert: graylog_eating_memory
    expr: sum(container_memory_rss{name=~"graylog"}) BY (name) &gt; 1.5e+09
    for: 5m
    labels:
      severity: warning
    annotations:
      description: '{{ $labels.name }} is eating up a LOT of memory. Memory consumption
        of {{ $labels.name }} is at {{ humanize $value}}.'
      summary: 'High memory usage: CONTAINER ''{{ $labels.name }}'' on ''{{ $labels.instance
        }}'''
  - alert: elasticsearch_eating_memory
    expr: sum(container_memory_rss{name=~"elasticsearch"}) BY (name) &gt; 1.5e+09
    for: 5m
    labels:
      severity: warning
    annotations:
      description: '{{ $labels.name }} is eating up a LOT of memory. Memory consumption
        of {{ $labels.name }} is at {{ humanize $value}}.'
      summary: 'High memory usage: CONTAINER ''{{ $labels.name }}'' on ''{{ $labels.instance
        }}'''
  - alert: prometheus_eating_memory
    expr: sum(container_memory_rss{name=~"prometheus"}) BY (name) &gt; 1.2e+09
    for: 5m
    labels:
      severity: warning
    annotations:
      description: '{{ $labels.name }} is eating up a LOT of memory. Memory consumption
        of {{ $labels.name }} is at {{ humanize $value}}.'
      summary: 'High memory usage: CONTAINER ''{{ $labels.name }}'' on ''{{ $labels.instance
        }}'''

groups:

- name: /var/www/monitor/prometheus/containers.rules

rules:

- alert: high_cpu_usage_on_container

expr: sum(rate(container_cpu_usage_seconds_total{name=~".+"}[1m])) BY (name, host)

* 100 > 70

for: 5m

labels:

severity: warning

annotations:

description: '{{ $labels.name }} is using a LOT of CPU. CPU usage is {{ humanize

$value}}%.'

summary: 'High CPU usage: CONTAINER ''{{ $labels.name }}'' on ''{{ $labels.host

}}'''

- alert: graylog_eating_memory

expr: sum(container_memory_rss{name=~"graylog"}) BY (name) > 1.5e+09

for: 5m

labels:

severity: warning

annotations:

description: '{{ $labels.name }} is eating up a LOT of memory. Memory consumption

of {{ $labels.name }} is at {{ humanize $value}}.'

summary: 'High memory usage: CONTAINER ''{{ $labels.name }}'' on ''{{ $labels.instance

}}'''

- alert: elasticsearch_eating_memory

expr: sum(container_memory_rss{name=~"elasticsearch"}) BY (name) > 1.5e+09

for: 5m

labels:

severity: warning

annotations:

description: '{{ $labels.name }} is eating up a LOT of memory. Memory consumption

of {{ $labels.name }} is at {{ humanize $value}}.'

summary: 'High memory usage: CONTAINER ''{{ $labels.name }}'' on ''{{ $labels.instance

}}'''

- alert: prometheus_eating_memory

expr: sum(container_memory_rss{name=~"prometheus"}) BY (name) > 1.2e+09

for: 5m

labels:

severity: warning

annotations:

description: '{{ $labels.name }} is eating up a LOT of memory. Memory consumption

of {{ $labels.name }} is at {{ humanize $value}}.'

summary: 'High memory usage: CONTAINER ''{{ $labels.name }}'' on ''{{ $labels.instance

}}'''

[/codesyntax]

Переписывать файл(ы) вручную — не вариант, поэтому воспользуемся утилитой promtool для автоматического обновления всех файлов правил. Сначала получим последнюю версию данной утилиты (2.0):

YAML
wget <a class="vglnk" href="https://github.com/prometheus/prometheus/releases/download/v2.0.0/prometheus-2.0.0.linux-amd64.tar.gz" rel="nofollow">https://github.com/prometheus/prometheus/releases/download/v2.0.0/prometheus-2.0.0.linux-amd64.tar.gz</a>

				1
2

						wget <a class="vglnk" href="https://github.com/prometheus/prometheus/releases/download/v2.0.0/prometheus-2.0.0.linux-amd64.tar.gz" rel="nofollow">https://github.com/prometheus/prometheus/releases/download/v2.0.0/prometheus-2.0.0.linux-amd64.tar.gz</a>

YAML
tar zxvf prometheus-2.0.0.linux-amd64.tar.gz

				1
2

						tar zxvf prometheus-2.0.0.linux-amd64.tar.gz

Переходим в каталог с утилитой и запускаем процесс обновления для всех rules-файлов:

YAML
<span class="hljs-built_in">cd</span> prometheus-2.0.0.linux-amd64/

				1
2

						<span class="hljs-built_in">cd</span> prometheus-2.0.0.linux-amd64/

YAML
./promtool update rules /var/www/monitor/prometheus/host.rules 

				1
2

						./promtool update rules /var/www/monitor/prometheus/host.rules

YAML
./promtool update rules /var/www/monitor/prometheus/targets.rules

				1
2

						./promtool update rules /var/www/monitor/prometheus/targets.rules

YAML
./promtool update rules /var/www/monitor/prometheus/containers.rules

				1
2

						./promtool update rules /var/www/monitor/prometheus/containers.rules

После успешного обновления файлов изменятся также из имена, поэтому в конфигурационном файле prometheus.yml также меняем соответствующие строки на:

[codesyntax lang="php"]

...
rule_files:
  - "targets.rules.yml"
  - "host.rules.yml"
  - "containers.rules.yml"
...

...

rule_files:

- "targets.rules.yml"

- "host.rules.yml"

- "containers.rules.yml"

...

[/codesyntax]

После применения изменений не забываем перезапустить prometheus.

Пн	Вт	Ср	Чт	Пт	Сб	Вс
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30

Centos

15.Prometheus 2.0: Новый формат Alert rules

https://github.com/midnight47/