Built-in plugins

• CPU usage
• Load average
• Memory
• Swap: usage, I/O
• Disks: usage, I/O
• All processes: CPU, memory, swap, disk I/O, open files
• TCP connections: states, ack backlog, RTT
• Memcached
• Redis
• Nginx access logs
• Raid
• Zookeeper

Configuring Nginx

1. Add a new log_format (or modify the existing one) in /etc/nginx/nginx.conf:

http {
        ...
        log_format combined_plus '$remote_addr - $remote_user [$time_local]'
                                 ' "$request" $status $body_bytes_sent "$http_referer"'
                                 ' "$http_user_agent" $request_time $upstream_cache_status'
                                 ' [$upstream_response_time]';
        ...
}           

2. Use this format for each access_log directive in Nginx configuration.
   Generally, you only need to make changes to /etc/nginx/nginx.conf:

http {
        ...
        access_log /var/log/nginx/access.log combined_plus;
        ...
}           

3. Reload Nginx: sudo /etc/init.d/nginx reload

PostgreSQL

$ sudo su postgres -c "psql -d postgres"

CREATE ROLE okagent WITH LOGIN PASSWORD 'EXAMPLE_PASSWORD_DONT_USE_THAT_please_!)';
CREATE SCHEMA okmeter; -- So that helper won't mix with anything else.
GRANT USAGE ON SCHEMA okmeter TO okagent; -- So okmeter agent will have access to it.
CREATE OR REPLACE FUNCTION okmeter.pg_stats(text) -- For okagent to get stats.
RETURNS SETOF RECORD AS
$$
DECLARE r record;
BEGIN
    FOR r IN EXECUTE 'SELECT r FROM pg_' || $1 || ' r' LOOP RETURN NEXT r;  -- To get pg_settings, pg_stat_activity etc.
    END loop;
    RETURN;
END
$$ LANGUAGE plpgsql SECURITY DEFINER;

host all okagent 127.0.0.1/32 md5 
local all okagent md5 

$ sudo su postgres -c "psql -d postgres"

SELECT pg_reload_conf();

PostgreSQL query statistics

shared_preload_libraries = 'pg_stat_statements'   # change requires DB restart.
pg_stat_statements.max = 500
pg_stat_statements.track = top
pg_stat_statements.track_utility = true
pg_stat_statements.save = false

# #Also consider enabling I/O timing traction by uncommenting this:
#track_io_timing = on

$ sudo su postgres -c "psql -d postgres"

CREATE EXTENSION pg_stat_statements;

PgBouncer

"okagent" "EXAMPLE_PASSWORD_DONT_USE_THAT_please_!)"

; the comma-separated list of users who are only allowed to use the SHOW command
stats_users = okagent

/etc/init.d/pgbouncer reload

JVM

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.host=127.0.0.1 -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.port=9099 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false 

Php-fpm

pm.status_path = /status ;you can use /status or any other URL, okagent will work with that

RabbitMQ

rabbitmq-plugins enable rabbitmq_management
rabbitmqctl add_user okagent EXAMPLE_PASSWORD_DONT_USE_THAT_please_!)
rabbitmqctl set_user_tags okagent monitoring

rabbitmqctl set_permissions -p / okagent ".*" ".*" ".*"
rabbitmqctl set_permissions -p /vhost1 okagent ".*" ".*" ".*"

rabbitmqctl list_vhosts

Mysql

CREATE USER 'okagent'@'%' IDENTIFIED BY 'EXAMPLE_PASSWORD_DONT_USE_THAT_please_!)';
GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'okagent'@'%';
GRANT SELECT ON `performance_schema`.* TO 'okagent'@'%';
FLUSH PRIVILEGES;

mysql> SELECT 'OK' FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA='performance_schema' AND TABLE_NAME='events_statements_summary_by_digest';
+----+
| OK |
+----+
| OK |
+----+

mysql> SHOW VARIABLES LIKE 'performance_schema';
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| performance_schema | ON    |
+--------------------+-------+

[mysqld]
performance_schema=ON

Remote and cloud databases — AWS RDS | AWS Aurora | Cloud SQL | Azure Database for PostgreSQL | Elasticsearch | Redis

To monitor cloud and remote databases (i.e., databases running on hosts that do not have Okagent installed), you can install Okagent on a separate host (e.g., an AWS EC2 instance) and configure it accordingly.

Currently, the following remote and cloud databases are supported: Elastic Search, Redis, MySQL, PostgreSQL

You can also use an existing server / EC2 instance with DB access, such as your web-application, which probably already has access to the database through a security group or similar.

Next, create a config (e.g., /usr/local/okagent/etc/config.d/remote_db.yaml) for the okmeter monitoring agent in the /usr/local/okagent/etc/config.d/ directory of the server | EC2 instance where Okagent is running with the following contents:

For MySQL or PostgreSQL:

plugin: postgresql # or mysql
config:
  host: db_ip # replace with your remote DB instance or cluster endpoint
  #port: db_port # uncomment and replace with your remote DB instance port if it's non-standard
  user: db_user # replace with your remote DB instance monitoring user
  password: db_password # replace with your remote DB instance monitoring user's password.
  #database: mydb # replace with the database name you used to connect to when you created the okmeter schema, user, and function in PostgreSQL, if different from "postgres"

For Redis:

plugin: redis
config:
  host: db_ip # replace with your remote DB instance or cluster endpoint
  #port: db_port # uncomment and replace with your remote DB instance port if it's non-standard
  #password: db_password # uncomment and replace with your remote DB instance monitoring user's password.

For Elasticsearch:

plugin: elasticsearch
config:
  host: elasticserch_url # replace with your Elasticsearch IP in the following format: xx.xx.xx.xx
  #port: db_port # uncomment and replace with your remote Elasticsearch port if it's non-standard (9200)
  #user: db_user # uncomment and replace with your remote Elasticsearch monitoring user
  #password: db_password # uncomment and replace with your remote Elasticsearch monitoring user's password.
  #insecureTls: true # uncomment if Elasticsearch is configured to use a self-signed certificate

Restart Okagent to apply changes: $ sudo /etc/init.d/okagent restart (or $ sudo systemctl restart okagent.service).

Make sure that the monitoring user has sufficient database permissions — refer to the Mysql plugin or Postgresql plugin documentation.

Zookeeper

For Zookeeper 3.4.10 or higher, add stat and mntr commands to the whitelist in your zoo.cfg:

4lw.commands.whitelist=stat, mntr

Sending custom metrics

In addition to built-in metrics, Okmeter can process custom metrics. There are several ways to send your own metrics:

1. Write an SQL query that returns some numerical values from the database using the SQL query plugin.
2. Parse log files using the Logparser plugin.
3. Write a script that dumps metrics to stdout and invoke it periodically using the Execute plugin.
4. Parse responses from the HTTP endpoint using the HTTP plugin.
5. Gather information from the output of Redis commands using the Redis query plugin.
6. Collect metrics from an application using Statsd.
7. Collect metrics from Promethus-compatible exporters using the Prometheus plugin.

These plugins require additional configuration files. Okagent reads all configuration files in the /usr/local/okagent/etc/config.d/ directory. The name of the configuration file can be anything, the extension must be .yaml, and the file format is YAML.

Verifying configuration files | dry run mode

You can check the syntax of the configuration file with the following command (replace PLUGIN_CONFIG_FILE with your file):

$ /usr/local/okagent/okagent -dry-run=/usr/local/okagent/etc/config.d/PLUGIN_CONFIG_FILE

If the command does not result in errors, apply the configuration file by restarting Okagent: $ sudo /etc/init.d/okagent restart (or $ sudo systemctl restart okagent.service).

plugin: execute
config:
  command: 'du /var/log/app/main.log'
  regexp: '(?P<value>\d+)'
  name: demo_app_log_size  # metric name
  value: value             # metric value
  labels:                  # metric labels
    log_name: main

JSON

{
    "name": "metric1",
    "labels": {"label1": "foo", "label2": "bar"},
    "value": 123.4
}

echo '{"name": "metric1", "labels": {"label1": "foo", "label2": "bar"}, "value": 123.4}'

plugin: execute
config:
  command: /tmp/calc_metrics.sh
  parser: json

echo '[{"name": "metric1", "value": 123.4}, {"name": "metric2", "value": 567}]'

Logparser plugin

This plugin sends metrics retrieved from custom log files. You can use Regexp or JSON parsers to generate metrics.

Regexp

The Regexp parser analyzes each line of the file and converts it into a metric value and additional labelset sets. For example, to send metrics based on the /var/log/app/stages.log log file, you need to create a /usr/local/okagent/etc/config.d/stage_log.yaml configuration YAML file with the following contents:

plugin: logparser
config:
  file: /var/log/app/stages.log
  regexes:
    # 2015-11-21 15:42:36,972 demo [DEBUG] page=item stages: db=0.007s, render=0.002s, total=0.010s count=1
    - regex: '(?P\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}).+ page=(?P\w+) stages: db=(?P\d+\.\d+)s, render=(?P\d+\.\d+)s, total=(?P\d+\.\d+)s count=(?P\d+)'
      time_field: datetime
      time_field_format: '2006-01-02 15:04:05'
      metrics:
        ...

Here:

• file — path to the file to be analyzed. Okagent handles file rotation correctly;
• regexes — list of regular expressions to apply to strings. Regular expressions are applied sequentially;
• regexp — Perl-compatible regular expression (regex). It is used to split the string into named groups for further transformation into metrics;
• time_field — name of the timestamp group;
• time_field_format — format of timestamps;
• metrics — list of metrics to send together with the rules for converting strings into metrics.

JSON

The JSON parser converts the JSON-formatted logs into metrics. This method is preferable because it creates less overhead and does not require parsing using regular expressions.

plugin: logparser
config:
  file: /var/log/app/stages.log
  json: true
  # {ts: "2015-11-21 15:42:36.000+0300", user: "demo", page: "item", db: "0.007", render: "0.002", total: "0.010", count: "1"}
  time_field: ts
  time_field_format: "2006-01-02 15:04:05.000-0700"
  metrics:
    ...

Here:

• file — path to the file to be analyzed. Okagent handles file rotation correctly;
• json — type of the parser (JSON);
• time_field — name of the timestamp group;
• time_field_format — timestamp format.

TOP

Log parsing often leads to significant metric churn which negatively affects the display of graphs. For metrics based on log files, we recommended using the TOP function. It limits the number of metrics to N (where N is the parameter defined in the configuration file) and sums up all other metrics and sends them as an additional metric with the label: ~other value.

Example. Suppose we want to parse the /var/log/app/stages.log log file containing information about users' requests to the service. However, we want metrics only for the ten most frequent requests over the last 10 minutes. To implement this logic, create a /usr/local/okagent/etc/config.d/top_url.yaml YAML configuration file with the following contents:

plugin: logparser
config:
    file: /var/log/app/stages.log
    #{"ts":"2018-09-12 13:07:11.500","logger": "requests","time":"33","method":"PUT","status":200,"uri":"/aaa/bbb","rid":noRequestId,ip":"2.2.2.2"}
    #{"ts":"2019-11-20 18:32:49.851+0300","logger":"requests","time":"157","method":"PUT","status":200,"uri":"/foo/bar?from=header_new","rid":"11","ip":"1.1.1.1"}
    json: true
    time_field: ts
    time_field_format: "2006-01-02 15:04:05.000-0700"
    top_vars:
      topurl:
        source: uri
        weight: 1
        threshold_percent: 1
        window_minutes: 10
    metrics:
    - type: rate
        name: service_requests_rate
        labels:
        method: =method
        url: =topurl
        status: =status
    - type: percentiles
        name: service_response_time_percentiles
        value: ms2sec:time
        args: [50, 75, 95, 99]
    - type: percentiles
        name: service_response_time_percentile_by_url
        value: ms2sec:time
        args: [95]
        labels:
        url: =topurl

The top_vars sections has the following fields:

• topurl is a new label for the metric to get TOPN URIs (the source: uri JSON field –);
• weight – increments the metric counter at each URI hit for;
• threshold_percent – metrics whose total sum is below this threshold are combined into a special cumulative metric (~other).
• window_minutes – window size to use for TOP calculation (in minutes).

The metric type can be one of the following:

• rate — average growth rate of the metric per minute. It is used by default.
• percentile — collects the n-th percentile of values per minute. The args key is an array of percentage values, e.g., [50, 95, 99].
• max or min — collects the maximum or minimum value for the metric per minute.
• threshold — collects the rate of value hits in different intervals (e.g., (-∞, 0.5], (0.5, 1], ...).

time_field — name of the field to store metric timestamps.

time_field_format can be one of the following:

• unix_timestamp — floating point number with the Unix time timestamp
• common_log_format — this format parses time and date values as follows: 2/Jan/2006:15:04:05 -0700
• time_iso8601 — the RFC 3339 format (also known as ISO 8601)
• The custom timestamp format. The user specifies how the reference time, e.g. Mon Jan 2 15:04:05 -0700 MST 2006, should be formatted to the time format of your log. It is used as a template for the Logparser plugin.

If no time zone information is available, the UTC time zone is used.

Metric may have a labels object with static (stage: db) and dynamic (page: =page) labels. The = sign before the label value (e.g., page: =page in the example above) means that the actual value should be taken from the regular expression group named page.

Example. Suppose that the application log contains the following entries:

2015-11-01 22:51:44,072 demo [DEBUG] page=item stages: db=0.005s, render=0.002s
2015-11-01 22:51:44,087 demo [DEBUG] page=list stages: db=0.003s, render=0.001s
...

If you apply the configuration file shown below,...

plugin: logparser
config:
  file: /var/log/app/stages.log
  regexes:
    # 2015-11-21 15:42:36,972 demo [DEBUG] page=item stages: db=0.007s, render=0.002s, total=0.010s count=1
    - regex: '(?P<datetime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}).+ page=(?P<page>\w+) stages: db=(?P<db>\d+\.\d+)s, render=(?P<render>\d+\.\d+)s, total=(?P<total>\d+\.\d+)s count=(?P<count>\d+)'
      time_field: datetime
      time_field_format: '2006-01-02 15:04:05'
      metrics:
        - type: percentiles
          args: [50, 75, 95, 98, 99]
          value: db
          name: demo_stages_percentiles
          labels:
            stage: db
        - type: percentiles
          args: [50, 75, 95, 98, 99]
          value: render
          name: demo_stages_percentiles
          labels:
            stage: render
        - type: rate
          name: demo_requests_rate
          labels:
            page: =page
        - type: rate
          name: demo_documents_rate
          value: count
          labels:
            page: =page
        - type: threshold
          value: total
          args: [0.05, 0.1]
          name: demo_render_histogram
          labels:
            page: =render

... the Logparser plugin would post the following metrics: demo_requests_rate, demo_documents_rate, demo_stages_percentiles, and demo_render_histogram.

Note: Verify that the configuration file is correct before using it.

Note: Restart Okagent to apply the configuration: $ sudo /etc/init.d/okagent restart (or $ sudo systemctl restart okagent.service).

HTTP plugin

This plugin sends custom metrics about the availability of the HTTP service, as well as metrics derived from parsing the response to an HTTP request.

Example: Suppose, there is an HTTP service with access restricted by Basic Auth. The response to the request at /service/stat looks as follows:

curl --user name:password -H 'Foo: bar' -v 'http://127.0.0.1:8080/service/stat'
> GET /service/stat HTTP/1.1
> Host: 127.0.0.1
> Authorization: Basic bmFtZTpwYXNzd29yZA==
> Foo: bar
>
< HTTP/1.1 200 OK
<
online_users=140 active_users=10

To post custom metrics obtained by parsing the response to an HTTP request, you have to create a /usr/local/okagent/etc/config.d/http.yaml configuration YAML-file as follows:

plugin: http
config:
  url: http://127.0.0.1:8080/service/stat
  username: name
  password: password
  #sslskip: on #optional, disable certificate verification, like curl --insecure
  headers:
    foo: bar
  metrics:
    - metric: users_online
      regex: 'online_users=(\d+)'
    - metric: users_active
      regex: 'active_users=(\d+)'

The request will be performed every minute, and the following two metrics will be sent upon parsing the response received:

metric(name="users_online")
metric(name="users_active")

On top of that, the plugin will generate three additional metrics:

metric(name="status", plugin="http", instance="<config filename>")

The request status will be "1" if the request was successful, and "0" otherwise.

metric(name="status", plugin="http", instance=" type="text", value="")

The value label will be empty if the request was completed without errors. Otherwise, the value label will contain an error that occurred while executing the request.

metric(name="http.request_time", plugin="http", instance="<config filename>")

The request execution time will be passed as the value of this metric.

Note: Metrics and label names can only contain ASCII characters/digits and must match the [a-zA-Z_][a-zA-Z0-9_]* regular expression.

Note: Verify that the configuration file is correct before using it.

Note: Restart Okagent to apply the configuration: $ sudo /etc/init.d/okagent restart (or $ sudo systemctl restart okagent.service).

Redis query

This plugin allows you to collect custom metrics based on Redis commands. An example of the /usr/local/okagent/etc/config.d/redis_example.yaml configuration file is shown below:

plugin: redis_query
config:
  #host: 127.0.0.1 #optional
  #port: 6379 #optional
  #database: 0 #optional
  commands:
    - LLEN achievement_queue
    - SCARD some_queue
    - HGET stat active_connections
    - GET mail_sender_queue_len

A plugin with this configuration will collect four metrics:

metric(name="achievement_queue.llen")
metric(name="some_queue.scard")
metric(name="stat.hget", param="active_connections")
metric(name="mail_sender_queue_len.get")

The following Redis commands are supported: BITCOUNT, GET, GETBIT, GETRANGE, HGET, HLEN, HSTRLEN, LINDEX, LLEN, PTTL, SCARD, STRLEN, TTL, ZCARD, ZCOUNT, ZLEXCOUNT, ZRANK, ZREVRANK, ZSCORE.

Prometheus

This plugin allows scraping data from Prometheus-compatible exporters and sending them as metrics.

It scrapes each discovered exporter and produces metrics, for example, the http_request_duration_seconds histogram:

metric(name="http_request_duration_seconds_count", handler="/", method="GET", code="200", ...)
metric(name="http_request_duration_seconds_sum", handler="/", method="GET", code="200", ...)
metric(name="http_request_duration_seconds_bucket", handler="/", method="GET", code="200", le="0.1", ...)
metric(name="http_request_duration_seconds_bucket", handler="/", method="GET", code="200", le="0.5", ...)
...

For applications running in Kubernetes, endpoint discovery is based on annotations:

apiVersion: apps/v1
kind: Deployment #or StatefulSet, DaemonSet, CronJob, ...
metadata:
    name: my-app
    annotations:
        prometheus.io/scrape: "true"
        prometheus.io/scheme: "http"
        prometheus.io/port: "80"
        prometheus.io/path: "/metrics"
...

Okagent can also discover exporters running in Docker containers with the appropriate labels:

docker run --name my-app \
    --label io.prometheus.scrape=true \
    --label io.prometheus.port=80 \
    --label io.prometheus.scheme=http \
    --label io.prometheus.path="/metrics" \
    ...

On top of that, Okagent can scrape metrics from custom targets defined in the configuration file. The data is collected as long as Okagent receives valid responses. An example of a plugin configuration file is shown below:

plugin: prometheus
config:
  targets:
    - http://localhost:9100/metrics
  # max cardinality (default 5000)
  limit: 1200
  # additional labels for all scraped metrics
  labels:
    exporter: node
        

If necessary, you can specify authentication credentials. Basic Authorization and Bearer tokens are currently supported. Example of a Basic Authorization configuration file:

plugin: prometheus
config:
  targets:
  - http://localhost:9100/metrics
  authorization:
    type: basic
    username: <user>
    password: <password>
        

Example of a Bearer token configuration file:

plugin: prometheus
config:
  targets:
  - http://localhost:9100/metrics
  authorization:
    type: bearer
    token: <token>
        

API

Okmeter has built-in limited support for Prometheus-like Query API. Currently, there are two types of queries supported: query and query_range.

You can use Okmeter as a Datasource for your Grafana installation. To do so, add a new Datasource of the Prometheus type with the following settings:

The Basic Auth Password is the project's access token. Please remember to set the Scrape interval to 60s.

Once set, this Datasource can be used to retrieve data from Okmeter.

Note that Okmeter does not support PromQL query language at the moment. Only the native query language Okmeter Query Language is supported.

Query Language

Lines expression syntax:

lines:
- expression: metric(a='b', c='d*', e=['f', 'g*']) #example some load averages

select all metrics with labels matching values

a='b' matches metrics with label a equals fixed string 'b'

c='d*' matches metrics with label c starts with 'd'

e=['f', 'g*'] matches metrics with label e equals fixed string 'f' or starts with 'g'

lines:
- expression: rate(EXPR) #example python cpu_user

derivative for each metric in EXPR

lines:
- expression: counter_rate(EXPR) #example python cpu_user

derivative for counters – like rate but doesn`t spikes for counter reset

lines:
- expression: sum(EXPR [, ignore_nan=True|False]) #example all python's cpu_user and cpu_system

sum of all metrics in EXPR

If ignore_nan=False, then result is NaN if one metric in EXPR was NaN. Default is ignore_nan=True

lines:
- expression: max(EXPR)
- expression: min(EXPR)
- expression: std(EXPR) #standard deviation
- expression: average(EXPR) #same as mean
- expression: mean(EXPR) #example mean load average

at each time-point take aggregation function for all metrics in EXPR

lines:
- expression: sum_by(label_name, [other_label,] EXPR) #example processes cpu usage
- expression: max_by(label_name, [other_label,] EXPR)
- expression: min_by(label_name, [other_label,] EXPR)
- expression: std_by(label_name, [other_label,] EXPR) #standard deviation
- expression: mean_by(label_name, [other_label,] EXPR) #same as average
- expression: average_by(label_name, [other_label,] EXPR) #example mean load average

group all metrics in EXPR by value of label_name label and aggregate metrics in the same group into one metric

Accepts parametrignore_nan=False|True, just like ordinary sum

lines:
- expression: win_sum(window_size_in_seconds, EXPR)
- expression: win_mean(window_size_in_seconds, EXPR) #same as win_avg
- expression: win_min(window_size_in_seconds, EXPR)
- expression: win_max(window_size_in_seconds, EXPR)
- expression: win_std(window_size_in_seconds, EXPR)
- expression: win_avg(window_size_in_seconds, EXPR) #example mean load average on hour window

Applies specified function sum|mean|min|max|std for each metric in EXPR on moving time window window_size_in_seconds. See Moving average

lines:
- expression: cum_sum(EXPR) #example

Cumulative sum for each metric in EXPR.

lines:
- expression: top(N, EXPR[, include_other=true|false][, by="exp"|"sum"|"max"]) #example top 5 processes by CPU
- expression: bottom(N, EXPR[, by="exp"|"sum"|"max"])

show top|bottom N metrics from EXPR by ews|exp(exponentialy weighted sum) or sum or max in current timespan

lines:
- expression: filter_with(EXPR, FILTER_EXPR) #example memory usage of long running processes

filters metrics in EXPR returning only those for which FILTER_EXPR not zero (or NaN).

lines:
- expression: const(v[, label="value", ...]) #example

constant metric with value v and additonal labels for legend

lines:
- expression: time()

timestamp from x-axis as y-value

lines:
- expression: from_string("1,2,3,3,2,1,", [,repeat=false] [,sep=' '] [,label="value", ...]) #example

construct metric from string like "1,2,3,3,2,1,", where each number becomes the value of the metric for corresponding minute

lines:
- expression: defined(EXPR) #example all processes

1 if there is data from EXPR in this time-point or 0 if there is NaN

lines:
- expression: replace(old_val, new_val, EXPR) #example
- expression: n2z(EXPR) #shortcut for "replace(nan, 0, EXPR)"
- expression: zero_if_none(EXPR) #shortcut for "replace(nan, 0, EXPR)"
- expression: z2n(EXPR) #shortcut for "replace(0, nan, EXPR)"
- expression: zero_if_negative(EXPR)
- expression: none_if_zero(EXPR) #shortcut for "replace(0, nan, EXPR)"
- expression: remove_below(EXPR, value)
- expression: remove_above(EXPR, value)
- expression: clamp_min(EXPR, min)
- expression: clamp_max(EXPR, max)

sets new_val instead of old_val

lines:
- expression: sum_by(label, [other_label,] metric(..)) / max_by(label, [other_label,] metric(.))
- expression: sum_by(label, [other_label,] metric(..)) * sum_by(label, [other_label,] metric(.))
- expression: sum_by(label, [other_label,] metric(..)) - min_by(label, [other_label,] metric(.))

if labels for bothsum_by are the same than it evaluates as / * or - for each pair of metrics (one from left and one from right metric)

lines:
- expression: sum_by(label, [other_label,] metric(..)) / EXPR
- expression: min_by(label, [other_label,] metric(..)) * EXPR
- expression: max_by(label, [other_label,] metric(..)) - EXPR

Applies / EXPR * EXPR or - EXPR for each metric from left XXX_by(label, ...)

Lines legend syntax:

lines:
- expression: metric(...)
  legend: '%s'

for each line show all label_name:label_value pairs in legend

lines:
- expression: metric(...)
  legend: '%(label_name)s anything'

for each line show `label_value` anything in legend

Colors syntax:

lines:
- expression: metric(...)
  color: '#81ff22'
- expression: metric(...)
  color: 'red'

color is color

lines:
- expression: metric(...)
  colors: ['#80AB00', 'red', 'rgb(127,0,20)', 'hsla(100,10%,20%,0.8)']

will cycle through specified colors

lines:
- expression: metric(...)
  colors: 
    /regex.*/: '#fff'
    /regex2/: 'gold'

will match legend to regexes

lines:
- expression: metric(...)
options:
  colors: semaphore
  #OR
  colors: semaphore inv

will color all, previously uncolored lines, with a gradient from red to green

or from greento red if semaphore inv

Sorting syntax:

lines:
- expression: metric(...)
options:
  sort: alpha|num

sort all lines by legend in alphabetical or numeric (default) order

lines:
- expression: metric(...)
options:
  sort: ['fixed', 'order', 'for', 'legend', 'items']

fixed sort order by item's legend

lines:
- expression: metric(...)
options:
  sort: ...
  order: DESC

change sort order to descending

Sorting tooltip syntax:

lines:
- expression: metric(...)
options:
  tooltip:
    sort_order: name|-name|value|-value

Sorting lines in tooltip (no sorting by default):

• in alphabetical(name | -name).
• by values(value | -value)

Captions:

lines:
- expression: metric(...)
- expression: metric(...)
title: 'some %(label_name)s'

format chart title with labels from all expressions combined

lines:
- expression: metric(...)
- expression: metric(...)
options:
  y_title: 'some text'

Y-axis vertical title as plain text

Alerting

Alerts are an essential part of the monitoring system. They inform the user about what is happening in a timely manner.

Configuring alerts in Okmeter consists of two parts: the triggers part (aka alerting rules) is where the rules that determine when a notification is sent are configured. The notifications part defines where the notification is sent.

Okmeter provides you with a large variety of preset triggers. You can also create your own triggers or override pre-configured ones.

We strongly recommend testing changes to triggers or notifications to make sure that notifications are triggered when they are needed. The easiest way to test is to put the trigger into an active state. You can do this for example by changing the threshold value.

expression: 'sum(n2z(metric(name="nginx.requests.rate", status="5*")))'
threshold: '>= 1'
severity: critical
message: '5xx nginx %(value).1f per sec'
notify_after: 120
resolve_after: 180
notification_config: ops

expression: 'sum_by(status, n2z(metric(name="nginx.requests.rate", status="5*")))'
message: '%(status)s nginx %(value).1f per sec'

renotify_interval: 600
notify_resolve: on

oncalls: # email and sms
  - admin_1
  - admin_2
  - boss

slack_url: https://hooks.slack.com/services/XXXX/XXXX/XXXXXXXX
slack_channel: '#ops'

telegram_chat_id: -XXXXXXXX

opsgenie:
  api_key: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

prometheus:
  url: http://HOST:PORT/api/v1/alerts
  basic_auth:
    username: XXXX
    password: XXXX