Disk writes
backend
bgwriter
checkpoint
temp files
wal
Lines expression syntax:
lines: - expression:
metric(a='b', c='d*', e=['f', 'g*'])#example some load averages- select all metrics with labels matching values
a='b'matches metrics with labelaequals fixed string'b'c='d*'matches metrics with labelcstarts with'd'e=['f', 'g*']matches metrics with labeleequals fixed string'f'or starts with'g'lines: - expression:
rate(EXPR)#example python cpu_user- derivative for each metric in
EXPR lines: - expression:
counter_rate(EXPR)#example python cpu_user- derivative for counters – like
ratebut doesn`t spikes for counter reset lines: - expression:
sum(EXPR [, ignore_nan=True|False])#example all python's cpu_user and cpu_system- sum of all metrics in
EXPR - If
ignore_nan=False, then result is NaN if one metric in EXPR was NaN. Default isignore_nan=True lines: - expression:
max(EXPR)- expression:min(EXPR)- expression:std(EXPR)#standard deviation - expression:average(EXPR)#same as mean - expression:mean(EXPR)#example mean load average- at each time-point take aggregation function for all metrics in
EXPR lines: - expression:
sum_by(label_name, [other_label,] EXPR)#example processes cpu usage - expression:max_by(label_name, [other_label,] EXPR)- expression:min_by(label_name, [other_label,] EXPR)- expression:std_by(label_name, [other_label,] EXPR)#standard deviation - expression:mean_by(label_name, [other_label,] EXPR)#same as average - expression:average_by(label_name, [other_label,] EXPR)#example mean load average- group all metrics in
EXPRby value oflabel_namelabel and aggregate metrics in the same group into one metric - Accepts parametr
ignore_nan=False|True, just like ordinarysum lines: - expression:
win_sum(window_size_in_seconds, EXPR)- expression:win_mean(window_size_in_seconds, EXPR)#same as win_avg - expression:win_min(window_size_in_seconds, EXPR)- expression:win_max(window_size_in_seconds, EXPR)- expression:win_std(window_size_in_seconds, EXPR)- expression:win_avg(window_size_in_seconds, EXPR)#example mean load average on hour window- Applies specified function
sum|mean|min|max|stdfor each metric inEXPRon moving time windowwindow_size_in_seconds. See Moving average lines: - expression:
cum_sum(EXPR)#example- Cumulative sum for each metric in
EXPR. lines: - expression:
top(N, EXPR[, include_other=true|false][, by="exp"|"sum"|"max"])#example top 5 processes by CPU - expression:bottom(N, EXPR[, by="exp"|"sum"|"max"])- show top|bottom
Nmetrics fromEXPRbyews|exp(exponentialy weighted sum) orsumormaxin current timespan lines: - expression:
filter_with(EXPR, FILTER_EXPR)#example memory usage of long running processes- filters metrics in
EXPRreturning only those for whichFILTER_EXPRnot zero (or NaN). lines: - expression:
const(v[, label="value", ...])#example- constant metric with value
vand additonal labels for legend lines: - expression:
time()- timestamp from x-axis as y-value
lines: - expression:
from_string("1,2,3,3,2,1,", [,repeat=false] [,sep=' '] [,label="value", ...])#example- construct metric from string like
"1,2,3,3,2,1,", where each number becomes the value of the metric for corresponding minute lines: - expression:
defined(EXPR)#example all processes1if there is data fromEXPRin this time-point or0if there is NaNlines: - expression:
replace(old_val, new_val, EXPR)#example - expression:n2z(EXPR)#shortcut for "replace(nan, 0, EXPR)" - expression:zero_if_none(EXPR)#shortcut for "replace(nan, 0, EXPR)" - expression:z2n(EXPR)#shortcut for "replace(0, nan, EXPR)" - expression:zero_if_negative(EXPR)- expression:none_if_zero(EXPR)#shortcut for "replace(0, nan, EXPR)" - expression:remove_below(EXPR, value)- expression:remove_above(EXPR, value)- expression:clamp_min(EXPR, min)- expression:clamp_max(EXPR, max)- sets
new_valinstead ofold_val lines: - expression:
sum_by(label, [other_label,] metric(..)) / max_by(label, [other_label,] metric(.))- expression:sum_by(label, [other_label,] metric(..)) * sum_by(label, [other_label,] metric(.))- expression:sum_by(label, [other_label,] metric(..)) - min_by(label, [other_label,] metric(.))- if labels for both
sum_byare the same than it evaluates as/*or-for each pair of metrics (one from left and one from rightmetric) lines: - expression:
sum_by(label, [other_label,] metric(..)) / EXPR- expression:min_by(label, [other_label,] metric(..)) * EXPR- expression:max_by(label, [other_label,] metric(..)) - EXPR- Applies
/ EXPR* EXPRor- EXPRfor each metric from leftXXX_by(label, ...)
Lines legend syntax:
lines: - expression: metric(...) legend:
'%s'- for each line show all
label_name:label_valuepairs in legend lines: - expression: metric(...) legend:
'%(label_name)s anything'- for each line show
`label_value` anythingin legend
Colors syntax:
lines: - expression: metric(...) color:
'#81ff22'- expression: metric(...) color:'red'- color is color
lines: - expression: metric(...) colors:
['#80AB00', 'red', 'rgb(127,0,20)', 'hsla(100,10%,20%,0.8)']- will cycle through specified colors
lines: - expression: metric(...) colors:
/regex.*/: '#fff' /regex2/: 'gold'- will match legend to regexes
lines: - expression: metric(...) options: colors:
semaphore#OR colors:semaphore inv- will color all, previously uncolored lines, with a gradient from
redtogreen - or from
greentoredifsemaphore inv
Sorting syntax:
lines: - expression: metric(...) options: sort:
alpha|num- sort all lines by legend in alphabetical or numeric (default) order
lines: - expression: metric(...) options: sort:
['fixed', 'order', 'for', 'legend', 'items']- fixed sort order by item's
legend lines: - expression: metric(...) options: sort: ... order:
DESC- change sort order to descending
Sorting tooltip syntax:
lines: - expression: metric(...) options: tooltip: sort_order:name|-name|value|-value-
Sorting lines in tooltip (no sorting by default):
- in alphabetical(
name|-name). - by values(
value|-value)
- in alphabetical(
Captions:
lines: - expression: metric(...) - expression: metric(...)
title: 'some %(label_name)s'- format chart title with labels from all expressions combined
lines: - expression: metric(...) - expression: metric(...) options:
y_title: 'some text'- Y-axis vertical title as plain text
Pages dirtied by queries
demo SELECT "labels"."key", "labels"."value", "labels"."updated", "labels"."update_type" FROM "labels" WHERE "labels"."key" = ?
demo SELECT "labels"."key", "labels"."value", "labels"."updated", "labels"."update_type" FROM "labels" WHERE "labels"."key"::text LIKE ? ORDER BY "labels"."key" ASC LIMIT ?
demo SELECT "labels"."key", "labels"."value", "labels"."updated", "labels"."update_type" FROM "labels" WHERE "labels"."key"::text LIKE ? ORDER BY "labels"."key" ASC LIMIT ? OFFSET ?
demo UPDATE "labels" SET "value" = ?, "updated" = ?, "update_type" = ? WHERE "labels"."key" = ?
demo select count(*) as value, update_type from labels where updated between now() - interval? and now() group by update_type
~other ~other
Lines expression syntax:
lines: - expression:
metric(a='b', c='d*', e=['f', 'g*'])#example some load averages- select all metrics with labels matching values
a='b'matches metrics with labelaequals fixed string'b'c='d*'matches metrics with labelcstarts with'd'e=['f', 'g*']matches metrics with labeleequals fixed string'f'or starts with'g'lines: - expression:
rate(EXPR)#example python cpu_user- derivative for each metric in
EXPR lines: - expression:
counter_rate(EXPR)#example python cpu_user- derivative for counters – like
ratebut doesn`t spikes for counter reset lines: - expression:
sum(EXPR [, ignore_nan=True|False])#example all python's cpu_user and cpu_system- sum of all metrics in
EXPR - If
ignore_nan=False, then result is NaN if one metric in EXPR was NaN. Default isignore_nan=True lines: - expression:
max(EXPR)- expression:min(EXPR)- expression:std(EXPR)#standard deviation - expression:average(EXPR)#same as mean - expression:mean(EXPR)#example mean load average- at each time-point take aggregation function for all metrics in
EXPR lines: - expression:
sum_by(label_name, [other_label,] EXPR)#example processes cpu usage - expression:max_by(label_name, [other_label,] EXPR)- expression:min_by(label_name, [other_label,] EXPR)- expression:std_by(label_name, [other_label,] EXPR)#standard deviation - expression:mean_by(label_name, [other_label,] EXPR)#same as average - expression:average_by(label_name, [other_label,] EXPR)#example mean load average- group all metrics in
EXPRby value oflabel_namelabel and aggregate metrics in the same group into one metric - Accepts parametr
ignore_nan=False|True, just like ordinarysum lines: - expression:
win_sum(window_size_in_seconds, EXPR)- expression:win_mean(window_size_in_seconds, EXPR)#same as win_avg - expression:win_min(window_size_in_seconds, EXPR)- expression:win_max(window_size_in_seconds, EXPR)- expression:win_std(window_size_in_seconds, EXPR)- expression:win_avg(window_size_in_seconds, EXPR)#example mean load average on hour window- Applies specified function
sum|mean|min|max|stdfor each metric inEXPRon moving time windowwindow_size_in_seconds. See Moving average lines: - expression:
cum_sum(EXPR)#example- Cumulative sum for each metric in
EXPR. lines: - expression:
top(N, EXPR[, include_other=true|false][, by="exp"|"sum"|"max"])#example top 5 processes by CPU - expression:bottom(N, EXPR[, by="exp"|"sum"|"max"])- show top|bottom
Nmetrics fromEXPRbyews|exp(exponentialy weighted sum) orsumormaxin current timespan lines: - expression:
filter_with(EXPR, FILTER_EXPR)#example memory usage of long running processes- filters metrics in
EXPRreturning only those for whichFILTER_EXPRnot zero (or NaN). lines: - expression:
const(v[, label="value", ...])#example- constant metric with value
vand additonal labels for legend lines: - expression:
time()- timestamp from x-axis as y-value
lines: - expression:
from_string("1,2,3,3,2,1,", [,repeat=false] [,sep=' '] [,label="value", ...])#example- construct metric from string like
"1,2,3,3,2,1,", where each number becomes the value of the metric for corresponding minute lines: - expression:
defined(EXPR)#example all processes1if there is data fromEXPRin this time-point or0if there is NaNlines: - expression:
replace(old_val, new_val, EXPR)#example - expression:n2z(EXPR)#shortcut for "replace(nan, 0, EXPR)" - expression:zero_if_none(EXPR)#shortcut for "replace(nan, 0, EXPR)" - expression:z2n(EXPR)#shortcut for "replace(0, nan, EXPR)" - expression:zero_if_negative(EXPR)- expression:none_if_zero(EXPR)#shortcut for "replace(0, nan, EXPR)" - expression:remove_below(EXPR, value)- expression:remove_above(EXPR, value)- expression:clamp_min(EXPR, min)- expression:clamp_max(EXPR, max)- sets
new_valinstead ofold_val lines: - expression:
sum_by(label, [other_label,] metric(..)) / max_by(label, [other_label,] metric(.))- expression:sum_by(label, [other_label,] metric(..)) * sum_by(label, [other_label,] metric(.))- expression:sum_by(label, [other_label,] metric(..)) - min_by(label, [other_label,] metric(.))- if labels for both
sum_byare the same than it evaluates as/*or-for each pair of metrics (one from left and one from rightmetric) lines: - expression:
sum_by(label, [other_label,] metric(..)) / EXPR- expression:min_by(label, [other_label,] metric(..)) * EXPR- expression:max_by(label, [other_label,] metric(..)) - EXPR- Applies
/ EXPR* EXPRor- EXPRfor each metric from leftXXX_by(label, ...)
Lines legend syntax:
lines: - expression: metric(...) legend:
'%s'- for each line show all
label_name:label_valuepairs in legend lines: - expression: metric(...) legend:
'%(label_name)s anything'- for each line show
`label_value` anythingin legend
Colors syntax:
lines: - expression: metric(...) color:
'#81ff22'- expression: metric(...) color:'red'- color is color
lines: - expression: metric(...) colors:
['#80AB00', 'red', 'rgb(127,0,20)', 'hsla(100,10%,20%,0.8)']- will cycle through specified colors
lines: - expression: metric(...) colors:
/regex.*/: '#fff' /regex2/: 'gold'- will match legend to regexes
lines: - expression: metric(...) options: colors:
semaphore#OR colors:semaphore inv- will color all, previously uncolored lines, with a gradient from
redtogreen - or from
greentoredifsemaphore inv
Sorting syntax:
lines: - expression: metric(...) options: sort:
alpha|num- sort all lines by legend in alphabetical or numeric (default) order
lines: - expression: metric(...) options: sort:
['fixed', 'order', 'for', 'legend', 'items']- fixed sort order by item's
legend lines: - expression: metric(...) options: sort: ... order:
DESC- change sort order to descending
Sorting tooltip syntax:
lines: - expression: metric(...) options: tooltip: sort_order:name|-name|value|-value-
Sorting lines in tooltip (no sorting by default):
- in alphabetical(
name|-name). - by values(
value|-value)
- in alphabetical(
Captions:
lines: - expression: metric(...) - expression: metric(...)
title: 'some %(label_name)s'- format chart title with labels from all expressions combined
lines: - expression: metric(...) - expression: metric(...) options:
y_title: 'some text'- Y-axis vertical title as plain text
Checkpoints issued
checkpoint_timeout reached
wal_size reached
Lines expression syntax:
lines: - expression:
metric(a='b', c='d*', e=['f', 'g*'])#example some load averages- select all metrics with labels matching values
a='b'matches metrics with labelaequals fixed string'b'c='d*'matches metrics with labelcstarts with'd'e=['f', 'g*']matches metrics with labeleequals fixed string'f'or starts with'g'lines: - expression:
rate(EXPR)#example python cpu_user- derivative for each metric in
EXPR lines: - expression:
counter_rate(EXPR)#example python cpu_user- derivative for counters – like
ratebut doesn`t spikes for counter reset lines: - expression:
sum(EXPR [, ignore_nan=True|False])#example all python's cpu_user and cpu_system- sum of all metrics in
EXPR - If
ignore_nan=False, then result is NaN if one metric in EXPR was NaN. Default isignore_nan=True lines: - expression:
max(EXPR)- expression:min(EXPR)- expression:std(EXPR)#standard deviation - expression:average(EXPR)#same as mean - expression:mean(EXPR)#example mean load average- at each time-point take aggregation function for all metrics in
EXPR lines: - expression:
sum_by(label_name, [other_label,] EXPR)#example processes cpu usage - expression:max_by(label_name, [other_label,] EXPR)- expression:min_by(label_name, [other_label,] EXPR)- expression:std_by(label_name, [other_label,] EXPR)#standard deviation - expression:mean_by(label_name, [other_label,] EXPR)#same as average - expression:average_by(label_name, [other_label,] EXPR)#example mean load average- group all metrics in
EXPRby value oflabel_namelabel and aggregate metrics in the same group into one metric - Accepts parametr
ignore_nan=False|True, just like ordinarysum lines: - expression:
win_sum(window_size_in_seconds, EXPR)- expression:win_mean(window_size_in_seconds, EXPR)#same as win_avg - expression:win_min(window_size_in_seconds, EXPR)- expression:win_max(window_size_in_seconds, EXPR)- expression:win_std(window_size_in_seconds, EXPR)- expression:win_avg(window_size_in_seconds, EXPR)#example mean load average on hour window- Applies specified function
sum|mean|min|max|stdfor each metric inEXPRon moving time windowwindow_size_in_seconds. See Moving average lines: - expression:
cum_sum(EXPR)#example- Cumulative sum for each metric in
EXPR. lines: - expression:
top(N, EXPR[, include_other=true|false][, by="exp"|"sum"|"max"])#example top 5 processes by CPU - expression:bottom(N, EXPR[, by="exp"|"sum"|"max"])- show top|bottom
Nmetrics fromEXPRbyews|exp(exponentialy weighted sum) orsumormaxin current timespan lines: - expression:
filter_with(EXPR, FILTER_EXPR)#example memory usage of long running processes- filters metrics in
EXPRreturning only those for whichFILTER_EXPRnot zero (or NaN). lines: - expression:
const(v[, label="value", ...])#example- constant metric with value
vand additonal labels for legend lines: - expression:
time()- timestamp from x-axis as y-value
lines: - expression:
from_string("1,2,3,3,2,1,", [,repeat=false] [,sep=' '] [,label="value", ...])#example- construct metric from string like
"1,2,3,3,2,1,", where each number becomes the value of the metric for corresponding minute lines: - expression:
defined(EXPR)#example all processes1if there is data fromEXPRin this time-point or0if there is NaNlines: - expression:
replace(old_val, new_val, EXPR)#example - expression:n2z(EXPR)#shortcut for "replace(nan, 0, EXPR)" - expression:zero_if_none(EXPR)#shortcut for "replace(nan, 0, EXPR)" - expression:z2n(EXPR)#shortcut for "replace(0, nan, EXPR)" - expression:zero_if_negative(EXPR)- expression:none_if_zero(EXPR)#shortcut for "replace(0, nan, EXPR)" - expression:remove_below(EXPR, value)- expression:remove_above(EXPR, value)- expression:clamp_min(EXPR, min)- expression:clamp_max(EXPR, max)- sets
new_valinstead ofold_val lines: - expression:
sum_by(label, [other_label,] metric(..)) / max_by(label, [other_label,] metric(.))- expression:sum_by(label, [other_label,] metric(..)) * sum_by(label, [other_label,] metric(.))- expression:sum_by(label, [other_label,] metric(..)) - min_by(label, [other_label,] metric(.))- if labels for both
sum_byare the same than it evaluates as/*or-for each pair of metrics (one from left and one from rightmetric) lines: - expression:
sum_by(label, [other_label,] metric(..)) / EXPR- expression:min_by(label, [other_label,] metric(..)) * EXPR- expression:max_by(label, [other_label,] metric(..)) - EXPR- Applies
/ EXPR* EXPRor- EXPRfor each metric from leftXXX_by(label, ...)
Lines legend syntax:
lines: - expression: metric(...) legend:
'%s'- for each line show all
label_name:label_valuepairs in legend lines: - expression: metric(...) legend:
'%(label_name)s anything'- for each line show
`label_value` anythingin legend
Colors syntax:
lines: - expression: metric(...) color:
'#81ff22'- expression: metric(...) color:'red'- color is color
lines: - expression: metric(...) colors:
['#80AB00', 'red', 'rgb(127,0,20)', 'hsla(100,10%,20%,0.8)']- will cycle through specified colors
lines: - expression: metric(...) colors:
/regex.*/: '#fff' /regex2/: 'gold'- will match legend to regexes
lines: - expression: metric(...) options: colors:
semaphore#OR colors:semaphore inv- will color all, previously uncolored lines, with a gradient from
redtogreen - or from
greentoredifsemaphore inv
Sorting syntax:
lines: - expression: metric(...) options: sort:
alpha|num- sort all lines by legend in alphabetical or numeric (default) order
lines: - expression: metric(...) options: sort:
['fixed', 'order', 'for', 'legend', 'items']- fixed sort order by item's
legend lines: - expression: metric(...) options: sort: ... order:
DESC- change sort order to descending
Sorting tooltip syntax:
lines: - expression: metric(...) options: tooltip: sort_order:name|-name|value|-value-
Sorting lines in tooltip (no sorting by default):
- in alphabetical(
name|-name). - by values(
value|-value)
- in alphabetical(
Captions:
lines: - expression: metric(...) - expression: metric(...)
title: 'some %(label_name)s'- format chart title with labels from all expressions combined
lines: - expression: metric(...) - expression: metric(...) options:
y_title: 'some text'- Y-axis vertical title as plain text