| Outages like: |
|---|
zookeeper [nosql1:zookeeper] cluster currently not available
zookeeper cluster is down
2021-06-03 05:20 (35 minutes)
|
| Found similar: |
|---|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 20:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 19:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 18:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 17:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 16:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 15:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 14:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 13:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 12:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 11:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 10:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 09:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 08:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 07:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 06:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 05:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 04:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 03:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 02:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 01:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-30 00:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 23:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 22:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 21:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 20:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 19:16 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 18:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 17:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 16:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 15:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 14:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 13:15 (4 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 12:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 11:15 (4 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 10:15 (4 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 09:15 (3 minutes)
|
zookeeper [nosql1:zookeeper] cluster currently not available
zookeeper cluster is down
2021-05-29 08:18 (1 minute)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 08:16 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 07:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 06:15 (4 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 05:15 (4 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 04:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 03:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 02:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 01:16 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-29 00:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 23:15 (4 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 22:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 21:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 20:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 19:15 (4 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 18:16 (2 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 17:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 16:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 15:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 14:15 (4 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 13:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 12:15 (4 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 11:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 10:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 09:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 08:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 07:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 06:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
possible false positive due to
okmeter outage
2021-05-28 05:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 04:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 03:16 (2 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 02:16 (2 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 01:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-28 00:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-27 23:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-27 22:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
possible false positive due to
okmeter outage
2021-05-27 21:16 (2 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
possible false positive due to
okmeter outage
2021-05-27 20:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
possible false positive due to
На ноде cl1ojovqhk3qm0ungvip-ujoh были ошибки udp пакетов.
Помог перезапуск collectorov на этой node.
Полезные график;
https://alfa.okmetric.com/okmeter/graph?duration=1h&graph_config=group_by%3A%20source_hostname%0Aoptions%3A%0A%20%20y_title%3A%20connections%0Alines%3A%0A%20%20-%20expression%3A%20%22sum_by(instance%2C%20metric(name%3D%27collector.tcp_collector.connections.count%27))%20*%20max_by(instance%2C%20defined(metric(name%3D%27collector.tcp_collector.connections.gc.time.sum%27)))%22%0A%20%20%20%20type%3A%20area%0A%20%20-%20expression%3A%20%22sum(max_by(instance%2C%20metric(name%3D%27collector.tcp_collector.connections.threshold%27))%20%20*%20max_by(instance%2C%20defined(metric(name%3D%27collector.tcp_collector.connections.gc.time.sum%27))))%22%0A%20%20%20%20legend%3A%20configured%20threshold%0Aevents%3A%0A%20%20-%20collector-errors%0A%20%20-%20tcp_collector_critical%0Atitle%3A%20Active%20connections%0A
https://alfa.okmetric.com/okmeter/hosts/cl1ojovqhk3qm0ungvip-ujoh/netstat?&duration=1h
2021-05-27 19:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
possible false positive due to
На ноде cl1ojovqhk3qm0ungvip-ujoh были ошибки udp пакетов.
Помог перезапуск collectorov на этой node.
Полезные график;
https://alfa.okmetric.com/okmeter/graph?duration=1h&graph_config=group_by%3A%20source_hostname%0Aoptions%3A%0A%20%20y_title%3A%20connections%0Alines%3A%0A%20%20-%20expression%3A%20%22sum_by(instance%2C%20metric(name%3D%27collector.tcp_collector.connections.count%27))%20*%20max_by(instance%2C%20defined(metric(name%3D%27collector.tcp_collector.connections.gc.time.sum%27)))%22%0A%20%20%20%20type%3A%20area%0A%20%20-%20expression%3A%20%22sum(max_by(instance%2C%20metric(name%3D%27collector.tcp_collector.connections.threshold%27))%20%20*%20max_by(instance%2C%20defined(metric(name%3D%27collector.tcp_collector.connections.gc.time.sum%27))))%22%0A%20%20%20%20legend%3A%20configured%20threshold%0Aevents%3A%0A%20%20-%20collector-errors%0A%20%20-%20tcp_collector_critical%0Atitle%3A%20Active%20connections%0A
https://alfa.okmetric.com/okmeter/hosts/cl1ojovqhk3qm0ungvip-ujoh/netstat?&duration=1h
2021-05-27 18:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-27 17:15 (4 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-27 16:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
possible false positive due to
okmeter outage
2021-05-27 15:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-27 14:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-27 13:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-27 12:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
possible false positive due to
Кратковременная потеря метрик. https://alfa.okmetric.com/okmeter/graph?duration=1h&graph_config=options%3A%0A%20%20y_title%3A%20metrics%2Fsecond%0Alines%3A%0A%20%20-%20expression%3A%20%27top(5%2C%20sum_by(site_name%2C%20metric(name%3D%5B%22gokserver.collector.metrics_written%22%2C%20%22gokserver.tcp_collector.metrics_written.rate%22%2C%20%22collector.tcp_collector.metrics_written.rate%22%5D)))%27%0A%20%20%20%20type%3A%20area%0A%20%20%20%20legend%3A%20%27%25(site_name)s%27%0Aevents%3A%0A%20%20-%20collector-errors%0A%20%20-%20tcp_collector_critical%0Atitle%3A%20Metrics%20written%0A
2021-05-27 11:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-27 10:16 (2 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-27 09:16 (2 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-27 08:15 (4 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-27 07:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-27 06:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
possible false positive due to
был какой-то скачок на графиках, но сейчас все ровно и хорошо
2021-05-27 05:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
possible false positive due to
был какой-то скачок на графиках, но сейчас все ровно и хорошо
2021-05-27 04:15 (4 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-27 03:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-27 02:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-27 01:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-27 00:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-26 23:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-26 22:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-26 21:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-26 20:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-26 19:15 (3 minutes)
|
zookeeper [nosql2:zookeeper] cluster currently not available
2021-05-26 18:15 (3 minutes)
|