Monitoring Your Kubernetes Cluster + Auto Scaling GCE + Regular GCE Instances on Google Cloud Platform with Prometheus, Grafana and Graylog

Monitoring Your Kubernetes Cluster + Auto Scaling GCE + Regular GCE Instances on Google Cloud Platform with Prometheus, Grafana and Graylog

Intro

As one can understand from the topic, I had a task to set up monitoring for a project on Google Cloud Platform, which consisted from:

  • regular GCE instances
  • GCE auto scaling instances
  • Kubernetes cluster

This project had to have its own monitoring, which, in turn, had to be integrated with the main Prometheus instance used to monitor for many other projects that did not have their own Prometheus instance.

The tech stack that is going to be used to achieve the goal:

  1. Prometheus
  2. Prometheus exporters
  3. Prometheus Alertmanager
  4. NGINX
  5. Consul
  6. Grafana
  7. Graylog

Prometheus

In my setup I had 3 Prometheus instances:

  • the first one was part of a Kubernetes cluster
  • the second one was on a regular GCE VM instance, because we have many instances outside the GKE cluster
  • and the third one was the main Prometheus for centralized monitoring

Prometheus Kubernetes

There is a nice plugin for Grafana called grafana-kubernetes-app. That plugins comes with configs for Prometheus and its exporters as well as JSON dashboards for Grafana. Unfortunately, the default configuration does not work for GKE due to RBAC auth. I expanded the set of configs supplied with grafana-kubernetes-app by adding the following deployment configs:

  • RBAC authentication
  • Ingress – Prometheus KUBE metrics will be scraped by another Prometheus used for centralized monitoring
  • NGINX Proxy – Prometheus does not have any kind of Basic Auth, neither has GCE Ingress

Those configs can be downloaded at https://gitlab.blackmesafacility.com/kubernetes/k8s-prometheus-monitoring

Service Discovery

  • regular GCE instances – Consul
  • GCE auto scaling insctances – Prometheus built in GCE discovery functionality
  • Kubernetes Pods – Prometheus built in Kubernetes discovery via Kubernetes API

Metrics Exporters

  • regular GCE instances – Prometheus exporters installed once with Ansible
  • GCE auto scaling insctances – Prometheus exporters installed on every new GCE instance during the initial setup
  • Kubernetes Pods – metrics exported via Kubernetes API + Node Exporter for exporting K8s nodes metrics

1) Set up Prometheus + Consul + Grafana: https://gitlab.blackmesafacility.com/monitoring/prometheus-monitoring

2) Deploy Prometheus exporters:

https://gitlab.blackmesafacility.com/monitoring/prometheus-exporters

3) Add dasboards for different exporters

https://gitlab.blackmesafacility.com/monitoring/grafana-dashboards

4) Install the grafana-kubernetes-app plugin

5) Deploy the Prometheus on Kubernetes

https://gitlab.blackmesafacility.com/kubernetes/k8s-prometheus-monitoring

6) Deploy Graylog + Elasticsearch + MongoDB

https://gitlab.blackmesafacility.com/monitoring/graylog

Comments are closed.