Reach Us

Setting up a secure Log Management System for K8S cluster using Loki, Promtail, and Grafana

While running an application on Kubernetes, it is essential to view/check the application logs. If the pods go down or if we are deploying a new image, we will lose log data. It is not possible to troubleshoot any application issue without the log data.

To solve this problem, Cloudifyops recommended implementing a log management solution using Loki, Promtail, and Grafana.

Loki:

Loki is a multi-tenant log aggregation system that works seamlessly with various frameworks such as Kibana and Promtrail. It can be used to collect and index logs, similar to the Elasticsearch, Logstash and Kibana (ELK)/ Elasticsearch, Fluentd and Kibana (EFK) stack. Instead of having contents of all the logs, Loki indexes just the metadata. This design makes the index smaller, which lowers the cost. Cloud storage allows you to store logs directly in the cloud without the need to store them on-disk. This simplifies operations and avoids issues like running out of disc-space.

Promtail:

Promtail is an agent that automatically detects and ships targets to Loki. It works seamlessly with all types of applications and services.

Grafana:

Grafana is a free and open source visualization and analytics software. It allows you to query, monitor, set up an alert, and study your data from any location. It gives you the tools to turn your time-series database (TSDB) data into stunning graphs and visualizations.

Requirements:

  • We need one Kubernetes cluster.

Steps to follow:

We will deploy the log management solution using the Helm charts. Helm must be installed in the machine before we begin. If you do not have it, install it by following the steps listed here.

Add the Helm repository using below command

Update the helm repo

  • helm repo update

Search for Grafana in Helm

  • helm search repo Grafana

You will get an output as below.

No alt text provided for this image

Before deploying, we need to change some values in the values.yaml file. To do this, store the file in tmp and make the necessary changes. Now we can deploy it.

  • helm show values grafana/loki-stack > /tmp/loki-stack-values.yaml

Delete unnecessary things like filebeat, logstash, Prometheus and fluent bit. By default, Grafana is not enabled. Once you enable Grafana, we need to enable and add storage persistence to the Loki at least.

To make changes to the file

  • nano /tmp/loki-stack-values.yaml

File should look like this after the changes are done

No alt text provided for this image

Next, install the Loki stack using the new values.yaml file

helm install loki-stack grafana/loki-stack –values /tmp/loki-stack-values.yaml -n loki –create-namespace

The above command will create one new namespace called Loki and everything is deployed into that namespace.

Check if the deployments have been successful, in Loki namespace

  • kubectl get -n loki all
No alt text provided for this image

For the Grafana password, we need to follow the below steps

  • kubectl -n loki get secret loki-stack-grafana -o yaml
No alt text provided for this image

Password is encoded in base64. We need to decode that with the command

  • echo ‘<provide the admin password here > ‘ | base64 -d; echo

The output is the password for Grafana and the username is “admin”.

To check the service in Loki namespace, use the below command.

  • Ku get svc -n loki

By default, we will get the cluster IP and we can add ingress to that.

To test this, we can edit svc and make it the node port so that we can connect remotely.

  • kubectl edit svc loki-stack-grafana -n loki

Change the service type to nodeport and add nodeport so that we can connect remotely (changing service type for testing only in prod we will add ingress).

After performing the above steps, run the below command for port number.

  • Ku get svc -n loki

Note the port number on which Loki-stack-Grafana is running

No alt text provided for this image

http://<.node.public.ip.>:portnumer

Note:

Check the cluster level firewalls and security groups. By default, the firewalls will not allow traffic from everywhere.

No alt text provided for this image

This is the Grafana login page with username admin and the password that we decoded earlier. We need to use these credentials to login to Grafana.

Once logged in, go to the explore tab and choose a data source as Loki.

No alt text provided for this image

To learn more about these cutting edge technologies & real time industry applied best practices, follow our LinkedIn Page. To explore our services, visit our website.

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Youtube
Consent to display content from - Youtube
Vimeo
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google
Spotify
Consent to display content from - Spotify
Sound Cloud
Consent to display content from - Sound
Contact Us