While running an application on Kubernetes, it is essential to view/check the application logs. If the pods go down or if we are deploying a new image, we will lose log data. It is not possible to troubleshoot any application issue without the log data.
To solve this problem, Cloudifyops recommended implementing a log management solution using Loki, Promtail, and Grafana.
Loki is a multi-tenant log aggregation system that works seamlessly with various frameworks such as Kibana and Promtrail. It can be used to collect and index logs, similar to the Elasticsearch, Logstash and Kibana (ELK)/ Elasticsearch, Fluentd and Kibana (EFK) stack. Instead of having contents of all the logs, Loki indexes just the metadata. This design makes the index smaller, which lowers the cost. Cloud storage allows you to store logs directly in the cloud without the need to store them on-disk. This simplifies operations and avoids issues like running out of disc-space.
Promtail is an agent that automatically detects and ships targets to Loki. It works seamlessly with all types of applications and services.
Grafana is a free and open source visualization and analytics software. It allows you to query, monitor, set up an alert, and study your data from any location. It gives you the tools to turn your time-series database (TSDB) data into stunning graphs and visualizations.
We will deploy the log management solution using the Helm charts. Helm must be installed in the machine before we begin. If you do not have it, install it by following the steps listed here.
Add the Helm repository using below command
Update the helm repo
Search for Grafana in Helm
You will get an output as below.
Before deploying, we need to change some values in the values.yaml file. To do this, store the file in tmp and make the necessary changes. Now we can deploy it.
Delete unnecessary things like filebeat, logstash, Prometheus and fluent bit. By default, Grafana is not enabled. Once you enable Grafana, we need to enable and add storage persistence to the Loki at least.
To make changes to the file
File should look like this after the changes are done
Next, install the Loki stack using the new values.yaml file
helm install loki-stack grafana/loki-stack –values /tmp/loki-stack-values.yaml -n loki –create-namespace
The above command will create one new namespace called Loki and everything is deployed into that namespace.
Check if the deployments have been successful, in Loki namespace
For the Grafana password, we need to follow the below steps
Password is encoded in base64. We need to decode that with the command
The output is the password for Grafana and the username is “admin”.
To check the service in Loki namespace, use the below command.
By default, we will get the cluster IP and we can add ingress to that.
To test this, we can edit svc and make it the node port so that we can connect remotely.
Change the service type to nodeport and add nodeport so that we can connect remotely (changing service type for testing only in prod we will add ingress).
After performing the above steps, run the below command for port number.
Note the port number on which Loki-stack-Grafana is running
http://<.node.public.ip.>:portnumer
Note:
Check the cluster level firewalls and security groups. By default, the firewalls will not allow traffic from everywhere.
This is the Grafana login page with username admin and the password that we decoded earlier. We need to use these credentials to login to Grafana.
Once logged in, go to the explore tab and choose a data source as Loki.
To learn more about these cutting edge technologies & real time industry applied best practices, follow our LinkedIn Page. To explore our services, visit our website.
CloudifyOps Pvt Ltd, Ground Floor, Block C, DSR Techno Cube, Survey No.68, Varthur Rd, Thubarahalli, Bengaluru, Karnataka 560037
Indiqube Vantage, 3rd Phase, No.1, OMR Service Road, Santhosh Nagar, Kandhanchavadi, Perungudi, Chennai, Tamil Nadu 600096.
CloudifyOps Inc.,
200, Continental Dr Suite 401,
Newark, Delaware 19713,
United States of America
