While running an application on Kubernetes, it is essential to view and check the application logs. If the pods go down or a new image is deployed, log data is lost, making it impossible to troubleshoot application issues effectively. To maintain visibility and ensure seamless debugging, implementing centralized logging solutions is crucial. Additionally, securing log data is an important aspect of Cloud Security, as it helps detect potential threats, unauthorized access, and system vulnerabilities. By leveraging robust logging and monitoring practices, organizations can enhance security, improve operational efficiency, and ensure compliance with industry standards.
To solve this problem, Cloudifyops recommended implementing a log management solution using Loki, Promtail, and Grafana. This approach ensures centralized logging, making it easier to troubleshoot application issues even if pods go down or a new image is deployed. Additionally, integrating a robust log management system is a key aspect of Cloud management services, helping organizations enhance visibility, maintain security, and optimize cloud operations
Loki:
Loki is a multi-tenant log aggregation system that works seamlessly with various frameworks such as Kibana and Promtrail. It can be used to collect and index logs, similar to the Elasticsearch, Logstash and Kibana (ELK)/ Elasticsearch, Fluentd and Kibana (EFK) stack. Instead of having contents of all the logs, Loki indexes just the metadata. This design makes the index smaller, which lowers the cost. Cloud storage allows you to store logs directly in the cloud without the need to store them on-disk. This simplifies operations and avoids issues like running out of disc-space.
Promtail:
Promtail is an agent that automatically detects and ships targets to Loki. It works seamlessly with all types of applications and services.
Grafana:
Grafana is a free and open source visualization and analytics software. It allows you to query, monitor, set up an alert, and study your data from any location. It gives you the tools to turn your time-series database (TSDB) data into stunning graphs and visualizations.
Requirements:
- We need one Kubernetes cluster.
Steps to follow:
We will deploy the log management solution using the Helm charts. Helm must be installed in the machine before we begin. If you do not have it, install it by following the steps listed here.
Add the Helm repository using below command
Update the helm repo
Search for Grafana in Helm
You will get an output as below.
Before deploying, we need to change some values in the values.yaml file. To do this, store the file in tmp and make the necessary changes. Now we can deploy it.
- helm show values grafana/loki-stack > /tmp/loki-stack-values.yaml
Delete unnecessary things like filebeat, logstash, Prometheus and fluent bit. By default, Grafana is not enabled. Once you enable Grafana, we need to enable and add storage persistence to the Loki at least.
To make changes to the file
- nano /tmp/loki-stack-values.yaml
File should look like this after the changes are done
Next, install the Loki stack using the new values.yaml file
helm install loki-stack grafana/loki-stack –values /tmp/loki-stack-values.yaml -n loki –create-namespace
The above command will create one new namespace called Loki and everything is deployed into that namespace.
Check if the deployments have been successful, in Loki namespace
For the Grafana password, we need to follow the below steps
- kubectl -n loki get secret loki-stack-grafana -o yaml
Password is encoded in base64. We need to decode that with the command
- echo ‘ ‘ | base64 -d; echo
The output is the password for Grafana and the username is “admin”.
To check the service in Loki namespace, use the below command.
By default, we will get the cluster IP and we can add ingress to that.
To test this, we can edit svc and make it the node port so that we can connect remotely.
- kubectl edit svc loki-stack-grafana -n loki
Change the service type to nodeport and add nodeport so that we can connect remotely (changing service type for testing only in prod we will add ingress).
After performing the above steps, run the below command for port number.
Note the port number on which Loki-stack-Grafana is running
http://<.node.public.ip.>:portnumer
Note:
Check the cluster level firewalls and security groups. By default, the firewalls will not allow traffic from everywhere.
This is the Grafana login page with username admin and the password that we decoded earlier. We need to use these credentials to login to Grafana.
Once logged in, go to the explore tab and choose a data source as Loki.
To learn more about these cutting edge technologies & real time industry applied best practices, follow our LinkedIn Page. To explore our services, visit our website
