Our client required an in-house Gitlab Runner instead of using a shared runner. A self managed runner offers more flexibility and higher security to run heavy jobs or multiple parallel jobs with efficiency. Let us consider a job where SonarQube that handles your code quality analysis is hosted in your internal environment. Here, if the runner is also in-house, then we don’t need to expose SonarQube publicly.
You can use a configuration template file to configure the runner and any field on it.
runners:
config: |
[[runners]]
[runners.kubernetes]
image = “ubuntu:16.04”
namespace = “{{.Release.Namespace}}”
All configuration options supported by the Kubernetes executor are listed in the Kubernetes executor docs, for many of the fields.
We can set memory and CPU limits for the helper runner under which our job will run.
config: |
[[runners]]
[runners.kubernetes]
namespace = “{{.Release.Namespace}}”
image = “ubuntu:16.04”
privileged = true
helper_cpu_limit = “2”
helper_memory_limit = “2Gi”
You can also enable the privileged option under this configuration template which will help you run docker-in-docker jobs. For this, you have to specify the executor under the runner section.
config: |
[[runners]]
executor = “kubernetes”
environment = [“DOCKER_HOST=tcp://localhost:2376”]
[runners.kubernetes]
privileged = true
[[runners.kubernetes.volumes.empty_dir]]
name = “docker-certs”
mount_path = “/certs/client”
medium = “Memory”
Install the runner
helm repo add gitlab https://charts.gitlab.io
helm install –namespace gitlab-runner gitlab-runner gitlab/gitlab-runner –create-namespace
Register the runner
helm upgrade gitlab-runner –set gitlabUrl=https://gitlab.com,runnerRegistrationToken=
<xxxxxxxx> gitlab/gitlab-runner –namespace=gitlab-runner
For gitlab Token goto, Settings -> CI/CD ->Runners
Create a Role and role binding for the jobs to run on K8s
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: gitlab-runner
namespace: gitlab-runner
rules:
– apiGroups: [“”]
resources: [“pods”, “secrets”, “configmaps”, “pods/exec”, “pods/attach”]
verbs: [“get”, “list”, “watch”, “create”, “patch”, “delete”, “update”]
Create Role binding
kubectl create rolebinding –namespace=gitlab-runner gitlab-runner-binding –role=gitlab-runner –serviceaccount=gitlab-runner:default
Your gitlab runner is installed on your cluster.
Now, for further configuration, you can add values under the config.runner section.
Let us see how we can do caching in runner with s3
To use the cache with your configuration template, set the following variables in values.yaml.
runners:
config: |
[[runners]]
executor = “kubernetes”
environment = [“DOCKER_HOST=tcp://localhost:2376”]
[runners.kubernetes]
namespace = “{{.Release.Namespace}}”
image = “ubuntu:16.04”
privileged = true
helper_cpu_limit = “2”
helper_memory_limit = “2Gi”
[[runners.kubernetes.volumes.empty_dir]]
name = “docker-certs”
mount_path = “/certs/client”
medium = “Memory”
[runners.cache]
Type = “s3”
Path = “cache”
Shared = true
[runners.cache.s3]
ServerAddress = “s3.amazonaws.com”
AccessKey = ” “
SecretKey = ” “
BucketName = “abcd”
BucketLocation = “REGION”
You can also use type secret to pass accesskey and secretkey.
kubectl create secret generic s3runner-access -n gitlab-runner
–from-literal=accesskey=”YourAccessKey”
–from-literal=secretkey=”YourSecretKey”
Under the runner.cache secretion, add a secret name.
cache:
secretName: s3runner-access
Now let us see autoscaling in EKS with spot instances for runners.
Solution approach
We need two node groups:
Create node group after EKS cluster creation with a tag of below keys and values
k8s.io/cluster-autoscaler/<cluster-name>: owned
k8s.io/cluster-autoscaler/enabled : TRUE
Creating an IAM policy for your service account will allow your CA pod to interact with the autoscaling groups.
Cat > k8s-asg-policy.json
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Action”: [
“autoscaling:DescribeAutoScalingGroups”,
“autoscaling:DescribeAutoScalingInstances”,
“autoscaling:DescribeLaunchConfigurations”,
“autoscaling:DescribeTags”,
“autoscaling:SetDesiredCapacity”,
“autoscaling:TerminateInstanceInAutoScalingGroup”,
“ec2:DescribeLaunchTemplateVersions”
],
“Resource”: “*”,
“Effect”: “Allow”
}
]
}
aws iam create-policy –policy-name k8s-asg-policy –policy-document file://k8s-asg-policy.json
Deploy the Cluster Autoscaler to your cluster with the following command:
https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml
Edit the yaml file “kind deployment” under the command section, give your cluster name and check the tag. It should be the same name given while creating the node group.
You can also pass other commands as required. By default, the cluster autoscaler will wait 10 minutes between scale-down operations. You can adjust this using –scale-down-delay-after-add, –scale-down-delay-after-delete, and –scale-down-delay-after-failure flag. E.g. –scale-down-delay-after-add=5m to decrease the scale-down delay to 5 minutes after a node has been added.
kubectl apply -f ./cluster-autoscaler-autodiscover.yaml
Now whenever a runner runs a job, the autoscaler will scale the node group based on the requirement and it will be zero node when there are zero jobs.
Follow us on our LinkedIn Page. To know more about our services, visit our website.
CloudifyOps Pvt Ltd, Ground Floor, Block C, DSR Techno Cube, Survey No.68, Varthur Rd, Thubarahalli, Bengaluru, Karnataka 560037
Indiqube Vantage, 3rd Phase, No.1, OMR Service Road, Santhosh Nagar, Kandhanchavadi, Perungudi, Chennai, Tamil Nadu 600096.
CloudifyOps Inc.,
200, Continental Dr Suite 401,
Newark, Delaware 19713,
United States of America
Copyright 2024 CloudifyOps. All Rights Reserved