Random   •   Archives   •   RSS   •   About   •   Contact

Selenium grid on Kubernetes

This post continues from where we left off on the Minikube guide. If you do not already have a Kubernetes cluster, you should read that first.

Selenium Grid allows you to build a cluster of Selenium nodes. Today we will create a Selenium cluster with 1 hub and 4 nodes on Kubernetes.

Selenium Hub

Let's launch the hub:

kubectl get pods
kubectl run selenium-grid --image selenium/hub:2.53.1 --port 4444
kubectl get pods

To access the deployment externally, we need to expose it:

kubectl get services
kubectl expose deployment selenium-grid --type=NodePort
kubectl get services

My deployment was exposed here:

http://192.168.99.100:31136/grid/console

To find out where your deployment was exposed:

minikube service selenium-grid --url

You may open this URI in a web browser.

Selenium Hub Overlearning

You can skip this section or try it for extra credit.

We may use kubectl exec for container introspection:

kubectl exec selenium-grid-3216163580-7pqtx -- ps aux

As you can see we have a java process running

kubectl exec selenium-grid-3216163580-7pqtx -- ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
seluser      1  0.0  0.1  18044  2688 ?        Ss   17:20   0:00 /bin/bash /opt/bin/entry_point.sh
seluser      7  0.1  3.1 2928868 64864 ?       Sl   17:20   0:13 java -jar /opt/selenium/selenium-server-standalone.jar -role hub -hubConfig /opt/selenium/config.json
seluser     87  0.0  0.1  34424  2856 ?        Rs   20:54   0:00 ps aux

We may also look at the config file and test the service internally:

# inspect the selenium config file.
kubectl exec selenium-grid-3216163580-7pqtx -- cat /opt/selenium/config.json

# see if selenium is really listening on port 4444.
kubectl exec selenium-grid-3216163580-7pqtx -- wget 127.0.0.1:4444 -O -

You can even shell into the container:

kubectl exec -it elenium-grid-3216163580-7pqtx -- /bin/bash

Exit out of the container and lets setup some Selenium Nodes!

Selenium Node

Lets spin up a Selenium Chrome node:

kubectl get pods
kubectl run selenium-node-chrome --image selenium/node-chrome:2.53.1 --env="HUB_PORT_4444_TCP_ADDR=selenium-grid" --env="HUB_PORT_4444_TCP_PORT=4444"
kubectl get pods

Kubernetes will use service discovery to resolve selenium-grid to the service (pods) running the hub!

If you refresh the hub browser window, you should see a connected Chrome Node, like this:

Selenium Hub with one connected Chrome Node.

Selenium Node Overlearning

You can skip this section or try it for extra credit.

The first time I tried to launch a Selenium node and I had trouble.

I ran this:

kubectl get pods
kubectl run selenium-node-chrome --image selenium/node-chrome:2.53.1
kubectl get pods

The new pod went into status CrashLoopBackOff:

NAME                                    READY     STATUS             RESTARTS   AGE
selenium-grid-3216163580-7pqtx          1/1       Running            1          3d
selenium-node-chrome-4019562870-mcpfg   0/1       CrashLoopBackOff   6          6m

To troubleshoot, I used the following commands:

kubectl describe pod selenium-node-chrome

This command allowed me to review the Kubernetes level logs. Everything seemed healthy so next I looked at the Docker level logs:

kubectl logs selenium-node-chrome-4019562870-mcpfg
Not linked with a running Hub container

Ok, the error Not linked with a running Hub container appears when Selenium node cannot find the hub.

Docker has a --link flag to link containers together, Kubernetes doesn't have this. After some research, it seems --link manages ENV vars.

You can see the environment vars of a pod using this command:

kubectl exec selenium-grid-3216163580-7pqtx -- printenv

I learned that the selinum-node-chrome docker image expects some ENV vars and if it doesn't get them, it goes into a crash loop.

I reached out over IRC in the #kubernetes and #selenium channels to ask about the ENV vars needed. A really helpful user named smccarthy linked me to this:

https://github.com/kubernetes/kubernetes/tree/master/examples/selenium

Apparently one of the example Kubernetes clusters is a Selenium Grid setup!

Looking over the example, I found the ENV vars that the selenium-node containers expect: HUB_PORT_4444_TCP_ADDR and HUB_PORT_4444_TCP_PORT

Man, why would they put the port (4444) in the key?

Anyways, we pass these key/values when creating the container like this:

kubectl run selenium-node-chrome --image selenium/node-chrome:2.53.1 --env="HUB_PORT_4444_TCP_ADDR=selenium-grid" --env="HUB_PORT_4444_TCP_PORT=4444"

Selenium Scale

Now we can scale up and down the Selenium Grid cluster. Lets scale the deployment to 4 replica node-chrome pods.

kubectl get pods
kubectl scale deployment selenium-node-chrome --replicas=4
kubectl get pods

Finally, if you refresh the hub browser window, you should see 4 connected Chrome nodes!

Selenium Hub with four connected Chrome Nodes.

If you liked this post, leave me a message in the comments!



© Russell Ballestrini.