How to upgrade Ansible AWX running in Kubernetes

The AWX project provides Developers and System Engineers with a REST API, web-based user interface, and task engine built on top of Ansible. AWX is one of the upstream projects for Red Hat Ansible Automation Platform designed for managing complex multi-tier deployments powered by Ansible automation framework.

By automating your Infrastructure and Applications deployment with AWX, you’ll experience a smooth process geared for better efficiency and productivity. In our previous articles we discussed the installation of AWX on Kubernetes using k3s as a distribution to run Kubernetes. The links to the recent articles are available in the following URLs:

• Install Ansible AWX on CentOS 8 / Rocky Linux 8
• How To Install Ansible AWX on Debian
• How To Install Ansible AWX on Ubuntu

Upgrade Ansible AWX running in Kubernetes

This upgrade is only for AWX deployed in a Kubernetes Cluster using AWX Operator. If your AWX instance was not deployed with the operator refer to official documentation for its upgrade process.

Upgrade pre-reqs:

• Working Kubernetes cluster with AWX deployed
• Access to Kubernetes API – in this example we’re using kubectl
• Granted permissions to the namespace where AWX is deployed (both Operator and instance)

1) Check operator image version

The process of upgrading AWX involves an upgrade of the awx-operator to the version that maps to the desired version of AWX.

Let’s find the version of AWX installed by the awx-operator:

# kubectl describe pod awx-operator-controller-manager | grep quay.io/ansible/awx-operator
    Image:         quay.io/ansible/awx-operator:0.17.0
    Image ID:      quay.io/ansible/[email protected]:2ffa0449b9ee0961df3e4794c5da5bcea2a0f7677df2ddad63e07652fd11ef54
  Normal  Pulling    12m   kubelet            Pulling image "quay.io/ansible/awx-operator:0.17.0"
  Normal  Pulled     12m   kubelet            Successfully pulled image "quay.io/ansible/awx-operator:0.17.0" in 655.666335ms

From the output we can confirm current image used is quay.io/ansible/awx-operator:0.17.0. This means we’re on AWX operator version 0.17.0. By upgrading the operator a newer version of AWX instance will be deployed.

To find the version of AWX that will be installed by the awx-operator by default, check the version specified in the image_version variable in roles/installer/defaults/main.yml for that particular release.

2) Clone AWX Operator project

Clone awx-operator project code from Github if you don’t have the latest base already.

git clone https://github.com/ansible/awx-operator.git
cd awx-operator

Set namespace to awx:

$ kubectl config set-context --current --namespace=awx
Context "default" modified.

Install curl and jq tools on your Workstation where kubectl is configured:

### Debian based systems ###
sudo apt update
sudo apt install curl jq

### RHEL based systems ###
sudo yum install epel-release
sudo yum install curl jq

Switch branch to the tag release of AWX operator you want to upgrade to, this example is for latest release:

RELEASE_TAG=`curl -s https://api.github.com/repos/ansible/awx-operator/releases/latest | grep tag_name | cut -d '"' -f 4`
echo $RELEASE_TAG
git checkout $RELEASE_TAG

If you don’t want latest release of Operator then specify Operator version:

git checkout <VERSION>

For the devel branch use:

git checkout devel

3) Remove deployed AWX Operator

Delete your old AWX Operator and existing awx-operator service account, role and role binding in awx namespace first:

$ kubectl delete  deployment awx-operator-controller-manager
deployment.apps "awx-operator-controller-manager" deleted

$ kubectl  delete serviceaccount awx-operator-controller-manager
serviceaccount "awx-operator-controller-manager" deleted

$ kubectl delete rolebinding awx-operator-awx-manager-rolebinding
rolebinding.rbac.authorization.k8s.io "awx-operator-awx-manager-rolebinding" deleted

$ kubectl delete role awx-operator-awx-manager-role
role.rbac.authorization.k8s.io "awx-operator-awx-manager-role" deleted

Check status of the Pods in the cluster after removal. There should be no Pods with the name awx-operator-controller-manager:

$ kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
awx-postgres-0         1/1     Running   0          30m
awx-66596c8fcb-s28tw   4/4     Running   0          29m

4) Upgrade Ansible AWX running in Kubernetes

Deploy new AWX Operator into your Kubernetes cluster:

export NAMESPACE=awx
make deploy

Here is a sample installation output:

namespace/awx unchanged
customresourcedefinition.apiextensions.k8s.io/awxbackups.awx.ansible.com unchanged
customresourcedefinition.apiextensions.k8s.io/awxrestores.awx.ansible.com unchanged
customresourcedefinition.apiextensions.k8s.io/awxs.awx.ansible.com configured
serviceaccount/awx-operator-controller-manager created
role.rbac.authorization.k8s.io/awx-operator-awx-manager-role created
role.rbac.authorization.k8s.io/awx-operator-leader-election-role unchanged
clusterrole.rbac.authorization.k8s.io/awx-operator-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/awx-operator-proxy-role unchanged
rolebinding.rbac.authorization.k8s.io/awx-operator-awx-manager-rolebinding created
rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/awx-operator-proxy-rolebinding unchanged
configmap/awx-operator-awx-manager-config unchanged
service/awx-operator-controller-manager-metrics-service unchanged
deployment.apps/awx-operator-controller-manager created

The awx-operator-controller-manager pod should be up in few seconds:

$ kubectl get pods
NAME                                               READY   STATUS    RESTARTS   AGE
awx-postgres-0                                     1/1     Running   0          46m
awx-66596c8fcb-s28tw                               4/4     Running   0          46m
awx-operator-controller-manager-5ddf49cc4f-ptskm   2/2     Running   0          25s

We can now check AWX Operator container image used in this deployment:

$ kubectl describe pod awx-operator-controller-manager | grep quay.io/ansible/awx
    Image:         quay.io/ansible/awx-operator:0.18.0
    Image ID:      quay.io/ansible/[email protected]:270cfc3c468bf44300fca3a1e952436fd7b5f9eea88b31f3c4e78eb31af17d6f
  Normal  Pulled     3m8s  kubelet            Container image "quay.io/ansible/awx-operator:0.18.0" already present on machine

The logs can be streamed as deployment is in progress with the commands below;

kubectl logs -f deployments/awx-operator-controller-manager -c awx-manager

If everything goes well there should be new Pods and old ones terminated – For both PostgreSQL and AWX.

$ kubectl get pods
NAME                                              READY   STATUS    RESTARTS   AGE
awx-operator-controller-manager-7c4b7bb98-86nms   2/2     Running   0          93s
awx-postgres-0                                    1/1     Running   0          54s
awx-77f6fd7586-qzgfc                              4/4     Running   0          35s

Check if there is an update on the Server NodePort after the upgrade. From version 0.18.0 of AWX Operator, NodePort is set to 30080.

$ kubectl get svc
NAME                                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
awx-operator-controller-manager-metrics-service   ClusterIP   10.43.165.144   <none>        8443/TCP       69m
awx-postgres                                      ClusterIP   None            <none>        5432/TCP       67m
awx-service                                       NodePort    10.43.213.192   <none>        80:30080/TCP   67m

5) Access AWX Web Dashboard

Login to your AWX on http://serverip:30080. Replace 30080 with NodePort from your service check output:

You admin user password can be checked with the following commands:

kubectl -n awx get secret awx-admin-password -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'

Sample output:

password: QCnhhg4L2iDAn5ZxpZlB6jPSyFurN4bV

Use admin as username and printed password to authenticate

Confirm you can access AWX dashboard once you’re authenticated.

If you experience any issue with the upgrade process let us know through the comments section and we’ll be happy to help with troubleshooting. For successful upgrades feel free to check other guides on Ansible available in our website: