PostgreSQL major versions are released every year, with each release delivering better performance and new features. With such rapid innovation, it is inevitable that there will be a need to upgrade from one version to another. Upgrade procedures are usually very complex and require thorough planning.
With the 2.4.0 release of Percona Operator for PostgreSQL, users can now upgrade to the new major version automatically. The feature is currently in technical preview.
Try it out
To demonstrate it I will deploy the Operator and a cluster with PostgreSQL version 15. The goal would be to upgrade to PostgreSQL version 16.
Deploy the Operator:
1 | kubectl apply -f deploy/bundle.yaml --server-side |
We use version 16 by default in our Custom Resources. To set the version, I set the following lines in the cr.yaml:
1 2 | image: percona/percona-postgresql-operator:2.4.0-ppg15-postgres postgresVersion: 15 |
Please note that right now, the major version upgrade only works for the images without the minor version tag. It will not work for images like percona/percona-postgresql-operator:2.4.0-ppg15.7-postgres.
Deploy the cluster:
1 | kubectl apply -f deploy/cr.yaml |
In 2.4.0, we introduced the new Custom Resource PerconaPGUpgrade. It has a simple structure:
1 2 3 4 5 6 7 8 9 | apiVersion: pgv2.percona.com/v2 kind: PerconaPGUpgrade metadata: name: cluster1-15-to-16 spec: postgresClusterName: cluster1 image: percona/percona-postgresql-operator:2.4.0-upgrade fromPostgresVersion: 15 toPostgresVersion: 16 |
- postgresClusterName – specify the name of the cluster you want to upgrade
- fromPostgresVersion – the current version of the cluster
- toPostgresVersion – the version to which the upgrade will happen
Before running the upgrade, test the procedure on the non-production environment and take the full backup.
Apply the upgrade resource:
1 | kubectl apply -f deploy/upgrade.yaml |
Below we explain what happens under the hood when you create the PerconaPGUpgrade resource.
What’s under the hood
Once PerconaPGUpgrade resource is created, the Operator does the following:
- Adds pgv2.percona.com/allow-upgrade annotation to the PerconaPGCluster resource. This instructs the Operator that the upgrade should start.
- Operator pauses the cluster. The cluster will be unavailable for the duration of the upgrade. Make sure that you plan maintenance accordingly.
- Pods are created from the image defined in the upgrade resource:
1 | image: percona/percona-postgresql-operator:2.4.0-upgrade |
This is a special image that holds multiple PostgreSQL versions in it. You can find its Dockerfile here.
The Pods mount the data volumes (Persistent Volume Claims) of the instances and executes pg_upgrade. It copies the data without the need to perform dump/restore procedures and does its best to make sure the old and new clusters are binary-compatible. As the data is copied, make sure you have enough disk space.
You can track the upgrade status by checking the logs of the upgrade Pods. A Pod will be created for every instance.
1 | kubectl logs cluster1-15-to-16-pgdata-qzzxh |
Once the data is copied, the cluster automatically starts up.
Post-upgrade
As mentioned above, the data is copied during the upgrade process, but it is also retained. Meaning that the data from the starting major version is kept on the disk. You can remove it at your discretion by executing into containers and running the following commands (example for PostgreSQL 15):
1 2 | rm -rf /pgdata/pg15 rm -rf /pgdata/pg15_wal |
You can also delete the PerconaPGUpgrade resource; this will clean up the jobs and Pods created during the upgrade:
1 | kubectl delete perconapgupgrade cluster1-15-to-16 |
Troubleshooting
If the upgrade fails for some reason, the cluster will stay in paused mode. You can unpause it manually, and it will start with the old version.
To debug the issue:
- Check the PerconaPGUpgrade resource – kubectl get perconapgupgrade -o yaml.
- Check the logs of the upgrade pods.
Extensions
If you use any custom extensions, you need to make sure that you have them rebuilt for the new major version before the upgrade; otherwise, they will not work.
Backups
When the upgrade finishes the Operator automatically takes the full backup. Previous backups can be used to restore only to the older version. It is very important to ensure that you have a backup for the new version after the upgrade.
Conclusion
The introduction of the PerconaPGUpgrade resource in Percona Operator for PostgreSQL 2.4.0 simplifies the traditionally complex process of PostgreSQL major version upgrades. While still in technical preview, this feature promises to streamline the upgrade process for users, saving time and effort while reducing potential risks. Remember, as with any upgrade, testing in a non-production environment and creating full backups are essential precautions. We encourage you to try out this new functionality and share your feedback with the Percona community.
About the Author
Sergey ProninSergey is a product leader at Percona focusing on delivering robust open-source database and cloud-native solutions. Prior to Percona Sergey led product management and engineering teams in other organizations with a primary focus on products in infrastructure and platforms space.
