The TYPO3 core CI uses the sysbox OCI runtime to run unprivileged DinD containers and some helper containers to increase performance.
TYPO3 core testing infrastructure – Part 1
GitLab CI for the TYPO3 Core Test Suite
This post is part of our TYPO3 core testing series
Introduction
The TYPO3 core has quite a history of system testing: A powerful and excessive test strategy has been developed over the years to verify stability and integrity at any given point in time.
Continuous integration is an invaluable helper for developers to create side-effect-free solutions, especially in critical and complex areas like the DataHandler where a broken patch could easily commit hazard to database structures. Lots of development time goes into related tests and they are a base allowing flawless refactorings of those structures in the first place.
All these testing efforts have a downside, though: They require continued maintenance and need quite a bit of infrastructure to not lead to frustration or slow down development. We find ourselves fiddling with test related details frequently. Test code and its infrastructure is a part of the project, it requires time to be kept up and running, just like all other parts of the core.
With this attitude in mind, this blog series is about a bigger chunk we tackled lately: The CI testing infrastructure based on bamboo didn’t age perfectly well and was recently superseded by a GitLab based setup. This comes with many advantages and a setup that is a generation younger. We’ll go into various DevOps details with this blog series and hope it is an interesting read for anyone looking into a decent (TYPO3 related) CI setup for agencies or projects.
Management summary
This blog is about technical insights and we’ll go into technical details. The overall picture can be interesting on a management level, though. Let’s sort that out first:
- GitLab is nowadays frequently used within agencies to host source code, track issues and perform continuous integration (CI) and continuous delivery (CD) tasks.
- The fully transparent and free setup of the TYPO3 core testing infrastructure can be used as a boilerplate, good practice, or at least as an integration example for an own solution. Some parts are supported or elevated by TYPO3 core to reduce agency efforts.
- The implementation keeps security in mind and takes actions to ensure an attacker can not break out of a test setup and peek around on what other jobs of different projects are doing. This can be an important asset for agencies preventing overly clever customers with code access from accessing foreign projects. A casual, ‘naive’ CI setup does not provide this kind of protection.
- The infrastructure auto-scales using the cost-effective hetzner cloud: When testing load is high, new machines are started automatically and ditched when there’s nothing left to do. This reduces hardware costs while still having a system that crunches through jobs quickly - Don’t be faced with bored developers waiting for jobs to finish, and don’t pay for hardware not needed at night.
Motivation
So we’ve put a couple of developer weeks into migrating away from the existing bamboo CI setup towards a GitLab based setup. bamboo served us well for about four years and it was the first setup that provided TYPO3 core with ‘pre-merge testing’: All major parts of the test suite are executed before a patch is merged. This allowed us to make sure no code is merged that makes our test suite “red”.
However, bamboo also showed some drawbacks:
- The test plans are cumbersome: They are based on Java code to define test jobs. But we’re PHP developers, Java is simply not deep in our blood. As a result, not too many core developers maintained and tuned the Java-based test plans over time.
- The one-agent-one-job restriction is inflexible: Atlassian bamboo licensing works based on the number of agents. The more online agents, the more you pay. The license construct leads to some technical limitations in bamboo: Each agent manages exactly one job at a time. This leads to an extensive setup with many agents on one hardware machine. Java is not exactly lightweight, having many agents per machine eats quite a bit of CPU and (more importantly) RAM.
- It feels as if bamboo is nowadays outpaced by more modern and less expensive solutions, especially in PHP-focused web agencies: We’d estimate that from 10 agencies maybe one uses bitbucket / bamboo and six use GitLab. This turns into a much broader knowledge towards GitLab.
- The atlassian ‘cloud only’ movement: Atlassian lately strives to ditch on-premise instances in favor of cloud solutions. This is unhandy for a CI setup that needs a significant amount of computational resources - a self-hosted solution can be much cheaper.
- The setup is partially intransparent: The particular docker based “docker-out-of-docker” setup was the right choice back then four years ago. Nowadays better solutions are available. We’ll go into these details later.
The TYPO3 GmbH maintained the given state of the infrastructure well over the years and did a great job keeping the service up and running. Don’t underestimate this work: Many thanks for the service and support! Nevertheless, there was a barrier between core development and maintaining bamboo: Updating the test plans needed a specialist from within the TYPO3 GmbH team which made them harder to roll out than it could be. - Hard to stabilize: We never succeeded in having truly stable test runs. Sometimes the docker networks did not initialize properly, or the acceptance tests suffered from timing issues when the test cluster was loaded. This is not the fault of bamboo per-se, but an issue with the setup we had in place. Nevertheless: It’s annoying for contributors pushing a patch and seeing unrelated tests fail.
Alternatives
As always, there are alternatives. Let’s have a rough look and some of them:
- Dedicated CI/CD projects like drone.io: drone is a pretty young project and looks like a fun thing to play with. The main reasons we didn’t have a deeper look at it were that the service seems to be a moving target at the moment and changes quite a bit as a young project, and not too much existing knowledge from other persons in our community was at hand.
- GitHub actions: GitHub actions is the go-to solution for many small projects and TYPO3 extensions out there: When there are just a couple of patches a day, the provided cost-free default plans crunch the casual testing and deployment jobs. GitHub actions can be extended with own on-premise runners, too. The API calls are limited however and a small calculation showed that the TYPO3 core testing could easily bust these limits.
Why GitLab?
So GitLab is a huge project with code hosting, issue tracker and a CI/CD integration. Even though we don’t need all of that for core testing, there are plenty of advantages left:
- It is frequently used within the community already, there are people with knowledge nearby. Of course b13 uses it, and the TYPO3 Server Team hosts a GitLab instance we can use for core testing purposes.
- GitLab is well documented with a powerful open source base and an active community. Searching for details almost always reveals answers from blog posts, official documentation or the GitLab project issue tracker.
- There are tons of CI options and the test plan is configured using straightforward YAML.
- There are flexible test-runner setups, including auto-scaling options.
Summary
We created a full new TYPO3 core testing setup based on GitLab CI since the existing bamboo setup aged and showed limits. We had a look at some alternatives and came to the conclusion our needs are best solved re-creating the setup from scratch. It should be a solid solution for the next few years. Some parts could be re-used by agencies.
We’ll start a deep dive into DevOps details with the next part of the series.