From the development department: How we achieve continuous improvement at Mollie
This week Mollie launched four major updates: we dropped the pricing of iDEAL and Bancontact/Mister Cash, changed the pricing model of our credit card product, launched Mollie Checkout and added the new Customer API.
All of this was done without any downtime or disturbances in the services provided to our merchants. What enables us to do this?
The core component which enables continuous improvement is our continuous integration set up. This consist of a range of systems which, after every change made by a developer, automatically run a series of tests to determine if the new changes are safe to deploy to the production environment. This includes about 10,000 unit- and integration tests, various linting checks and static analysis checks.
Any defects that make it to production result in an extra check added to our CI environment, preventing future regressions.
Having continuous integration is not free. It needs to be set up and maintained just like any other system. As test suites grow (our test suite size doubles about every year), the build time grows with it. This will slow down your team, so you must invest in keeping build time low. Tests need to be absolutely reliable – they must only fail because of a defect in the code and not for any other reason (such as network or timing issues). Finally, the CI environment must mirror the production environment as closely as possible. We have invested heavily in a quick and reliable CI environment.
Continuous integration gives us the confidence that a release can be deployed to the production environment safely, without negatively impacting our customers.
Instant error notification
Even with the safety net of continuous integration, exceptions and errors can still occur in the production environment. We use the PHP APM extension to log these errors including their stack traces to a separate database. The errors are then immediately evaluated by our operations team. Because the stack trace with arguments and the request environment is available to this team, most of the time the error can easily be diagnosed and resolved.
Usually, a new test case is created to automatically prevent future regressions.
Immediate notification of the occurrence of errors or exceptions in the production environment enables us to keep the impact of defects extremely small. In most cases, only a single HTTP request is affected before the issue is resolved.
The most important requirement for deployments is that they should not affect our customers in the slightest way. If we can deploy with confidence, deploying becomes easy and a quick turnaround for fixes and features is enabled.
We use Capistrano to deploy our code to the application servers. The deployment is scripted and can be started with a single command. There are no manual steps, hence no human errors to be made.
The new release is copied to the application servers and then a symlink to the Apache DocumentRoot is flipped from the current release to the new release. This enables an atomic deploy where the entire application is instantly replaced by the new release.
Because the deployment process is so reliable, we are able to deploy dozens of times a day.
These three components enable us to release new changes with confidence. Defects in code are mostly prevented by our continuous integration environment. Problems that make it to production can easily be diagnosed and resolved thanks to swift error notification. Continuous deployment enables us to immediately push fixes and features to production without affecting any merchants.
Releasing with confidence allows us to quickly release new innovations, features and other improvements to all our customers.
Mollie is hiring
We are looking for talented developers to expand our team. Mollie is growing quickly and has ambitious plans for 2016. If you’d like to have a coffee with us and discuss the possibilities, get in touch.