Magento 2: pipeline deployment and configuration management

Since the very first day of working with the Magento 2 platform, our DevOps teams had headache after headache because of “modern” deployments — hard to automate, hard to track results of separate tasks and DOWNTIMES.

Let’s start at the very beginning. Having worked with Magento 1 for a long time it was easy to deploy it. Even violating Continuous Delivery best practices, it was very, very simple. Just deliver new code to the server and reload the page. You could immediately see the result e.g. if migration was broken. You could just restore the previous code version and keep it running while the team fixed the issues.

Magento 2 introduced a big enhancement in this process. Actually, a lot of tools were available right out of the box, that was fully integrated into the platform, i.e. grunt, less, etc. Apart from the external tools, the application itself became more complex in terms of bootstrapping, launching, etc.

Let’s just skip discussion of the functionality hidden in the plugins, that tends to automate some actions and avoid misuse of tools and commands. Zero downtime deployment was impossible for us, and often we did not know whether deployment will give us some upsetting surprise.

Step-by-step of Magento 2 deployment:

1. First of all — get the code! We got Composer support out of the box, that’s awesome, but you need credentials for Magento core and modules hosted in the marketplace. Don’t store them under the git! (auth.json is NOT a good approach).
2. Configure service connections: database, redis, etc. The database is the most important here.
3. Our code should work now. But you must determine the state of your database:

a) The application is not installed (env.php is missing ‘install’ => ‘date’ section). Install Magento

b) The application is installed and up to date (it will work normally). It will run normally

c) The application is installed, but module versions are outdated (module version > db entry version) — run migration.

d) The application is installed, but the database version is higher than the code version. Human action is needed: downgrade DB version, upgrade module version, etc.

1. In production environment: compile “caches”: DI, static content

All of these steps must be done in maintenance mode (downtime) that is measured in minutes and can reach up to 1 hour for multi-server setups, which is a real problem in a world of 100.1% uptime offers!

We were working on moving all those steps to a building stage on our CI/CD server to reduce downtime, however, not everything could be easily moved there. Meanwhile, due to a core bug, we had to run setup:static-content:deploy multiple times with different flags. It could take even 10 minutes, and it was impossible to move it to the build server, because of a dependency on the database. By that time we had no idea what exactly is needed for static-content, but we assumed we can not just have a single DB for all deployments.

And we continued suffering until Magento 2.2.0 when Pipeline Deployment was introduced.

Apart from deployment, or I would rather say, as a part of it, the new possibility of configuration management was introduced.

Magento 2 Pipeline Deployments

The entire pipeline deployment and configuration management is almost all about the 2 files: config.php and env.php within the app/etc/ folder. In this section, I will explain connections, use cases, functionalities and the content of both of these files. These allow, at least partially, to store application configuration in an environment, not in the database, following 12-factor app definition.

Initial or “default” state

By default, config.php consists of an array of installed modules with an enable/disable (0/1) switch.

Env.php consists of external service configurations: database, redis, varnish; internal caching switch: enable/disable and “install” => “date” element; which indicates Magento status: installed/not-installed.

CLI command: app:config:dump

I hope you are familiar with the core_config_data table, storing all the application configurations.

Executing this command, config.php and env.php will grow multiple times, as the entire core_config_data data are exported to these files. These configurations can be easily transferred to another installation and imported using app:config:import. However, this command does not make changes to the database values — all values are overlapping them in runtime instead.

CLI commands: edit config.php and edit.php

To utilize the power of config.php and env.php, a help command was introduced, however, you must keep in mind it has a bug in versions earlier than 2.2.4 and allows to edit env.php only.

config:set — changes your database entry, no matter what type is it

config:set — lock ( — lock-env) — change value in env.php, no matter what pool it belongs to

config:set — lock-config — change value in config.php, no matter what type it is

config:sensitive:set — change value in env.php, works only for “sensitive” paths

Config files downside

Each time, any setting appears in the config.php or env.php, it is automatically locked for edit from the admin panel. That also makes perfect sense for config:set — lock flag. It makes a great solution for us, developers and maintainers, as we can be sure nobody actually can misconfigure the application without notice. But Magento is often the platform of choice because of the tweaking, and reconfiguring, and adopting possibilities using a “friendly” backend panel.

Config hierarchy

Other than different pools and sections, there is also a config hierarchy, that includes environment variables for sensitive fields. Here is a fallback order, previous overrides next.

1. environment variables
2. env.php
3. config.php
4. Database

As you see, there is one more trick available for us: you can override a specific field from config.php or a database for a specific environment, adding it to the env.php! Sometimes it might be useful, but I would suggest to document it well, as otherwise, it might get too complex to determine the real runtime config!

How different config types are defined

This is probably the most interesting and useful part. Following Magento logic and colleague questions, I was intensively testing different behavior of commands and flag, and came to the conclusion — there are actually 3 pools:

1. General or shared — config.php — by default, all values are dumped here.
2. Environment — env.php — values, specific to the environment, each config path can be added to environment pool using this di.xml configuration:

   <type name=”MagentoConfigModelConfigTypePool”>
   
   <arguments>
   
   <argument name=”environment” xsi:type=”array”>
   
   <item name=”scandiweb/testit/environment” xsi:type=”string”>1</item>
   
   </argument>
   
   </type> 

As you see, it is enough to add config path to a specific array.

1. Sensitive — env.php — secrets, passwords, token and other sorts of sensitive data must be added to that pool. Similarly to the environment, it is enough to append the following snippet to your di.xml:

       <type name=”MagentoConfigModelConfigTypePool”>
   
       <arguments>
   
       <argument name=”sensitive” xsi:type=”array”>
   
       <item name=”scandiweb/testit/sensitive” xsi:type=”string”>1</item>
   
       </argument>
   
       </type>

You can discover the details of implementation on your own by taking a close look at the MagentoConfigModelConfigTypePool class. But that is not necessary, as I will try to explain the internal logic in more details, to allow you to utilize the power and avoid downsides of it in your daily routine.

DB-less static-content deployments

After we figured out there are 2 files and 3 pools, we found something more. If you take a look at config.php there are “sections”, which are defined in the main array returned from the file. These are as follows: modules, scopes, themes, i18n, system.

Modules is a default section, starting from the first Magento 2 versions, allowing to enable/disable specific modules.

i18n is a common abbreviation for “internationalization” and is not a point of interest to us given the article’s topic.

System — the most controversial part of pipeline deployment. On one hand — we can finally migrate all the configs easily, without writing tons of migration scripts for configuring the platform properly. On the other hand — it removes the possibility for anyone but developers, DevOps, etc. from adjusting these parameters, because a part of the admin panel, that is related to these parameters, is not editable anymore. Until configuration is removed from the config.php and becomes non-shareable, merchant inconvenience and functionality limitations will never be an easy topic.

Scopes — describes ‘websites’, ‘store groups’ and ‘stores’

Themes — describes installed themes.

Luckily, you only need the scopes and themes sections to perform a DB-less static-content:deploys, sacrificing the convenience of config management, you can still achieve low to zero downtime deployments.

“Zero downtime” can still cause a downtime

Not digging too deep in technical details, there is a case, when you can truly achieve zero downtime deployment without complex service manipulations. Therefore, when you don’t need to import the config (app:config:import) or run migrations (setup:upgrade) — you can achieve zero downtime deployments. In case either of the above-mentioned processes are executed — we can not count on a zero downtime. The only thing we can do it is to reduce this downtime to seconds.

Conclusions

As you see, there is now a convenient way of storing, sharing and managing configurations between environments, but I hope the Magento team will not stop here, as for now, it is far away from being really useful. Cutting out a big advantage of Magento, that allows to configure and alter multiple settings from the simple admin panel is freaking out a lot of our colleagues here, in Scandiweb. Probably, in case of custom functionality, some configs must be migrated from the System section, but at the same time — many of Magento’s core functionalities remain. And merchants want to preserve the possibility to impact them from the admin panel.

We offer a huge range of Magento services at Scandiweb, check them out here. If you have any questions or suggestions, or perhaps some tips you’d like to share — please comment below or get in touch with us at [email protected]!

Share on: