This morning project deployment to live went completely wrong and resulted with non-working site.
Requests to instance resulted with blank screen, although BE was O.K. Support dev tried to comment out his changes, but it didn’t help. The failure cause was hard to detect because log files had no useful information — “/var/log/httpd” folder contained only HTTP server errors, but “alpha/logs/” folder contained only MySQL errors. Basically, the project had NO log file for php errors. Or at least we were not aware of any. I had to add custom error handler to get the info about error and pass it to the dev.
Some technical details:
The project uses Smarty templating engine. It’s like Symfony’s Twig or Laravel’s Blade, but older. Working principle is quite simple: you write html code with some custom tags and symbols (instead of pure php tags and code) which then are compiled to html under the hood. Advantage — you don’t need to use complex php statements. Disadvantage — you have to learn new statements from this engine.
After adding new Google Tag Manager code and removing old tracking code, template files are recompiled by Smarty. This happens automatically and doesn’t require any action from developer. This time, new JS code contained some Smarty-specific symbol sequences and templating engine tried to compile it to php template. Of course it failed with fatal errors, causing the site malfunction. And since this code was added to every site page, the whole site was inaccessible.
After brief investigation it became obvious that it was our deployment’s fault. But it remained unclear what exactly is wrong with the modified files. I added custom error handler to collect error messages, info and source.
This custom-made error handler helped us to find the source of the issue and release the fix.
Technical details ended.
Bottom line, we all can learn a few lessons from this:
- git is a crucial tool in our work. It allows you to revert your changes instantly and completely. This is another proof that projects w/o git are hard to deploy and manage. Long story short — git or GTFO from Support.
- clearing cache may solve your problems, but don’t count on it all the time. If you have flushed the cache once, don’t expect that repeating the procedure 5 times you will get different result on the 6th time.
- sometimes, it’s just not enough to comment our your code. Smarty ignored html comments, the changes had to be reverted completely. This is where GIT is your best friend — to revert any changes to desired state.
- last but not least (I’d say the most important!), carefully and thoroughly test your changes on local setup, even on the most simple and straightforward tasks. There were (and will be) so many cases when overconfidence ruined even experienced senior devs.
Don’t let failure break you. Keep your head up and remember that everybody fucks up — one is lifeless that is faultless.
Author: Alex Rehov, Senior Developer in SW.
Check out more of our [Magento development services and support] (https://scandiweb.com/services/technology) here!