You have to maintain an active subscription in order to be able to use the repository!
A recommended read before implementing deployment with CircleCI is Magento 2.2 Deployment Improvements.
Need for pipeline deployment
Magento 2 builds on the two main modes: production and developer. The production website has to run with live settings and with “compiled” assets.
The assets compilation is a very slow step in bringing Magento 2 to production state.
When you install a new plugin, you have to run compilation of static assets. This brings your store into maintenance mode for a few minutes, at best, and the site is essentially down. What can we do about this?
Answer: pipeline deployment.
Let’s review best practices approach for Magento 2 development and deployment.
bin/magento app:config:dumpand ensure it’s tracked in git
deploy.shin your Magento project directory (listing below)
.circleci/config.ymlin your Magento project directory (listing below), adjust as needed with your Magento server’s username and hostname, etc.
.shadowempty directory in your Magento project directory and gitignore its contents (ensure presence via
- Add your git repository to CircleCI, and specify SSH keys in project settings
- Commit, push. Enjoy smaller downtimes when deploying Magento to production.
Store Magento 2 in git
Code should be stored in BitBucket or another centralised git repository (paid GitHub for example). There are going to be 2 branches:
- master (reflects live code)
- staging (reflects staging.example.com)
We assume that the dev. instance is not needed, since developers have a development environment of their own.
- A developer is member of the respective online git repository (BitBucket)
- They would commonly check out the
stagingbranch and work with it
- Pushes to
stagingremote branch will trigger build of static assets and deploying them to
staging.example.com. The site would be used for checking development progress as well as some QA before putting changes live
Deployment to Live
Once the website owner is happy with the state of things at staging, they (or developer) would create a pull request in BitBucket: from staging to master branch. Once the pull request is merged, we’ve essentially pushed to the
Alternatively, developers would be working on their feature specific branch, and merge it to both staging (for QA) and master (for making the feature live).
Either way, git push (merge) to master branch should result only in one thing – putting those changes live. CircleCI is something we’re going to use for exactly that – deployment.
CircleCI is an interesting tool that allows you to run arbitrary tasks involving your git codebase. At the same time those tasks will use CircleCi’s servers infrastructure to be run. So your web server does not even incur any performance penalty for running those tasks. Moreover it’s free for private BitBucket repositories. Sounds exciting?
Sure it does. Because by leveraging CircleCI, we are not only able to deploy something, we can actually build that something to reach deployable state.
1. Build Magento static assets using CircleCI
Once you conncect CircleCI to your Github / Bitbucket repository, it really acts as a sort of hook/trigger program for your repository. What we want it to do, on a high level:
- we want CircleCI to take our Magento 2 codebase, and build static assets
- we want CircleCI to take the built assets, then transfer them over to an arbitrary directory on the live server
- we want CircleCI to swap the live site’s compiled assets with the newly built ones and run a few other commands to make things final
We’re going to build a CircleCI workflow which includes 2 main tasks: the “build” and the “deploy”. We can have multiple copies of “deploy” steps in order to deploy things over to different places: dev, staging, live, etc.
build steps will:
- spin up a Linux system (Docker container, really) with PHP 7.1
- checkout source code for the
masterbranch of your git repository
composer installto install whatever packages which have to be in the
vendor(remember, we excluded it from git)
php bin/magento setup:static-content:deployto compile the static assets
php bin/magento setup:di:compileto compile Magento 2 stuff
- save those as “artefacts” (for “deploy” step) upon success
Note on compiling static assets
This step is ridiculously slow at all times, especially if you start with empty
setup:static-content:deploy, when run without parameters, is “flawed” in a way that it always compiles assets for all languages. Internally
deploy:mode:set production uses
setup:static-content:deploy (see here and compiles for only the necessary languages.
But we cannot use
deploy:mode:set production since it also runs a few steps which require Magento 2 to be setup, whereas we want to compile static assets using nothing but files from git.
So we have to explicitly specify the website locales we want to compile assets for, in order to reduce time spent for the task. We can also specify themes we want assets to be compiled for as such:
php bin/magento setup:static-content:deploy --area=frontend --theme=Gw/frontend en_US
A quick way to check for active themes in a multi-store Magento 2 is the following query:
SELECT code FROM core_config_data AS ccd LEFT JOIN theme AS t ON t.theme_id = ccd.value WHERE path = 'design/theme/theme_id';
--jobs=X will allow to use multiple CPU cores, e.g.:
php bin/magento setup:static-content:deploy --area=frontend --theme=Gw/frontend --jobs=32 en_US
version: 2 jobs: # a collection of steps build: docker: # run the steps with Docker - image: getpagespeed/m2builder:latest working_directory: /sources # directory where steps will run steps: # a set of executable commands - checkout - run: name: "Install Mage Composer keys" command: composer global config http-basic.repo.magento.com $MAGENTO_KEY_PUBLIC $MAGENTO_KEY_PRIVATE - run: name: "Install Plumrocket composer keys" command: composer global config http-basic.store.plumrocket.com $PLUMROCKET_KEY_PUBLIC $PLUMROCKET_KEY_PRIVATE - run: name: "Install composer cache" command: ./extract-composer-cache.sh - run: name: "Install Magento plugin dependencies (vendor)" command: composer install - run: name: "Compile PHP stuff (generated)" command: php -d memory_limit=-1 bin/magento setup:di:compile - run: name: "Compile static (pub/static)" command: php -d memory_limit=-1 bin/magento setup:static-content:deploy -f en_US - persist_to_workspace: root: /sources paths: - "vendor" - "generated" - "pub/static" - "var/view_preprocessed" deploy: docker: # run the steps with Docker - image: getpagespeed/m2builder:latest working_directory: /sources steps: - attach_workspace: at: /sources - add_ssh_keys: fingerprints: - "e2:65:63:c0:04:b6:82:f0:23:f3:d3:8e:9d:06:3f:fd" - run: name: "Copy generated transient files to shadow directory" command: rsync -e "ssh -p 22 -o StrictHostKeyChecking=no" -avz --delete --exclude=/.gitkeep ./ firstname.lastname@example.org:/srv/www/example.com/.shadow/ - run: name: "Deploy run: maintenance mode + git pull + overwrite compiled files + upgrade data + cache clear + maintenance off" command: ssh -v -o StrictHostKeyChecking=no email@example.com "/srv/www/example.com/deploy.sh" deploy-staging: docker: # run the steps with Docker - image: getpagespeed/m2builder:latest working_directory: /sources steps: - attach_workspace: at: /sources - add_ssh_keys: fingerprints: - "7b:72:9c:aa:85:52:3f:77:55:24:5b:db:c9:98:50:df" - run: name: "Copy generated transient files to shadow sub directory" command: rsync -e "ssh -p 22 -o StrictHostKeyChecking=no" -avz --delete --exclude=/.gitkeep ./ firstname.lastname@example.org:/srv/www/staging.example.com/.shadow/ - run: name: "Deploy run: maintenance mode + git pull + overwrite compiled files + upgrade data + cache clear + maintenance off" command: ssh -v -o StrictHostKeyChecking=no email@example.com "/srv/www/staging.example.com/deploy.sh" workflows: version: 2 build-deploy: jobs: - build - deploy: requires: - build filters: branches: only: master - deploy-staging: requires: - build filters: branches: only: staging
The “Install composer cache” is optional, but it’s meant to speed up subsequent
composer install. This
extract-composer-cache.sh script will download composer cache zip and might look like the following:
#!/bin/bash # yum -y install unzip # Hosting with admin_ prefix ensures that Varnish will deliver the file even if Magento is "down" curl https://www.example.com/admin_composer_cache.zip --output /tmp/composer-cache.zip mkdir -p ~/.composer/cache rm -rf ~/.composer/cache/* unzip /tmp/composer-cache.zip -d ~/.composer/cache
To generate composer cache, simply zip your
~/.composer/cache on the live system and put it so it’s web accessible at
Reiterating again on the
build steps of our CircleCI program, what they do is:
- Installing composer keys into our CircleCI build machine. This will ensure we can
composer installthe many Magento packages into an empty
vendor/without any prompt for authentication. We use environment / organisation variables to define the keys
- We also install composer keys specific to other composer package repositories
- To speed up
composer install, you may prepare a snapshot of composer cache from your live system, and install it into CircleCI docker using special script (optional)
composer installstep actually downloads Magento packages to
vendorand gives us complete source files of our project
- Next, we compile static assets using well known Magento CLI commands
- Save generated files as artefacts for the deploy step.
2. The deployment
Now as for
deploy tasks, those are which will be run exclusively upon push to either
master branches after the
build steps succeed:
- push the “artefacts” over to respective website’s
.shadowsubdirectory (which is
.gitignore-d but ensured of its presence via
- our target deployment server location, e.g. /srv/www/example.com/.shadow/ will receive all our compiled files using
- launch a
on the server in order to apply the generated/uploaded assets
deploy.sh script will:
- put live to maintenance:
php bin/magento maintenance:enable
vendordirs with the one from the
bin/magento setup:upgrade --keep-generated
- flush caches;
- maintenance off.
#!/bin/bash # called by CircleCI or other CI after generating compiled stuff and putting those into .shadow subdirectory cd "$(dirname "$0")" php bin/magento maintenance:enable git pull #overwrite pub/static, generated and vendor dirs with the one from the build; rsync -avz --delete .shadow/pub/static/ ./pub/static/ rsync -avz --delete .shadow/vendor/ ./vendor/ rsync -avz --delete .shadow/generated/ ./generated/ rsync -avz --delete .shadow/var/view_preprocessed/ ./var/view_preprocessed/ php bin/magento setup:upgrade --keep-generated cachetool opcache:reset n98-magerun2 cache:flush cachetool opcache:reset php bin/magento maintenance:disable
Naturally, we have to ensure SSH keys in CircleCI allow connecting to our staging/live server and run the necessary commands.
- Faster updates – the “deploy” step will take less than a minute. This is the time your M2 is essentially off during deployment.
- The CPU-heavy assets compilation doesn’t even happen in our servers – it happens in CircleCI, using their power and not affecting live server load at all.
- You know that the assets compile without errors before pushing new plugins on the live server. No more failed compilations (and complications) at live.