Magento

Magento 2 deployment with CircleCI

by , , revisited on


We have by far the largest RPM repository with dynamic stable NGINX modules and VMODs for Varnish 4.1 and 6.0 LTS. If you want to install NGINX, Varnish and lots of useful modules for them, this is your one stop repository to get all performance related software.
You have to maintain an active subscription in order to be able to use the repository!
This post is well within its draft state. So pardon for any typos.

A recommended read before implementing deployment with CircleCI is Magento 2.2 Deployment Improvements.

Need for pipeline deployment

Magento 2 builds on the two main modes: production and developer. The production website has to run with live settings and with “compiled” assets.
The assets compilation is a very slow step in bringing Magento 2 to production state.
When you install a new plugin, you have to run compilation of static assets. This brings your store into maintenance mode for a few minutes, at best, and the site is essentially down. What can we do about this?

Answer: pipeline deployment.

Let’s review best practices approach for Magento 2 development and deployment.

;tldr

  • Run bin/magento app:config:dump and ensure it’s tracked in git
  • Create deploy.sh in your Magento project directory (listing below)
  • Create .circleci/config.yml in your Magento project directory (listing below), adjust as needed with your Magento server’s username and hostname, etc.
  • Create .shadow empty directory in your Magento project directory and gitignore its contents (ensure presence via .gitkeep)
  • Add your git repository to CircleCI, and specify SSH keys in project settings
  • Commit, push. Enjoy smaller downtimes when deploying Magento to production.

Store Magento 2 in git

Code should be stored in BitBucket or another centralised git repository (paid GitHub for example). There are going to be 2 branches:

  • master (reflects live code)
  • staging (reflects staging.example.com)

We assume that the dev. instance is not needed, since developers have a development environment of their own.

Development workflow

  • A developer is member of the respective online git repository (BitBucket)
  • They would commonly check out the staging branch and work with it
  • Pushes to staging remote branch will trigger build of static assets and deploying them to staging.example.com. The site would be used for checking development progress as well as some QA before putting changes live

Deployment to Live

Once the website owner is happy with the state of things at staging, they (or developer) would create a pull request in BitBucket: from staging to master branch. Once the pull request is merged, we’ve essentially pushed to the master branch.

Alternatively, developers would be working on their feature specific branch, and merge it to both staging (for QA) and master (for making the feature live).

Either way, git push (merge) to master branch should result only in one thing – putting those changes live. CircleCI is something we’re going to use for exactly that – deployment.

CircleCI is an interesting tool that allows you to run arbitrary tasks involving your git codebase. At the same time those tasks will use CircleCi’s servers infrastructure to be run. So your web server does not even incur any performance penalty for running those tasks. Moreover it’s free for private BitBucket repositories. Sounds exciting?

Sure it does. Because by leveraging CircleCI, we are not only able to deploy something, we can actually build that something to reach deployable state.

1. Build Magento static assets using CircleCI

Once you conncect CircleCI to your Github / Bitbucket repository, it really acts as a sort of hook/trigger program for your repository. What we want it to do, on a high level:

  • we want CircleCI to take our Magento 2 codebase, and build static assets
  • we want CircleCI to take the built assets, then transfer them over to an arbitrary directory on the live server
  • we want CircleCI to swap the live site’s compiled assets with the newly built ones and run a few other commands to make things final

We’re going to build a CircleCI workflow which includes 2 main tasks: the “build” and the “deploy”. We can have multiple copies of “deploy” steps in order to deploy things over to different places: dev, staging, live, etc.

The build steps will:

  • spin up a Linux system (Docker container, really) with PHP 7.1
  • checkout source code for the master branch of your git repository
  • composer install to install whatever packages which have to be in the vendor (remember, we excluded it from git)
  • php bin/magento setup:static-content:deploy to compile the static assets
  • php bin/magento setup:di:compile to compile Magento 2 stuff
  • save those as “artefacts” (for “deploy” step) upon success

Note on compiling static assets

This step is ridiculously slow at all times, especially if you start with empty pub/static.

The setup:static-content:deploy, when run without parameters, is “flawed” in a way that it always compiles assets for all languages. Internally deploy:mode:set production uses setup:static-content:deploy (see here and compiles for only the necessary languages.

But we cannot use deploy:mode:set production since it also runs a few steps which require Magento 2 to be setup, whereas we want to compile static assets using nothing but files from git.

So we have to explicitly specify the website locales we want to compile assets for, in order to reduce time spent for the task. We can also specify themes we want assets to be compiled for as such:

php bin/magento setup:static-content:deploy --area=frontend --theme=Gw/frontend en_US

A quick way to check for active themes in a multi-store Magento 2 is the following query:

SELECT code FROM core_config_data AS ccd LEFT JOIN theme AS t ON t.theme_id = ccd.value WHERE path = 'design/theme/theme_id';

Adding --jobs=X will allow to use multiple CPU cores, e.g.:

php bin/magento setup:static-content:deploy --area=frontend --theme=Gw/frontend --jobs=32 en_US 

.circleci/config.yml

version: 2
jobs: # a collection of steps
  build:
    docker: # run the steps with Docker
      - image: getpagespeed/m2builder:latest
    working_directory: /sources # directory where steps will run
    steps: # a set of executable commands
      - checkout
      - run:
          name: "Install Mage Composer keys"
          command: composer global config http-basic.repo.magento.com $MAGENTO_KEY_PUBLIC $MAGENTO_KEY_PRIVATE
      - run:
          name: "Install Plumrocket composer keys"
          command: composer global config http-basic.store.plumrocket.com $PLUMROCKET_KEY_PUBLIC $PLUMROCKET_KEY_PRIVATE
      - run:
          name: "Install composer cache"
          command: ./extract-composer-cache.sh
      - run:
          name: "Install Magento plugin dependencies (vendor)"
          command: composer install
      - run:
          name: "Compile PHP stuff (generated)"
          command: php -d memory_limit=-1 bin/magento setup:di:compile
      - run:
          name: "Compile static (pub/static)"
          command: php -d memory_limit=-1 bin/magento setup:static-content:deploy -f en_US
      - persist_to_workspace:
          root: /sources
          paths:
            - "vendor" 
            - "generated"
            - "pub/static"
            - "var/view_preprocessed"
  deploy:
    docker: # run the steps with Docker
      - image: getpagespeed/m2builder:latest
    working_directory: /sources
    steps:
      - attach_workspace:
          at: /sources
      - add_ssh_keys:
          fingerprints:
            - "e2:65:63:c0:04:b6:82:f0:23:f3:d3:8e:9d:06:3f:fd"
      - run:
          name: "Copy generated transient files to shadow directory"
          command: rsync -e "ssh -p 22 -o StrictHostKeyChecking=no" -avz --delete --exclude=/.gitkeep ./ live-user@m2.example.com:/srv/www/example.com/.shadow/
    - run:
        name: "Deploy run: maintenance mode + git pull + overwrite compiled files + upgrade data + cache clear + maintenance off"
        command: ssh -v -o StrictHostKeyChecking=no staging@m2.example.com "/srv/www/example.com/deploy.sh"
  deploy-staging:
    docker: # run the steps with Docker
    - image: getpagespeed/m2builder:latest
    working_directory: /sources
    steps:
    - attach_workspace:
        at: /sources
    - add_ssh_keys:
        fingerprints:
        - "7b:72:9c:aa:85:52:3f:77:55:24:5b:db:c9:98:50:df"
    - run:
        name: "Copy generated transient files to shadow sub directory"
        command: rsync -e "ssh -p 22 -o StrictHostKeyChecking=no" -avz --delete --exclude=/.gitkeep ./ staging@m2.example.com:/srv/www/staging.example.com/.shadow/
    - run:
        name: "Deploy run: maintenance mode + git pull + overwrite compiled files + upgrade data + cache clear + maintenance off"
        command: ssh -v -o StrictHostKeyChecking=no staging@m2.example.com "/srv/www/staging.example.com/deploy.sh"
workflows:
  version: 2
  build-deploy:
    jobs:
      - build
      - deploy:
          requires:
            - build
          filters:   
            branches:
              only: master
      - deploy-staging:
          requires:
          - build
          filters:
            branches:
              only: staging

The “Install composer cache” is optional, but it’s meant to speed up subsequent composer install. This extract-composer-cache.sh script will download composer cache zip and might look like the following:

#!/bin/bash

# yum -y install unzip
# Hosting with admin_ prefix ensures that Varnish will deliver the file even if Magento is "down"
curl https://www.example.com/admin_composer_cache.zip --output /tmp/composer-cache.zip
mkdir -p ~/.composer/cache
rm -rf ~/.composer/cache/*
unzip /tmp/composer-cache.zip -d ~/.composer/cache

To generate composer cache, simply zip your ~/.composer/cache on the live system and put it so it’s web accessible at https://www.example.com/admin_composer_cache.zip.

Reiterating again on the build steps of our CircleCI program, what they do is:

  • Installing composer keys into our CircleCI build machine. This will ensure we can composer install the many Magento packages into an empty vendor/ without any prompt for authentication. We use environment / organisation variables to define the keys
  • We also install composer keys specific to other composer package repositories
  • To speed up composer install, you may prepare a snapshot of composer cache from your live system, and install it into CircleCI docker using special script (optional)
  • The composer install step actually downloads Magento packages to vendor and gives us complete source files of our project
  • Next, we compile static assets using well known Magento CLI commands
  • Save generated files as artefacts for the deploy step.

2. The deployment

Now as for deploy tasks, those are which will be run exclusively upon push to either staging or master branches after the build steps succeed:

  • push the “artefacts” over to respective website’s .shadow subdirectory (which is .gitignore-d but ensured of its presence via .gitkeep file
  • our target deployment server location, e.g. /srv/www/example.com/.shadow/ will receive all our compiled files using rsync
  • launch a deploy.sh script as on the server in order to apply the generated/uploaded assets

The deploy.sh script will:

  • put live to maintenance: php bin/magento maintenance:enable
  • git pull
  • overwrite/rsync pub/static, generated and vendor dirs with the one from the .shadow subdirectory;
  • bin/magento setup:upgrade --keep-generated
  • flush caches;
  • maintenance off.

deploy.sh

#!/bin/bash
# called by CircleCI or other CI after generating compiled stuff and putting those into .shadow subdirectory

cd "$(dirname "$0")"

php bin/magento maintenance:enable
git pull
#overwrite pub/static, generated and vendor dirs with the one from the build;
rsync -avz --delete .shadow/pub/static/ ./pub/static/
rsync -avz --delete .shadow/vendor/ ./vendor/
rsync -avz --delete .shadow/generated/ ./generated/
rsync -avz --delete .shadow/var/view_preprocessed/ ./var/view_preprocessed/
php bin/magento setup:upgrade --keep-generated
cachetool opcache:reset
n98-magerun2 cache:flush
cachetool opcache:reset
php bin/magento maintenance:disable

Naturally, we have to ensure SSH keys in CircleCI allow connecting to our staging/live server and run the necessary commands.

Benefits:

  • Faster updates – the “deploy” step will take less than a minute. This is the time your M2 is essentially off during deployment.
  • The CPU-heavy assets compilation doesn’t even happen in our servers – it happens in CircleCI, using their power and not affecting live server load at all.
  • You know that the assets compile without errors before pushing new plugins on the live server. No more failed compilations (and complications) at live.
  1. Mark

    Thanks for posting this. Do you have an example CircleCI config file with the steps in that you described?

    Reply
  2. Rich

    Hi Danila

    Thank you for the post, I have an issue when using the example config file. i have adjusted it to what i need however i get the below error when i build via CircleCI.

    “/bin/bash: ./extract-composer-cache.sh: No such file or directory
    Exited with code 1”

    im not really sure what this does, if i remove the run command i get the below error.

    “Composer could not find a composer.json file in /tmp/project
    To initialize a project, please create a composer.json file as described in the https://getcomposer.org/ “Getting Started” section
    Exited with code 1″

    Any help would be greatly appreciated.

    Reply
    • Danila Vershinin

      Added notes about extract-composer-cache.sh. It’s an optional step / script.

      If you have “Composer could not find a composer.json file in /tmp/project”, then it is what is – no composer.json file found.

      Likely you’re running composer install without - checkout step, or don’t even have Magento installed via composer in the first place.

      Reply
  3. Adil Kadiyawala

    Hi Danila,
    I am already working on one Magento2 project. So I have directly create a folder of .circleci and put config.yml in to that folder. So is it correct way?

    Reply
    • Danila Vershinin

      Sure, that’s the way to get started with it. You will then need to specify MAGENTO_KEY_PUBLIC and MAGENTO_KEY_PRIVATE environment variables in your CircleCi settings. And remove step “Install Plumrocket composer keys” in case you’re not using any of those plugins.

      Reply
      • Adil

        Hi Danila,
        Thanks for your reply,
        I have one confusion :
        Is there need any database configuration in config.yml? because when I enter this command “circleci config process .circleci/config.yml” then config files shows errors

        I have enter Magento_key_public and private like this :

        From
        command: composer global config http-basic.repo.magento.com $MAGENTO_KEY_PUBLIC $MAGENTO_KEY_PRIVATE

        To
        command: composer global config http-basic.repo.magento.com 768ddadea087d8b434cafd71065cdf1b 350922b07dc71b5bf0ae0fed167e7bb4

        Reply
        • Danila Vershinin

          I’m not really sure what circleci config process but I assume it does parsing .yml? So errors are likely with formatting. Hardcoding keys is ok as long as the git repository is private.

          Reply
          • Adil

            Hi Danila,
            When I push my changes on bitbucket and job create in circleci.
            Build process has been done successfully but error throw in deploy:

            rsync -e "ssh -p 22 -o StrictHostKeyChecking=no" -avz --delete --exclude=/.gitkeep ./ adil.kadiyawala@ia.ooo:/srv/app/.shadow/
            

            adil.kadiyawala@ia.ooo : this email address I used in circleci so is it correct?

            rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(638) [sender=3.1.1]
            Too long with no output (exceeded 10m0s).
            Can you please help me to solved it?

          • Danila Vershinin

            The user@m2.example.com in my example is not an email address. It’s what rsync uses: remote SSH username @ remote SSH hostname.

            So you should likely put it as `@‘, if you have not set up FQDN name for your server.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.