Server Setup

Where compilation went wrong…

by , , revisited on


Although we already have a post about fake optimizers, this is going to be a bit more specific.

I’ve been observing some things online on how the community does in regards to distributing NGINX and its modules.
Some folks want you to compile NGINX and they build mighty scripts for that. And some others build a package for a mighty NGINX binary that includes all the modules in the world.
So I’ve been in a bit of awe at it. Things seem to have gone the wrong way. And we are pulling things in different directions.

This post might have been also called: GetPageSpeed versus… because I will review some Internet folks who think they do well with distributing NGINX through compilation scripts.

&tldr;

All they do is:

  • pollute your system with compilation software
  • compile “best” NGINX with latest OpenSSL and a selection of modules they think are a must
  • run the compilation with -O3 compilation flag and most recent gcc, seemingly declaring that it’s the best thing after the collapse of the Soviet Union because it will result in “faster”, more efficient binaries
  • (maybe the key point of this post, haha) totally neglect stripping binaries from debug symbols, which results in bloatware, slow, memory hungry beast of NGINX (or PHP-FPM, see below); so what efficiency are we talking about?
  • git clone entire (huge) repository (centminmod, see below)

I’ll start my little review from least to worst beasts

nginx-ee

The guys want you to install Ubuntu only, OK.
We fired up Ubuntu 18 and proceeded with install script.
The execution took ~30 mins.

##################################
 Compilation summary 
##################################

 Detected OS : Ubuntu 18.04.2 LTS
 Detected Arch : x86_64

  - Nginx release : 1.17.0
  - OPENSSL : 1.1.1b Stable
  - Dynamic modules NO
  - Pagespeed : NO
  - Naxsi : NO
  - RTMP : NO

       Installing dependencies                [OK]
       Setting Up Nginx configurations        [OK]
       Installing gcc-8                       [OK]
       Downloading additionals modules        [OK]
       Downloading zlib                       [OK]
       Downloading pcre                       [OK]
       Downloading brotli                     [OK]
       Downloading openssl                    [OK]
       Downloading nginx                      [OK]
       Applying nginx patches                 [OK]
       Configuring nginx                      [OK]
       Compiling nginx                        [OK]
       Updating Nginx manual                  [OK]
       Performing final steps                 [OK]
       Checking nginx configuration           [OK]
       Nginx-ee was compiled successfully !

       Installation log : /tmp/nginx-ee.log

Let’s see what we’ve got there.

root@localhost:~# which nginx
/usr/sbin/nginx
root@localhost:~# du -csh /usr/sbin/nginx
16M /usr/sbin/nginx
16M total

What? 16MB only? I think it’s the best thing in the world, super efficient NGINX with latest OpenSSL, it can’t be better, right?

Now look at the packaged install from GetPageSpeed (or stable nginx.org repository) that is under a minute of installation time:

[centos@m2 ~]$ du -csh /sbin/nginx
1.3M    /sbin/nginx
1.3M    total

Why? Sure enough, we have compiled in OpenSSL so the “nginx-ee” binary takes a bunch of stuff in. But that’s not it.
If the “nginx-ee” binary was stripped off debugging symbols, it would be only 5.9M.

And why do we need to compile in OpenSSL now that the recent version is recent enough to support HTTP/2 in Chrome and such?
Absolutely no we don’t need, thank you.

Users of CentOS / RHEL based systems should rely on OpenSSL shipped by distribution unless absolutely required to avail of newer OpenSSL features like TLS 1.3, which (when used without any precautions) bear a security risk of their own.

centminmod

Now this one is special.

The guy behind centminmod promotes his stuff quite more aggressively. His blog is full of info that seems to be relevant to himself only or to a few of his crazy adepts.
Instead of quick concise info on how to get things up and running, you get an ocean of text (SEO matters?).

The install page installation notes actually tell you to disable SELinux:

SELINUX must be disabled.

There, in one phrase you’re offered a downgrade from a perfectly secure system over to a regular, insecure “some kind of Linux”.
What’s your point of using CentOS if you’re disabling one of the primary security features of the system?

Then on his second point, he offers the first security tip, lol:

Tip #1: For best security and performance you would want to use CentOS 7 with KVM based VPS server or dedicated server

I think it should rather state:

Tip #1: don’t follow the first note

Now, for the one line installer:

yum -y update; curl -O https://centminmod.com/installer.sh && chmod 0700 installer.sh && bash installer.sh

I thought I’d try this on VirtualBox machine first. So after minutes of frustration, having it fetch stuff from what seems to be everywhere on the internet, it finished with no nginx anywhere.

Ok, I thought, maybe I didn’t put attention to things, I need to log out and in or something… Done.
Upon login I was welcomed with MotD message saying:

! This server maybe running CSF firewall !

Well, maybe you knew what you installed and could have told me better. Although I prefer using FirewallD 😀

To make a long story short, I guess the VirtualBox “wasn’t powerful enough” for centminmod.

So I fired my Linode manager to get up something for the hungry centminmod….:)

Watching things carefully now, promise.
After running the one line installer thing, I had no output aside from curl download for like 3 minutes…. and then it kicked off.
Lots of dev stuff being installed, repos mixed with compiled stuff, then dev, then epel-release, then something else,
axel (which centminmod man is particularly fond of, apparently for faster(big question) downloads of the stuff to compile), shmuxel 🙂
Then more repositories installed in between packaged installs of something (why not all repos at the beginning?).
Shortly, a scrolling nightmare of compilations and installations. I take back my words that I will watch it carefully, impossible. Too verbose.

It passed ~20 mins before it started compiling something. It’s NGINX… Some time later I saw it “do” PHP and its modules…

Finally, after all, I got:

[root@li358-114 ~]# which nginx
/usr/local/sbin/nginx
du -csh /usr/local/sbin/nginx
3.4M    /usr/local/sbin/nginx
3.4M    total

Winner! The binary is stripped.. But maybe it would have been smaller if it relied on system OpenSSL and not LibreSSL? Scratch that “maybe”, of course, it would have been smaller, ~1MB.

How about php?

[root@li358-114 bin]# du -csh /usr/local/sbin/php-fpm
39M /usr/local/sbin/php-fpm
39M total
[root@li358-114 bin]# strip /usr/local/sbin/php-fpm
[root@li358-114 bin]# du -csh /usr/local/sbin/php-fpm
9.2M    /usr/local/sbin/php-fpm
9.2M    total

Holy moly, that’s a big one. Unstripped php-fpm from centminmod compilation is whopping 39M, which can be reduced to 9M by stripping debug symbols and other unnecessary information from binaries.

To centminmod and the likes: please strip 🙂 And stop polluting servers with bloatware compiled binaries “since 2011″…

As for the other community efforts that are not crazy about compilation and more on the lesser evil, the packaging.
Things aren’t bright there either.

  • There’s “uloyhla” (haha, I can’t get the name right) RHEL repository and it builds on some outdated versions of some NGINX modules. The module versions can’t be identified properly and have a loose dependency on NGINX versions, which makes things far from stable. The repository package claims to be compatible with RHEL, but using releasever yum variable in .repo definition makes it not.
  • The aeris network’s Karl (who banned me from his repository for genuinely addressing people’s problems offering my repository with what people requested) goes even worst with his bloatware nginx. All that when nginx has supported dynamic modules since 2016…

So typically, you get it what you have with compiled / hobbyist package installs.
They would:

  • Neglect details like stripping binaries
  • Have your server turned to a garbage can full with spoilt soup of packages mixed with compiled software. centminmod brought +105 -devel packages from 0 initial.
  • Compile in latest OpenSSL for reasons largely known to them only, aside from “latest OpenSSL is secure”, which is not always the case anyway
  • Just suck big time

There, you got it. I just told you why GetPageSpeed extras repository is the thing you need to use for installing fast, lightweight NGINX – the way you love it and want it to be. And with virtually any module you want

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.