HOWTO Build and Install Mcrouter for RHEL, OL, CENTOS 7.x

UPDATE: Unless you have a very specific environment, Facebook likely now has packages or updated compile instructions that will work better than this: https://github.com/facebook/mcrouter

The instructions below are likely out of date and may no longer work.

This post will show you how to build and install Mcrouter. Once complete, you will have a portable Mcrouter installation that you can apply to multiple machines via RPM.

Building Mcrouter is not easy, but this step-by-step process should get you to a collection of RPMs which make up a functional and upgrade-able Mcrouter environment.

About Mcrouter

Mcrouter is a memcached protocol router for scaling memcached (http://memcached.org/) deployments. It’s a core component of cache infrastructure at Facebook and Instagram where mcrouter handles almost 5 billion requests per second at peak.

Mcrouter is developed and maintained by Facebook.

You can get the source and more information here: https://github.com/facebook/mcrouter

You are going to do a lot of compiling and packaging.  Once you have this perfected, it is not a big stretch to hand it off to Jenkins or some other build tool for automation.

Getting Started

You need to build two environments.

Environment 1: Build Environment for compiling and packaging Mcrouter

• An EL7 Server (RedHat, Oracle Linux, CentOS)
• Yum updates to the latest stable everything (yum -y update, then reboot)
• At LEAST 4GB RAM
• At LEAST 4GB available disk space
• A decent internet connection
• Root access

Environment 2: Operating Environment for actually running Mcrouter

• An EL7 Server (RedHat, Oracle Linux, CentOS)
• Yum updates to the latest stable everything (yum -y update, then reboot)
• Sufficient privilege to install RPMs (sudo, root, etc)

(Env 1) Setting up The Build Environment

NOTE 1: All of the following assumes you are root.  Insert your sudo where required if you aren’t just living and breathing as root.

NOTE 2: You are going to install a lot of crap on this system.  I assume you are OK with that.

Before we begin to build Mcrouter, we need to build a lot of the dependencies and install packages required for both running and building the software.

Install a bunch of packages

NOTE 1: Some of these will not exist. We will compile our own RPMs and grab some from Fedora.  All of it is outlined below.

yum -y install gcc-c++ ruby ruby-devel ragel cyrus-sasl cyrus-sasl-devel cmake bison python-libs python-devel numactl-libs numactl-devel scons snappy snappy-devel gflags gflags-devel glog-devel glog boost-devel automake autoconf libtool libevent libevent-devel openssl-devel libatomic flex rpm-build


Most likely the following packages are missing: ragel, scons, glfags, gflags-devel, glog, glog-devel.  Let’s get building and packaging for those.  If you are missing others, you’ll have to build and package them too.

Create a build location

This is where you need the disk space and where you will download, compile and package everything.

mkdir /build

Install fpm

(Effing package management! https://github.com/jordansissel/fpm)

We are going to use FPM to create RPMs as we go.

gem install fpm

If you see some errors at the end similar to this:
Parsing documentation for fpm-1.5.0
unable to convert "\xF3" from ASCII-8BIT to UTF-8 for lib/fpm/package/pyfpm/__init__.pyc, skipping
unable to convert "\xF3" from ASCII-8BIT to UTF-8 for lib/fpm/package/pyfpm/get_metadata.pyc, skipping

You can ignore them.

Build and package ragel

NOTE: Get the latest version of ragel here: http://www.colm.net/open-source/ragel/

Download

cd /build
curl -O http://www.colm.net/files/ragel/ragel-6.9.tar.gz

Build

cd /build
tar zxf ragel-6.9.tar.gz
cd ragel-6.9
./configure
make
make install

Package

mkdir /build/ragelpkg
make install DESTDIR=/build/ragelpkg
cd /build
fpm --verbose -f -s dir -t rpm -n ragel -v 6.9 -C /build/ragelpkg \
usr/local/bin \
usr/local/share/doc/ragel \
usr/local/share/man/man1

You should have an RPM that will install ragel.

Download and install scons, glfags, gflags-devel, glog, glog-devel

Download

cd /build
curl -O http://tenet.dl.sourceforge.net/project/scons/scons/2.5.0/scons-2.5.0-1.noarch.rpm
curl -O https://kojipkgs.fedoraproject.org//packages/gflags/2.1.1/6.el7/x86_64/gflags-2.1.1-6.el7.x86_64.rpm
curl -O https://kojipkgs.fedoraproject.org//packages/gflags/2.1.1/6.el7/x86_64/gflags-devel-2.1.1-6.el7.x86_64.rpm
curl -O https://kojipkgs.fedoraproject.org//packages/glog/0.3.3/8.el7/x86_64/glog-0.3.3-8.el7.x86_64.rpm
curl -O https://kojipkgs.fedoraproject.org//packages/glog/0.3.3/8.el7/x86_64/glog-devel-0.3.3-8.el7.x86_64.rpm

Install

rpm -ivh ./scons-2.5.0-1.noarch.rpm ./gflags-2.1.1-6.el7.x86_64.rpm ./glog-0.3.3-8.el7.x86_64.rpm ./gflags-devel-2.1.1-6.el7.x86_64.rpm ./glog-devel-0.3.3-8.el7.x86_64.rpm

Build and Package Double-Conversion

NOTE: Double-Conversion is a library from Google which provides binary-decimal and decimal-binary routines for IEEE doubles.  Details and source code here: https://github.com/google/double-conversion

Download

cd /build
git clone https://github.com/google/double-conversion.git

Build

cd /build/double-conversion
make
make test
scons install
chmod 0644 /usr/local/lib/libdouble-conversion*
mkdir /usr/local/include/double-conversion
cp double-conversion/*.h /usr/local/include/double-conversion/
chmod 0755 /usr/local/include/double-conversion/
chmod 0644 /usr/local/include/double-conversion/*

Package

cd /build
mkdir -p /build/dcpkg/usr/local/lib/
mkdir -p /build/dcpkg/usr/local/include/double-conversion/

cp -rp /usr/local/lib/libdouble-conversion* /build/dcpkg/usr/local/lib/
cp -rp /usr/local/include/double-conversion/* /build/dcpkg/usr/local/include/double-conversion/

cd /build/double-conversion
VER=$(git tag|tail -1)
cd /build

fpm --verbose -f -s dir -t rpm -n double-conversion -v $VER -C /build/dcpkg \
usr/local/lib/ \
usr/local/include/double-conversion/

You should have an RPM that will install double-conversion.

Build and Package Folly

Lots of information on Folly: https://github.com/facebook/folly

Folly (acronymed loosely after Facebook Open Source Library) is a library of C++11 components designed with practicality and efficiency in mind. Folly contains a variety of core library components used extensively at Facebook. In particular, it’s often a dependency of Facebook’s other open source C++ efforts and place where those projects can share code.

Download

cd /build
git clone https://github.com/facebook/folly

Build

cd /build/folly/folly
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
VER=$(git tag|tail -1)
autoreconf -ivf
LDFLAGS=-L/usr/local/lib CPPFLAGS=-I/usr/local/include ./configure
make -j2
make install

Package

cd /build/folly/folly
mkdir /build/follypkg
make install DESTDIR=/build/follypkg
cd /build
fpm --verbose -f -s dir -t rpm -n folly -v $VER -C /build/follypkg \
usr/local/include \
usr/local/lib

You should have an RPM that will install Folly.

Upgrade your Automake

NOTE: This is only required if you have a version of Automake lower than 1.15

curl -O https://kojipkgs.fedoraproject.org/packages/automake/1.15/1.fc22/noarch/automake-1.15-1.fc22.noarch.rpm
rpm -Uvh automake-1.15-1.fc22.noarch.rpm

Download and Install jemalloc

curl -O https://kojipkgs.fedoraproject.org/packages/jemalloc/3.6.0/1.el7/x86_64/jemalloc-3.6.0-1.el7.x86_64.rpm
curl -O https://kojipkgs.fedoraproject.org//packages/jemalloc/3.6.0/1.el7/x86_64/jemalloc-devel-3.6.0-1.el7.x86_64.rpm

rpm -ivh jemalloc-devel-3.6.0-1.el7.x86_64.rpm jemalloc-3.6.0-1.el7.x86_64.rpm

Build and Package Wangle

More on Wangle: https://github.com/facebook/wangle

Wangle provides a full featured, high performance C++ futures implementation.

Download

cd /build
git clone https://github.com/facebook/wangle

Build

cd /build/wangle/wangle
VER=$(git tag|tail -1)
sed -i -r -e "s@\ \ -lpthread@\ \ -lgflags\n\ \ -lgflags@" CMakeLists.txt
cmake .
make
make test
make install

Package

cd /build/wangle/wangle
mkdir /build/wanglepkg
make install DESTDIR=/build/wanglepkg
cd /build
fpm --verbose -f -s dir -t rpm -n wangle -v $VER -C /build/wanglepkg \
usr/local/include \
usr/local/lib

You should have an RPM that will install Wangle.

Build and Package FBThrift

Read lots about fbthrift here: https://github.com/facebook/fbthrift

FBThrift: Facebook’s branch of apache thrift

Download

cd /build
git clone https://github.com/facebook/fbthrift

Build

cd /build/fbthrift/thrift
VER=$(git tag|tail -1)
autoreconf -ifv
LDFLAGS=-L/usr/local/lib CPPFLAGS=-I/usr/local/include ./configure
make -j2
make install


Package

cd /build/fbthrift/thrift
mkdir /build/fbthriftpkg
make install DESTDIR=/build/fbthriftpkg
cd /build
fpm --verbose -f -s dir -t rpm -n fbthrift -v $VER -C /build/fbthriftpkg \
usr/local/bin \
usr/local/include \
usr/local/lib

You should have an RPM that will install Thrift.

FINALLY!!! Build and Package McRouter!

Download

cd /build
git clone https://github.com/facebook/mcrouter

Build

cd /build/mcrouter/mcrouter
VER=$(git tag|tail -1)
autoreconf -ifv
export THRIFT2_COMP_DIR=/usr/lib/python2.7/site-packages/thrift_py-0.9.0-py2.7.egg/thrift_compiler
LDFLAGS=-L/usr/local/lib CPPFLAGS=-I/usr/local/include ./configure --prefix=/usr/local/mcrouter
make -j2
make install

Package

cd /build/mcrouter/mcrouter
mkdir /build/mcrouterpkg
make install DESTDIR=/build/mcrouterpkg
cd /build
fpm --verbose -f -s dir -t rpm -n mcrouter -v $VER -C /build/mcrouterpkg \
usr/local/mcrouter

You should have an RPM that will install mcrouter

Defining Your Portable Installation

The following RPMs from this tutorial are required to operate a mcrouter environment. Set them aside.

As of this writing, my specific RPMs and versions are:

• gflags-2.1.1-6.el7.x86_64.rpm
• glog-0.3.3-8.el7.x86_64.rpm
• jemalloc-3.6.0-1.el7.x86_64.rpm
• ragel-6.9-1.x86_64.rpm
• double-conversion-v2.0.1-1.x86_64.rpm
• folly-v0.57.0-1.x86_64.rpm
• wangle-v0.9.0-1.x86_64.rpm
• fbthrift-v0.31.0-1.x86_64.rpm
• mcrouter-v0.9.0-1.x86_64.rpm

(Env 2) Setting up The Operating Environment

In order to run mcrouter on another node, you will need to install a the following system packages and packages you downloaded and created:

Install Packages

System Packages

yum -y install cyrus-sasl python-libs numactl-libs snappy boost libtool libevent openssl libatomic

Downloaded Packages

rpm -ivh gflags-2.1.1-6.el7.x86_64.rpm glog-0.3.3-8.el7.x86_64.rpm jemalloc-3.6.0-1.el7.x86_64.rpm

Built Packages

rpm -ivh ragel-6.9-1.x86_64.rpm double-conversion-v2.0.1-1.x86_64.rpm folly-v0.57.0-1.x86_64.rpm wangle-v0.9.0-1.x86_64.rpm fbthrift-v0.31.0-1.x86_64.rpm mcrouter-v0.9.0-1.x86_64.rpm

Test Mcrouter

/usr/local/mcrouter/bin/mcrouter --version

Done!

Did you find this post useful or have questions or comments? Please let me know!