The instructions below are likely out of date and may no longer work.

This post will show you how to build and install Mcrouter. Once complete, you will have a portable Mcrouter installation that you can apply to multiple machines via RPM.

Building Mcrouter is not easy, but this step-by-step process should get you to a collection of RPMs which make up a functional and upgrade-able Mcrouter environment.

About Mcrouter

Mcrouter is a memcached protocol router for scaling memcached (http://memcached.org/) deployments. It’s a core component of cache infrastructure at Facebook and Instagram where mcrouter handles almost 5 billion requests per second at peak.

Mcrouter is developed and maintained by Facebook.

You can get the source and more information here: https://github.com/facebook/mcrouter

You are going to do a lot of compiling and packaging.  Once you have this perfected, it is not a big stretch to hand it off to Jenkins or some other build tool for automation.

Getting Started

You need to build two environments.

Environment 1: Build Environment for compiling and packaging Mcrouter

• An EL7 Server (RedHat, Oracle Linux, CentOS)
• Yum updates to the latest stable everything (yum -y update, then reboot)
• At LEAST 4GB RAM
• At LEAST 4GB available disk space
• A decent internet connection
• Root access

Environment 2: Operating Environment for actually running Mcrouter

• An EL7 Server (RedHat, Oracle Linux, CentOS)
• Yum updates to the latest stable everything (yum -y update, then reboot)
• Sufficient privilege to install RPMs (sudo, root, etc)

(Env 1) Setting up The Build Environment

NOTE 1: All of the following assumes you are root.  Insert your sudo where required if you aren’t just living and breathing as root.

NOTE 2: You are going to install a lot of crap on this system.  I assume you are OK with that.

Before we begin to build Mcrouter, we need to build a lot of the dependencies and install packages required for both running and building the software.

Install a bunch of packages

NOTE 1: Some of these will not exist. We will compile our own RPMs and grab some from Fedora.  All of it is outlined below.

yum -y install gcc-c++ ruby ruby-devel ragel cyrus-sasl cyrus-sasl-devel cmake bison python-libs python-devel numactl-libs numactl-devel scons snappy snappy-devel gflags gflags-devel glog-devel glog boost-devel automake autoconf libtool libevent libevent-devel openssl-devel libatomic flex rpm-build


Most likely the following packages are missing: ragel, scons, glfags, gflags-devel, glog, glog-devel.  Let’s get building and packaging for those.  If you are missing others, you’ll have to build and package them too.

Create a build location

This is where you need the disk space and where you will download, compile and package everything.

mkdir /build

Install fpm

(Effing package management! https://github.com/jordansissel/fpm)

We are going to use FPM to create RPMs as we go.

gem install fpm

If you see some errors at the end similar to this:
Parsing documentation for fpm-1.5.0
unable to convert "\xF3" from ASCII-8BIT to UTF-8 for lib/fpm/package/pyfpm/__init__.pyc, skipping
unable to convert "\xF3" from ASCII-8BIT to UTF-8 for lib/fpm/package/pyfpm/get_metadata.pyc, skipping

You can ignore them.

Build and package ragel

NOTE: Get the latest version of ragel here: http://www.colm.net/open-source/ragel/

Download

cd /build
curl -O http://www.colm.net/files/ragel/ragel-6.9.tar.gz

Build

cd /build
tar zxf ragel-6.9.tar.gz
cd ragel-6.9
./configure
make
make install

Package

mkdir /build/ragelpkg
make install DESTDIR=/build/ragelpkg
cd /build
fpm --verbose -f -s dir -t rpm -n ragel -v 6.9 -C /build/ragelpkg \
usr/local/bin \
usr/local/share/doc/ragel \
usr/local/share/man/man1

You should have an RPM that will install ragel.

Download and install scons, glfags, gflags-devel, glog, glog-devel

Download

cd /build
curl -O http://tenet.dl.sourceforge.net/project/scons/scons/2.5.0/scons-2.5.0-1.noarch.rpm
curl -O https://kojipkgs.fedoraproject.org//packages/gflags/2.1.1/6.el7/x86_64/gflags-2.1.1-6.el7.x86_64.rpm
curl -O https://kojipkgs.fedoraproject.org//packages/gflags/2.1.1/6.el7/x86_64/gflags-devel-2.1.1-6.el7.x86_64.rpm
curl -O https://kojipkgs.fedoraproject.org//packages/glog/0.3.3/8.el7/x86_64/glog-0.3.3-8.el7.x86_64.rpm
curl -O https://kojipkgs.fedoraproject.org//packages/glog/0.3.3/8.el7/x86_64/glog-devel-0.3.3-8.el7.x86_64.rpm

Install

rpm -ivh ./scons-2.5.0-1.noarch.rpm ./gflags-2.1.1-6.el7.x86_64.rpm ./glog-0.3.3-8.el7.x86_64.rpm ./gflags-devel-2.1.1-6.el7.x86_64.rpm ./glog-devel-0.3.3-8.el7.x86_64.rpm

Build and Package Double-Conversion

NOTE: Double-Conversion is a library from Google which provides binary-decimal and decimal-binary routines for IEEE doubles.  Details and source code here: https://github.com/google/double-conversion

Download

cd /build
git clone https://github.com/google/double-conversion.git

Build

cd /build/double-conversion
make
make test
scons install
chmod 0644 /usr/local/lib/libdouble-conversion*
mkdir /usr/local/include/double-conversion
cp double-conversion/*.h /usr/local/include/double-conversion/
chmod 0755 /usr/local/include/double-conversion/
chmod 0644 /usr/local/include/double-conversion/*

Package

cd /build
mkdir -p /build/dcpkg/usr/local/lib/
mkdir -p /build/dcpkg/usr/local/include/double-conversion/

cp -rp /usr/local/lib/libdouble-conversion* /build/dcpkg/usr/local/lib/
cp -rp /usr/local/include/double-conversion/* /build/dcpkg/usr/local/include/double-conversion/

cd /build/double-conversion
VER=$(git tag|tail -1)
cd /build

fpm --verbose -f -s dir -t rpm -n double-conversion -v $VER -C /build/dcpkg \
usr/local/lib/ \
usr/local/include/double-conversion/

You should have an RPM that will install double-conversion.

Build and Package Folly

Lots of information on Folly: https://github.com/facebook/folly

Folly (acronymed loosely after Facebook Open Source Library) is a library of C++11 components designed with practicality and efficiency in mind. Folly contains a variety of core library components used extensively at Facebook. In particular, it’s often a dependency of Facebook’s other open source C++ efforts and place where those projects can share code.

Download

cd /build
git clone https://github.com/facebook/folly

Build

cd /build/folly/folly
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
VER=$(git tag|tail -1)
autoreconf -ivf
LDFLAGS=-L/usr/local/lib CPPFLAGS=-I/usr/local/include ./configure
make -j2
make install

Package

cd /build/folly/folly
mkdir /build/follypkg
make install DESTDIR=/build/follypkg
cd /build
fpm --verbose -f -s dir -t rpm -n folly -v $VER -C /build/follypkg \
usr/local/include \
usr/local/lib

You should have an RPM that will install Folly.

Upgrade your Automake

NOTE: This is only required if you have a version of Automake lower than 1.15

curl -O https://kojipkgs.fedoraproject.org/packages/automake/1.15/1.fc22/noarch/automake-1.15-1.fc22.noarch.rpm
rpm -Uvh automake-1.15-1.fc22.noarch.rpm

Download and Install jemalloc

curl -O https://kojipkgs.fedoraproject.org/packages/jemalloc/3.6.0/1.el7/x86_64/jemalloc-3.6.0-1.el7.x86_64.rpm
curl -O https://kojipkgs.fedoraproject.org//packages/jemalloc/3.6.0/1.el7/x86_64/jemalloc-devel-3.6.0-1.el7.x86_64.rpm

rpm -ivh jemalloc-devel-3.6.0-1.el7.x86_64.rpm jemalloc-3.6.0-1.el7.x86_64.rpm

Build and Package Wangle

More on Wangle: https://github.com/facebook/wangle

Wangle provides a full featured, high performance C++ futures implementation.

Download

cd /build
git clone https://github.com/facebook/wangle

Build

cd /build/wangle/wangle
VER=$(git tag|tail -1)
sed -i -r -e "s@\ \ -lpthread@\ \ -lgflags\n\ \ -lgflags@" CMakeLists.txt
cmake .
make
make test
make install

Package

cd /build/wangle/wangle
mkdir /build/wanglepkg
make install DESTDIR=/build/wanglepkg
cd /build
fpm --verbose -f -s dir -t rpm -n wangle -v $VER -C /build/wanglepkg \
usr/local/include \
usr/local/lib

You should have an RPM that will install Wangle.

Build and Package FBThrift

Read lots about fbthrift here: https://github.com/facebook/fbthrift

FBThrift: Facebook’s branch of apache thrift

Download

cd /build
git clone https://github.com/facebook/fbthrift

Build

cd /build/fbthrift/thrift
VER=$(git tag|tail -1)
autoreconf -ifv
LDFLAGS=-L/usr/local/lib CPPFLAGS=-I/usr/local/include ./configure
make -j2
make install


Package

cd /build/fbthrift/thrift
mkdir /build/fbthriftpkg
make install DESTDIR=/build/fbthriftpkg
cd /build
fpm --verbose -f -s dir -t rpm -n fbthrift -v $VER -C /build/fbthriftpkg \
usr/local/bin \
usr/local/include \
usr/local/lib

You should have an RPM that will install Thrift.

FINALLY!!! Build and Package McRouter!

Download

cd /build
git clone https://github.com/facebook/mcrouter

Build

cd /build/mcrouter/mcrouter
VER=$(git tag|tail -1)
autoreconf -ifv
export THRIFT2_COMP_DIR=/usr/lib/python2.7/site-packages/thrift_py-0.9.0-py2.7.egg/thrift_compiler
LDFLAGS=-L/usr/local/lib CPPFLAGS=-I/usr/local/include ./configure --prefix=/usr/local/mcrouter
make -j2
make install

Package

cd /build/mcrouter/mcrouter
mkdir /build/mcrouterpkg
make install DESTDIR=/build/mcrouterpkg
cd /build
fpm --verbose -f -s dir -t rpm -n mcrouter -v $VER -C /build/mcrouterpkg \
usr/local/mcrouter

You should have an RPM that will install mcrouter

Defining Your Portable Installation

The following RPMs from this tutorial are required to operate a mcrouter environment. Set them aside.

As of this writing, my specific RPMs and versions are:

• gflags-2.1.1-6.el7.x86_64.rpm
• glog-0.3.3-8.el7.x86_64.rpm
• jemalloc-3.6.0-1.el7.x86_64.rpm
• ragel-6.9-1.x86_64.rpm
• double-conversion-v2.0.1-1.x86_64.rpm
• folly-v0.57.0-1.x86_64.rpm
• wangle-v0.9.0-1.x86_64.rpm
• fbthrift-v0.31.0-1.x86_64.rpm
• mcrouter-v0.9.0-1.x86_64.rpm

(Env 2) Setting up The Operating Environment

In order to run mcrouter on another node, you will need to install a the following system packages and packages you downloaded and created:

Install Packages

System Packages

yum -y install cyrus-sasl python-libs numactl-libs snappy boost libtool libevent openssl libatomic

Downloaded Packages

rpm -ivh gflags-2.1.1-6.el7.x86_64.rpm glog-0.3.3-8.el7.x86_64.rpm jemalloc-3.6.0-1.el7.x86_64.rpm

Built Packages

rpm -ivh ragel-6.9-1.x86_64.rpm double-conversion-v2.0.1-1.x86_64.rpm folly-v0.57.0-1.x86_64.rpm wangle-v0.9.0-1.x86_64.rpm fbthrift-v0.31.0-1.x86_64.rpm mcrouter-v0.9.0-1.x86_64.rpm

Test Mcrouter

/usr/local/mcrouter/bin/mcrouter --version

Done!

Did you find this post useful or have questions or comments? Please let me know!

The post HOWTO Build and Install Mcrouter for RHEL, OL, CENTOS 7.x first appeared on Brain Goo.

This is useful when trying to match an known string with a key name returned from another source where the characters are correct, but may come back in camelCase or in the wrong or just unknown case.

First: Write the function.
def renameKeys(iterable):
    if type(iterable) is dict:
        for key in iterable.keys():
            iterable[key.lower()] = iterable.pop(key)
            if type(iterable[key.lower()]) is dict or type(iterable[key.lower()]) is list:
                iterable[key.lower()] = renameKeys(iterable[key.lower()])
    elif type(iterable) is list:
        for item in iterable:
            item = renameKeys(item)
    return iterable


Next, create some sort of test list:
somelist = [
  {
  'myKey': 'asdA',
  'ASDFASDurKey':
    [
      {'QDD' : 'booger'},
      'asdf',
      [1,2]
    ]
  }
]


Finally call it:
somelist = renameKeys(somelist)


CAUTION: If your dictionary has two elements with the same ascii characters in two separate cases such as mydict[‘key1’] and mydict[‘KEY1’], the function works on a last-match basis. That means whichever dictionary element gets renamed last wins.

Did you find this post useful or have questions or comments? Please let me know!

The post HOW TO Lower Case All Dictionary Keys in a Complex Python Dictionary first appeared on Brain Goo.

Lists are one-dimensional arrays and dicts are associative arrays or hash arrays.  This means we can do the same thing we do in other languages, with syntax to match Python’s object-oriented data structures.

import pprint

# Define the list
somelist = []

# Do add some elements to the list
somelist.append({'key1':'value1', 'key2': 'value2'})
somelist.append({'key1':'value1', 'key2': 'value2'})
somelist.append({'key1':'value1', 'key2': 'value2'})

# Print it out
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(somelist)

Will give you:

[   {   'key1': 'value1', 'key2': 'value2'},
{   'key1': 'value1', 'key2': 'value2'},
{   'key1': 'value1', 'key2': 'value2'}]

Did you find this post useful or have questions or comments? Please let me know!

The post Python HOWTO Push a Dict on to a List (push a hash on to an array) first appeared on Brain Goo.

join EXPR,LIST
Joins the separate strings of LIST into a single string with fields separated by the value of EXPR, and returns that new string. Example:
$rec = join(':', $login,$passwd,$uid,$gid,$gcos,$home,$shell);

From the code example, you can make CSV output and all kinds of goodies, but what the doc misses and the example doesn’t show is that combining join() with a print statement makes writing lists to STDOUT or a file handle a snap. This is where join() really shines.

Example:

Code
@names = ('Mark', 'Jim', 'Bob','Mary','Steven','Gomer');
print join("\n", @names);

Output
Mark
Jim
Bob
Mary
Steven
Gomer

Note that this will NOT print a final or beginning string. Join() concatenates the elements, meaning it puts the string value BETWEEN the list elements.

Did you find this post useful or have questions or comments? Please let me know!

The post PERL How To Print A List Without A Loop first appeared on Brain Goo.

Warning: Unknown: Releasing SysV semaphore id 58 key 0x152b in request cleanup in Unknown on line 0


This is not an error from WordPress, but from the WP Super Cache Plugin you forgot you installed. The error comes when from the cache directory is not writable.

Change the permissions on the wp-content/cache directory to allow the web user to write and the error will disappear.

The solution is obvious if you check the source code on the page when the error appears. WP Super Cache spits out the error, but it is hidden in HTML comment tags. Your source code will look something like this:

<!-- Dynamic Page Served (once) in 0.297 seconds -->
<!-- File not cached! Super Cache Couldn't write to: wp-content/cache/wp-cache-2876abde8ce3a2aa817c8289ea85486e.html -->

87
<br />
<b>Warning</b>: Unknown: Releasing SysV semaphore id 58 key 0x152b in request cleanup in <b>Unknown</b> on line <b>0</b><br />

The post Word Press problem Unknown: Releasing SysV semaphore first appeared on Brain Goo.

The fastest way to deny based on RBL is to add it to whatever ACL you specify in acl_smtp_rcpt

However, you MUST put the declaration AFTER any relay allow definitions. ACLs are based on first-match which means they run in order and stop when they hit a match. Implicit allow.

Here is my ACL declared as acl_check_rcpt

acl_check_rcpt:
  accept  hosts = :
  deny    local_parts   = ^.*[@%!/|] : ^\\.   
  accept  local_parts   = postmaster
          domains       = +local_domains
  require verify        = sender
  deny    dnslists = zen.spamhaus.org
          message = Message rejected because $sender_fullhost is blacklisted at $dnslist_domain see $dnslist_text
  accept  domains       = +local_domains
          endpass
          message       = unknown user        
          verify        = recipient
  accept  domains       = +relay_to_domains
          endpass
          message       = unrouteable address
          verify        = recipient
  accept  hosts         = +relay_from_hosts               
  accept  authenticated = *
  deny    message       = relay not permitted

The RBL definition is toward the bottom, after we allow everyone in that we want in. This lets people relay via SMTP-AUTH or explicit allow before checking the RBL. If they aren’t allowed via anything we allow, then we check the RBL and die with a nice message.

Adverts:
The Exim SMTP Mail Server
Exim: The Mail Transfer Agent
The Exim SMTP Mail Server: Official Guide for Release 4
A Practical Guide to Linux(R) Commands, Editors, and Shell Programming
Understanding the Linux Kernel
Ubuntu Hacks: Tips & Tools for Exploring, Using, and Tuning Linux (Hacks)
Linux Pocket Guide

The post RBLs with Exim4 (debian friendly) first appeared on Brain Goo.

What you need:

• iCal client like Sunbird or Lightning from Mozilla
• Apache web server with WebDav installed

How to do it:

1. Install WebDav and password protect a public directory on your web server.
2. Touch a .ics file for your calendar. I called mine matt.ics
3. chmod the directory so the web user can create files and modify the .ics file
4. Install and configure an iCal client that can publish via WebDav. Sunbird is excellent for this.
5. Remove your local calendar from Sunbird
6. Add a calendar to Sunbird using the URL to your empty .ics file
7. Add an event to your calendar. Sunbird should modify the .ics file on your web server. You will probably be prompted for the username and password on the first try.
8. Add that calendar to as many clients as you want. They can all share the calendar. Create more calendars! Party!

The post Poor man’s calendar server with iCal and Mozilla first appeared on Brain Goo.

The post Panic Releases Coda, and I can’t use it first appeared on Brain Goo.