FeedMyMedia

Scaling WordPress: A FeedMyMedia Case Study

Ashley Shaw — Sat, 01 Oct 2011 17:58:28 +0000

I spoke at WordCamp Cape Town on 15th of September 2011 about the FeedMyMedia.com infrastructure and scaling WordPress.

We have been researching how to setup an optimized WordPress Multisite install as well as how to scale the server resources to server hundreds of thousands of HTTP requests. Its so hard to tell when you will need to scale, my advice is to scale before you need to.

Figure out how much your system can handle before it cracks. Here is what we did to setup a self-monitoring optimized dedicated redundant server, which is not scalable (will get to that later) yet but at least a solid base. Read about FeedMyMedia over the coming weeks:

Ashley Shaw (Scaling WordPress) Presentation

View more presentations from WordCamp Cape Town

FeedMyMedia (The Setup!)

We have the following in our cluster:

3x Load Balancers
3x web-tier servers
1x Dedicated Database servers
1x Caching / Backup / Database server
2x Development servers

Our web-tier servers each run an optimised NGiNX configuration, php-fpm dynamic pool and a memcached instance, the three servers are synchronised using Unison.

The dedicated database server’s run Percona MySQL Server (which offers higher performance and stability over standard MySQL)

Caching we use the memcached object cache + WordPress Batcache (which utilizes the memcached-backend object cache)

STATS

Launched 2009 at Linode
Number of HTTP requests: FMM ~100k per week, M24 ~200K per week
Number of MySQL queries: ~500 qps (queries per second)
Number of sites hosted: ~100 Live Sites, ~670 Dev Sites
Monthly bandwidth usage: ~1TB per month, excluding internal traffic between servers

WordPress Caching

Ashley Shaw — Sat, 01 Oct 2011 15:24:21 +0000

We use Batcache (WordPress Caching) + Memcached Object Cache

Memcached Backend (Object Cache)

Caches queries, not page
Typical WordPress queries:

bloginfo() options (site title, description, language, theme stylesheet URL)
post or page title and contents
comments
sidebar widgets

Batcache

Store cached pages in memcached
Reduces disk I/O load

We touched on the use of batcache last we spoke. I would like to know as much as possible about how we have implemented batcache, issues encountered, solutions, as much as possible.

Batcache Implementation is fairly easy

Download latest version http://downloads.wordpress.org/plugin/batcache.1.0.zip-
Upload advanced-cache.php to the /wp-content/ directory
Add this line the top of wp-config.php to activate Batcache: define(‘WP_CACHE’, true);
Tweak the options near the top of advanced-cache.php
Optionally, upload batcache.php to /wp-content/plugins/ directory.
Issues encountered – none really, works as it is intended to.

XCACHE (Opcode Caching)

compiled PHP cache stored in memory
Alternatives available (eAccellerator and APC), XCACHE works best with PHP-FPM and WordPress

MySQL with InnoDB I/O and HyperDB

Ashley Shaw — Sat, 01 Oct 2011 15:24:18 +0000

Percona MySQL 5.5

The dedicated database server’s run Percona MySQL Server (which offers higher performance and stability over standard MySQL)

We use a special version of MySQL from Percona – http://www.percona.com/software/

Percona Server is an enhanced drop-in replacement for MySQL.

Percona Server Feature Comparison

Queries will run faster and more consistently.
Consolidate servers on powerful hardware.
Delay sharding, or avoid it entirely.
Save money on hosting fees and power.
Spend less time tuning and administering.
Achieve higher uptime.
Ttroubleshoot without guesswork.

Benchmark for Percona Server

Percona Server is more than 40% faster than MySQL in overall throughput.
Percona Server performance remains more stable over time than MySQL.

Benchmarking

Super Smack (http://jeremy.zawodny.com/mysql/super-smack/)
mysqlslap (http://dev.mysql.com/doc/refman/5.5/en/mysqlslap.html)

Tuning configuration (my.cnf)

mysqltuner (http://mysqltuner.pl/mysqltuner.pl)
Tuning Primer (https://launchpad.net/mysql-tuning-primer)

InnoDB I/O Scalability

InnoDB is a complex storage engine, but must be configured properly to perform at its best.

Tuning my.cnf for InnoDB I/O scalability:

innodb_buffer_pool_size – Set the amount of memory allocated to both InnoDB and the index buffer cache, when the server requests data available in the cache it can be processed right away. Otherwise, the OS will request the data be loaded from disk to buffer, therefore it is important to set the value as high as possible to use the more efficient InnoDB data and index buffer cache instead of operating system buffer. Recommended to be set to 70 – 80% of available primary memory.

innodb_flush_log_at_trx_commit – InnoDB flushes the transaction log to disk once per second in the background. Default value is set to 1. innodb_flush_log_at_trx_commit = 0 – No log flushing on each transaction commit, however transaction data will be lost in case of MySQL crash. Recommended use on replication slaves that can offer a performance improvement of ~4%. innodb_flush_log_at_trx_commit = 2 – flush the log to Operating Systems Cache to save disk I/O on each transaction commit.

HyperDB

Replacement for the WPDB class which adds the ability to use multiple databases. HyperDB allows multiple servers to be configured and multiple routes to servers.

NGINX with PHP-FPM

Ashley Shaw — Sat, 01 Oct 2011 15:24:11 +0000

NGINX

How to achieve optimal NGiNX configuration:

gzip static assets (jpg, css, javascript)
worker_processes = number of CPU’s
try_files not if
expires

PHP-FPM

Process management
Dynamic process spawning
Auto recovery if crash
Easy to integrate with NGiNX
Use Unix sockets (not TCP/IP) achieve optimal and secure PHP-FPM setup

NGINX Configuratrion

This NGINX configuration guide and setup is optimised for Ubuntu 10.04 or later releases

Stable – nginx-1.0.5 (http://nginx.org/download/nginx-1.0.5.tar.gz)
Linux Kernel 3.0 builds, use dev version 1.1.1 (http://nginx.org/download/nginx-1.1.1.tar.gz)

./configure \
–prefix=/opt/nginx \
–conf-path=/etc/nginx/nginx.conf \
–pid-path=/var/run/nginx.pid \
–lock-path=/var/lock/nginx.lock \
–http-log-path=/var/log/nginx/access.log \
–error-log-path=/var/log/nginx/error.log \
–http-client-body-temp-path=/var/lib/nginx/body \
–http-proxy-temp-path=/var/lib/nginx/proxy \
–http-fastcgi-temp-path=/var/lib/nginx/fastcgi \
–http-uwsgi-temp-path=/var/lib/nginx/uwsgi \
–http-scgi-temp-path=/var/lib/nginx/scgi \
–with-http_ssl_module \
–with-http_gzip_static_module \
–user=www-data \
–group=www-data \
–without-mail_pop3_module \
–without-mail_imap_module \
–without-mail_smtp_module

NGiNX startup script (e.g. /etc/init.d/nginx)

Download http://nginx-init-ubuntu.googlecode.com/files/nginx-init-ubuntu_v2.0.0-RC2.tar.bz2
Change:
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/nginx/sbin
DAEMON=/opt/nginx/sbin/nginx
lockfile=/var/lock/nginx.lock
NGINX_CONF_FILE=”/etc/nginx/nginx.conf”

Basic NGiNX WordPress MS config (e.g. /etc/nginx/sites-enabled/sitename.com)

server {
listen 80;
server_name sitename.com *.sitename.com;
error_log /var/log/nginx/sitename.com/error.log;
access_log /var/log/nginx/sitename.com/access.log;
root /var/www/sitename.com/;
index index.php;location / {
try_files $uri $uri/ /index.php?$args;
}# Expires for static assets
location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
expires 24h;
log_not_found off;
}

# Rewrite multisite upload files
location ^~ /files/ {
rewrite /files/(.+) /wp-includes/ms-files.php?file=$1 last;
}

location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
include fastcgi_params;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_pass php;
}
}

Main NGiNX Configuration (e.g. /etc/nginx/nginx.conf) - Optimised for Linux O/S

# Generic startup file.
user {user} {group};
#worker_processes should be equal to the amount of available processing threads/less
worker_processes 4;
error_log /var/log/nginx/error.log;
access_log /var/log/nginx/access.log
pid /var/run/nginx.pid;events {

#max allowable worker_connections per worker_process
worker_connections 1024;

multi_accept on;

use epoll;
}

http {
include mime.types;
default_type application/octet-stream;

sendfile on;

tcp_nodelay on;

tcp_nopush on;

keepalive_timeout 65;

client_header_timeout = 300;

client_body_timeout = 300;

send_timeout = 300;

connection_pool_size 256;

client_header_buffer_size 128k;

large_client_header_buffers 4 2k;

output_buffers 1 32k;

postpone_output 1460;

ignore_invalid_headers on;

marker

#Reset timed out connections

reset_timedout_connection on;

##Compression

gzip on;

gzip_http_version 1.1;

gzip_buffers 16 8k;

gzip_vary on;

gzip_proxied any;

gzip_comp_level 2;

gzip_disable “msie6″;

gzip_types text/plain text/css application/x-javascript application/javascript text/xml application/xml application/xml+rss text/javascript;

#Serve already compressed files directly

gzip_static on;

# Upstream to abstract backend connection(s) for PHP.
upstream php {
server unix:/tmp/php-fpm.sock;
}

include sites-enabled/*;
}

PHP-FPM Configuration (Part of PHP-5.3+)

This PHP-FPM configuration guide and setup is optimised for Ubuntu 10.04 or later releases

Stable – PHP-5.3.8 (http://us2.php.net/distributions/php-5.3.8.tar.gz)

./buildconf –force./configure \
–prefix=/opt/php5 \
–with-config-file-path=/opt/php5/etc \
–with-config-file-scan-dir=/opt/php5/etc/conf.d \
–with-fpm-user=www-data \
–with-fpm-group=www-data \
–enable-fpm \
–with-mcrypt \
–enable-mbstring \
–with-openssl \
–with-mysql \
–with-mysql-sock \
–with-gd \
–with-png-dir \
–with-jpeg-dir \
–with-xpm-dir= \
–enable-gd-native-ttf \
–with-pdo-mysql \
–with-libxml-dir \
–with-mysqli \
–with-curl \
–enable-zip \
–enable-sockets \
–with-zlib \
–enable-exif \
–enable-ftp \
–with-iconv \
–with-gettext \
–enable-gd-native-ttf \
–with-t1lib \
–with-freetype-dir

make
make install

PHP Configuration (e.g. /opt/php5/etc/php.ini)

[Optimised variables to set for WPMS + NGiNX]

; Maximum execution time of each script, in seconds
max_execution_time = 300
; Maximum amount of time each script may spend parsing request data.
max_input_time = 600
; Maximum amount of memory a script may consume (Needs to be set fairly high for hosting a WordPress MS install)
memory_limit = 512M
;Allow file uploads.
file_uploads = On
; Maximum size of post data that PHP will accept. Note: this value needs to be the same as set in nginx.conf under the client_max_body_size paramater.
upload_max_filesize = 100M
; Maximum number of files that can be uploaded via a single request
max_file_uploads = 20
; cgi.fix_pathinfo provides *real* PATH_INFO/PATH_TRANSLATED support for CGI. Note: needs to be set to use fastcgi_split_path_info in nginx.
cgi.fix_pathinfo=0
; Define default timezone
date.timezone = Africa/Johannesburg
; Session
; Handler used to store/retrieve data. Note this needs to be set to tie into your entire memcached pool
session.save_handler = memcache
; Use a comma separated list of server urls to use for storage:
session.save_path = “tcp://localhost:11211?persistent=1&weight=1&timeout=1&retry_interval=15″
;Use cookies – needs to be enabled for WP
session.use_cookies = 1

PHP-FPM Configuration (e.g. /opt/php5/etc/php-fpm.conf)

Optimised variables to set:

;Global Options
;Pid file
pid = run/php-fpm.pid
; Send FPM to background
daemonize = yes;Pool Definitions
;The address on which to accept FastCGI requests, needs to be the same as configured in nginx.conf, we use the unix socket which offers the highest optimisation and security
listen = /var/run/php5-fpm.sock
;Set permissions for unix sockets, needs to be the same as configured in nginx.conf
listen.owner = www-data
listen.group = www-data
;Unix user/group of processes
user = www-data
group – www-data;Choose how the process manager will control the number of child processes.
pm = dynamic
;This value sets the limit on the number of simultaneous requests that will be served, these values have been optimised for a 2Gb box, start with a low enough value and monitor
pm.max_children = 25
; The number of child processes created on startup.
; Default Value: min_spare_servers + (max_spare_servers – min_spare_servers) / 2
pm.start_servers = 4
; The desired minimum number of idle server processes.
pm.min_spare_servers = 2
; The desired maximum number of idle server processes.
pm.max_spare_servers = 10
; The number of requests each child process should execute before respawning.
pm.max_requests = 500;Additional variables that could be set:
php_admin_value[sendmail_path] = /usr/sbin/sendmail -t -i -f email@yoursysadmin.com
php_admin_value[memory_limit] = 512M
php_admin_value[upload_max_filesize] = 100M
php_admin_value[max_file_uploads] = 20
php_admin_value[post_max_size] = 100M
php_admin_value[date.timezone] = Africa/Johannesburg

File Sychronisation using Unison

Ashley Shaw — Sat, 01 Oct 2011 15:24:02 +0000

Unison allows the same version of files to be maintained on multiple servers, therefore when multiple folders/files are synchronised the current version of the file is available on all servers, regardless when it was last modified.

Features:

’Conflicts’ detection, where a file has been modified on two sources
Communication via TCP/IP protocol or by secured tunnel over encrypted SSH
Uses rsync algorithm, only transferring the parts of a file that has changed
Robust design in the event of a program or system crash or communication falure
Extremely fast synchronisation

NFS versus Unison:

NFS allows a system to share directories and files with others over a network, by using NFS users and programs can access files on remote systems, therefore requires the primary NFS mount to always be online or clients won’t be able to access files if its down.

Load Balancing

Ashley Shaw — Sat, 01 Oct 2011 14:59:57 +0000

We use HAProxy Load Balancer, more specifically we are using Linode.com NodeBalancer.

What is it?

Load balancing is a technique to spread work between two or more computers
Load balancer is a tool that directs a client to the least busy or most appropriate web server among several servers that contain mirrored contents

Why?

Web server may not be able to handle high volumes of incoming traffic
Upgrading server resources will no longer be cost effective
Users will have to wait until the web server is free to process their requests

How?

More servers need to be added to distribute the load among the group of servers (aka server cluster)
Load distribution among these servers is known as load balancing.
Load balancing applies to all types of servers

Load balancing algorithms:

Round-robin: requests are rotated among the servers
Leastconn: request is sent to the server with the lowest number of connections
Source: a hash of the source IP is divided by the total weight of the running servers to determine which server will receive the request.