Florent Clairambault » PHP

Prestashop and APC

Florent Clairambault — Mon, 04 Oct 2010 22:18:45 +0000

When you do a fresh install of prestashop, just after it finishes you need to rename the “admin” directory. The problem is that with the last two installs we did (as part of WebIngenia), we ended up with a blank/empty/white webpage. The only way to solve it was to clear the opcode cache from APC.
So, if you encounter the same problem, you know what to do, you just need to clear the APC opcode cache and continue the install.

BTW, sorry for not having properly maintained this blog recently. Not enough free time. I’ll release an updated version of the TC65 guide pretty soon.

WP-Codebox fix so that it doesn’t prevent posts export

Florent Clairambault — Sat, 15 May 2010 07:00:08 +0000

If you have a PHP configured with some open_basedir restriction (which is recommended), the WP-Codebox plugin will prevent you from exporting the posts of your blog.

Go to your blog, go in the “wp-content/plugins/wp-codebox” dir and edit the “wp-codebox.php” file :
On line 27, you should have :

1 2	include_once "../../../wp-config.php"; include_once "../../../wp-settings.php";

Well, it fails because the plugin doesn’t use the good practices, so it’s pretty easy to fix :

1 2	require_once( ABSPATH.'/wp-config.php' ); require_once( ABSPATH.'/wp-settings.php' );

I’d like to thank the developer of this plugin for doing such a great job. It brings the power of geshi (not the latest version by the way) into wordpress.

Linux prioritization : do more with less

Florent Clairambault — Sun, 07 Mar 2010 18:34:04 +0000

I find the concept of prioritization very interesting. It just enables you to do more with less. Doesn’t that sound great ?

Let’s say you want to be able to respond to user requests as fast as possible but update your data in a low priority manner :
You can set the process CPU priority from -20 (high priority) to 19 (low priority) by using the command :

nice -n

You can set the process IO priority in 4 classes (0: none, 1: realtime, 2: best-effort, 3: idle) with some priorities within these classes (0-7, lower being higher prio). But you have to enable the CFQ (Complete Fair Queueing) scheduler first by typing something like that :

1	echo cfq >/sys/block/hda/queue/scheduler

So the ultimate low priority command will be

1	ionice -c 3 nice -n 20

But sometimes changing the CPU and IO priority won’t change much because the problem you might have might occur within the SQL server for say. So what you do ? Well, you could slow down your low priority program. If you have a low priority php script, you could do it like that :


while( true ) {
    list( $load ) = explode(' ', file_get_contents('/proc/loadavg') );
    echo 'sleeping '.$load."s\n";
    usleep( $load * 1000000);
}
?>

This program will make a loop that will slow down with the increasing load. It means that this program will always keep space of the system, even if it’s run hundreads of time.

So yeah, you can now manage efficiently CPU and disk. Are you done ? Not really, there’s still the memory issue. Memory (RAM) is always fast unless there’s no memory left, then it’s paged and everything becomes thousand times slower. You can only disable the virtual memory (swap on Linux), set some memory allocation limits (with ulimit), but that’s pretty much it.
I would personnally recommend to disable swap and always take more RAM than needed. On servers swap prevent them from having to kill process, but they are so much slowed by it that the whole system is slowed down. And then even ssh server is so slow you have an ssh timeout before reaching the shell.

So, Linux has a great scheduling capacity. But what about NOT scheduling AT ALL ? Well, Linux is also very able to do that. You can put some process in real-time mode, these processes won’t be interupted by anything unless they are sleeping or waiting for an I/O event.
You can use the Real-Time (RT) mode using the rtprio command :

rtprio 99

Dirty WordPress APC caching

Florent Clairambault — Sat, 13 Feb 2010 08:00:36 +0000

One or two weeks ago, I made a simple AB benchmarking test on a PHP site I built, it was ok. Then I did the same test on this blog and well… It was freaking slow… On 100 pages with 10 concurrent access, it took 3.5 to 10s to render. Well, I thought I should remove all these plugins I installed to make me and my blog famous (they didn’t perform well). It reduced the generation time by something like 100 ms.

So I had the brilliant idea to put everything in an APC cache, someone had the same idea.

So I approximatively did the same thing :

In the beginning of the index.php file :

if ( $_SERVER['REQUEST_METHOD'] == 'GET' ) {
 
        // This key will identify the page used
        $cacheKey = 'WP_'.$_SERVER['HTTP_HOST'].md5( $_SERVER['REQUEST_URI'] );
 
        // We try to get this page
        $cacheData = apc_fetch($cacheKey);
 
        // If we got something
        if (is_array( $cacheData )) {
 
                // We give each header
                foreach( $cacheData[0] as $h )
                        header( $h );
 
                // And the content
                die($cacheData[1]);
        }
 
        // If we're still here, we have no cache, so we'll start caching right now
        ob_start();
}

In the end of the index.php file :

1
2
3

// We have a cache key, it means we have something to cache
if ( $cacheKey )
        apc_store( $cacheKey, array( headers_list(), ob_get_contents() ), 900 );

And once it’s cached it performs well, to make it short it can serve in mean time of 1 ms and gives a general bandwidth of 200 MB/s. As you can guess, these tests are made locally and this is normal.

[root@s15342968 ~]# ab -n 100  "http://florent.clairambault.fr/"
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
 
Benchmarking florent.clairambault.fr (be patient).....done
 
 
Server Software:        Apache/2.2.8
Server Hostname:        florent.clairambault.fr
Server Port:            80
 
Document Path:          /
Document Length:        161165 bytes
 
Concurrency Level:      1
Time taken for tests:   0.119899 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      16156100 bytes
HTML transferred:       16116500 bytes
Requests per second:    834.04 [#/sec] (mean)
Time per request:       1.199 [ms] (mean)
Time per request:       1.199 [ms] (mean, across all concurrent requests)
Transfer rate:          131585.75 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     1    1   0.0      1       1
Waiting:        0    0   0.1      0       1
Total:          1    1   0.0      1       1
 
Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      1
  90%      1
  95%      1
  98%      1
  99%      1
 100%      1 (longest request)

[root@s15342968 ~]# ab -n 100 -c 10  "http://florent.clairambault.fr/"
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
 
Benchmarking florent.clairambault.fr (be patient).....done
 
 
Server Software:        Apache/2.2.8
Server Hostname:        florent.clairambault.fr
Server Port:            80
 
Document Path:          /
Document Length:        161165 bytes
 
Concurrency Level:      10
Time taken for tests:   0.47897 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      16156100 bytes
HTML transferred:       16116500 bytes
Requests per second:    2087.81 [#/sec] (mean)
Time per request:       4.790 [ms] (mean)
Time per request:       0.479 [ms] (mean, across all concurrent requests)
Transfer rate:          329394.31 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     1    4   1.3      4      10
Waiting:        0    1   1.0      1       4
Total:          1    4   1.3      4      10
 
Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      5
  80%      5
  90%      6
  95%      6
  98%      8
  99%     10
 100%     10 (longest request)

[root@s15342968 ~]# ab -n 10000 -c 10  "http://florent.clairambault.fr/"
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
 
Benchmarking florent.clairambault.fr (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Finished 10000 requests
 
 
Server Software:        Apache/2.2.8
Server Hostname:        florent.clairambault.fr
Server Port:            80
 
Document Path:          /
Document Length:        161165 bytes
 
Concurrency Level:      10
Time taken for tests:   7.898135 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1616094683 bytes
HTML transferred:       1612133495 bytes
Requests per second:    1266.12 [#/sec] (mean)
Time per request:       7.898 [ms] (mean)
Time per request:       0.790 [ms] (mean, across all concurrent requests)
Transfer rate:          199821.47 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.9      0       7
Processing:     2    6   1.6      7      13
Waiting:        0    1   1.4      1       8
Total:          2    7   1.5      7      16
 
Percentage of the requests served within a certain time (ms)
  50%      7
  66%      8
  75%      8
  80%      8
  90%      9
  95%     10
  98%     11
  99%     11
 100%     16 (longest request)

WordPress with APC

Florent Clairambault — Fri, 12 Feb 2010 20:00:17 +0000

I updated APC from 3.0 to 3.1 because it totally locked my webserver, it was making it accept HTTP connection but never give any reply.

I had some problem accessing the WordPress 3.0 (latest SVN version) admin interface after that upgrade :

1	Fatal error: Call to undefined function wp_dashboard_setup() in /home/sites/clairambault.fr/florent/wp-admin/index.php on line 15

the bad line is :

1	require_once(ABSPATH . 'wp-admin/includes/dashboard.php');

I just found out here it was because I activated APC with the “apc.include_once_override” option set to 1. If you encounter the same problem, just set it to 0 or change the include by something without the ABSPATH constant like for instance :

1	require_once('./includes/dashboard.php');

Note : requiring local files with “./” makes you save some CPU time.

I personally chose to change the wp-admin/index.php file because I didn’t want to reduce the performance of the other applications (I haven’t done any benchmark of the performance difference though).

Push (Comet) or pull ?

Florent Clairambault — Mon, 23 Nov 2009 12:45:30 +0000

What is it ?
Most of the modern web applications display recently updated data, and to do that they need to get he latest data very frequently. Some of them even include some real-time chat (Gmail Chat, Facebook chat).

How ?
That’s the interesting part.

Pull is pretty dumb. You do a request very frequently and you see if anything new appeared. This consumes some bandwidth, some resources (because server has to check if data has actually changed).
Push is going back to the source : Once you’ve made the request on the server, it doesn’t reply instantly. It will wait for something before sending anything. So push over HTTP is in fact a pull with a sleeping response. Using push over HTTP is called Comet.

So pushing data isn’t very complex, it just requires a special server to transmit some data (text, html, xml, or image) over an already opened HTTP connection.

Why do we need some special servers ? Because our current servers aren’t build to make people wait. Some geeks made some example of PHP code being able to make some Comet response but they all use looping usleep. If you used it, you would end up totally killing your server with the growing number of clients.
This is the very interesting part of Comet : It doesn’t consumes more resources, it just requires additional server software.

Two types of push
They are two types of push :

The multiple limited time push requests or long poll mode : You make a request and each time you get the response the connection is closed and you have to make an other pull. It’s easier to implement but you could end up doing a lot of http requests, which means generating quite some data.
The streaming push mode : You open the connection once, and then data comes asynchronously, and the connection is never closed. If it “accidentally” closes, the client reopens it.

How do do some push enabled web applications
You rethink your software so that it uses the push server.

On PHP, you can use The Ajax Push Engine.
On pure .Net environnement, you can use WCF WsDualHttpBinding to serve data.
The CometD is a stable opensource comet server project. JQuery and Dojo can consume it.
You have LightStreamer, a commercial product with a free license. It can be connected with nearly anything (the only unsupported type of server seems to be PHP) or StreamHub which has a very restrictive free license.

Why some people still use Flex ou SilverLight ?
Http push / Comet requires a complex parallel server infrastructure and it only solves the push (server to client realtime data transmission) problem. When data is received, you still have to treat and display it. Managing this is a lot harder with javascript than it is in Flex ou SilverLight.

By the way, on flex, silverlight, flash, java applets you can use your own sockets to transmit data. This is a very good solution, there a good chance it will consume less bandwidth. But you have to remember that not using the HTTP protocol can create some problems as some companies block every other protocols on their internet gateways.

W3C is on it !
W3C is thinking about adding a WebSocket specification. This could be a very good thing as it would standardize this non-standardized part of the web.

Push outside the web
Push isn’t a web specific concept. Any instant messaging service (like MSN, Jabber, Yahoo or ICQ) does support push.

Any system supporting sockets enables you to make push enabled applications. If you consider the mobile OSes, they all support socket, so they all support push. Androïd supports push by socket, J2ME supports push by sockets, iPhone supports push by Apple push system or sockets and it seems BlackBerry supports it by the BlackBerry push service or sockets.

Automatic error reporting in PHP

Florent Clairambault — Sun, 15 Feb 2009 08:35:37 +0000

I edited this page on the 21 March 2010 because a lot of people seem interested and the code as since improved !

PHP has a pretty interesting feature, you can define a callback method to “catch” any error “thrown” in your code. And I’m sure most of you don’t use it. It’s really usefull when you want to make sure to detect error before any user reports it (which can takes time). This is all about avoiding to demolish with some lame errors your “user experience”.

I now use it in each of my index.php pages (which generally loads every other pages), but to speed things up I make it load the actual method only when the error is “catched”.

This is the code :

function lightErrorHandler($errno, $errstr, $errfile, $errline, $shutdown = false ) {
	// When called from the shutdown function, the relative path doesn't work anymore.
	// You have to load the errorHandler function from its absolute path
	// If you don't like that method, you can always preload this function.
	require_once('/home/website/mysite.com/dev-www/include/error/errorHandler.inc.php');
	return errorHandler($errno, $errstr, $errfile, $errline, $shutdown);
}
set_error_handler( 'lightErrorHandler', E_ALL ^ E_NOTICE);
 
function lightExceptionHandler( $exception ) {
	require_once('./include/error/exceptionHandler.inc.php');
	return exceptionHandler( $exception );
}
set_exception_handler( 'lightExceptionHandler' );
 
function shutdown_function() {
	if(is_null($e = error_get_last()) === false && $e['type'] & (E_ALL ^ E_NOTICE) ) {
		lightErrorHandler( $e['type'], $e['message'], $e['file'], $e['line'], true );
	}
}
register_shutdown_function('shutdown_function');

include/error/errorHandler.inc.php :

function errorHandler($errno, $errstr, $errfile, $errline, $shutdown) {
	global $engine;
 
	$tab = array(
	        'no'    => $errno,
	        'str'   => $errstr,
	        'file'  => $errfile,
	        'line'  => $errline
	);
 
	$message = 'An error happened :'."\n\n".'Error : '."\n".print_r( $tab, true )."\n\n".'StackTrace : '."\n\n".print_r( debug_backtrace(), true )."\r\n".'Memory state : '."\n".print_r( $GLOBALS, true )."\n";
 
	mail(
	        'email@company.com',
	        'MyProject : Error : '.$errfile.':'.$errline,
	        $message
	);
 
	$target = $errfile.':'.$errline;
 
	if ( ! $engine['bug'] && ! $shutdown ) {
		$engine['bug'] = true;
		Logger::log(array(
			'message'		=> 'Error : '.$errstr.' ('.$errno.')',
			'type'			=> 'error/codeError',
			'target'		=> substr( $target, 0-min(250, strlen( $target ))),
			'data'			=> serialize($GLOBALS),
			'level'			=> Logger::CRITICAL
		));
	}
 
	return false;
}

include/error/exceptionHandler.inc.php :

function exceptionHandler( $exception ) {
	global $engine;
 
 
	$_SESSION['lastException'] = $exception;
 
	$message = 'An error happened :'."\n\n".'Error : '."\n".$exception."\n\n".'StackTrace : '.debug_backtrace()."\r\n".'Informations diverses : '."\n".print_r( $GLOBALS, true )."\n";
 
	mail(
	        'email@company.com',
	        'MyProject : Error : '.$exception->getFile().':'.$exception->getLine(),
	        $message
	);
 
	$target = $exception->getFile().':'.$exception->getLine();
 
	if ( ! $engine['bug'] ) {
		$engine['bug'] = true;
		Logger::log(array(
			'message'		=> 'An exception was thrown',
			'type'			=> 'error/exception',
			'target'		=> substr( $target, 0-min(250, strlen( $target ))),
			'data'			=> serialize($GLOBALS),
			'level'			=> Logger::CRITICAL
		));
	}
 
	return false;
}

debug_backtrace requires PHP 4.3 and set_error_handler only supports error types since PHP 5.0. So, if you plan on using this on a PHP 4.X host, you have to make sure your code doesn’t throw E_NOTICE errors. My code is never E_NOTICE error safe.