Florent Clairambault

There’s a definition on wikipedia. It’s much much wider than my tiny vision of this business management strategy.

But basically this strategy says : Fix the source of every problem and not each problem. In a factory, it means that have to change the production process to reduce the probability of making defective pieces instead of fixing pieces faster. That means you have to spend time on fixing the problem. So this stategy also means : Spend time to save time. (I know bold formatting is evil but well… I’m evil sorry)

How that applies to me or you ? In fact everywhere. For me it can be seen as the creation of automated processes to replace some humanely driven processes. If you’ve already done a task twice and it can be optimized / automatized, then do it.

I’ve seen a lot of Sharepoint projects that would benefit of some “six sigma” thoughts. They often end up with a huge installation manual instead of optimizing the Sharepoint deployment package of each project.

I find the concept of prioritization very interesting. It just enables you to do more with less. Doesn’t that sound great ?

Let’s say you want to be able to respond to user requests as fast as possible but update your data in a low priority manner :
You can set the process CPU priority from -20 (high priority) to 19 (low priority) by using the command :

1

nice -n <priority> <command>

You can set the process IO priority in 4 classes (0: none, 1: realtime, 2: best-effort, 3: idle) with some priorities within these classes (0-7, lower being higher prio). But you have to enable the CFQ (Complete Fair Queueing) scheduler first by typing something like that :

1

echo cfq >/sys/block/hda/queue/scheduler

So the ultimate low priority command will be

1

ionice -c 3 nice -n 20 <command>

But sometimes changing the CPU and IO priority won’t change much because the problem you might have might occur within the SQL server for say. So what you do ? Well, you could slow down your low priority program. If you have a low priority php script, you could do it like that :

1
2
3
4
5
6
7

<?php
while( true ) {
    list( $load ) = explode(' ', file_get_contents('/proc/loadavg') );
    echo 'sleeping '.$load."s\n";
    usleep( $load * 1000000);
}
?>

This program will make a loop that will slow down with the increasing load. It means that this program will always keep space of the system, even if it’s run hundreads of time.

So yeah, you can now manage efficiently CPU and disk. Are you done ? Not really, there’s still the memory issue. Memory (RAM) is always fast unless there’s no memory left, then it’s paged and everything becomes thousand times slower. You can only disable the virtual memory (swap on Linux), set some memory allocation limits (with ulimit), but that’s pretty much it.
I would personnally recommend to disable swap and always take more RAM than needed. On servers swap prevent them from having to kill process, but they are so much slowed by it that the whole system is slowed down. And then even ssh server is so slow you have an ssh timeout before reaching the shell.

So, Linux has a great scheduling capacity. But what about NOT scheduling ? Well, Linux is also great like with that. You can put some process in real-time mode, these processes won’t be interupted by anything unless they are sleeping or waiting for an I/O event.
You can use the Real-Time (RT) mode using the rtprio command :

1

rtprio 99 <command>

I recently did a simple and stupid comparison test between C++ (g++) and C# .Net (mono).

My point here is that C# can actually be considered as a very fast (as fast as C++) language. It allows automatic hardware-specific optimization.
Lot of people think it’s like a java, it has a virtual machine. But in fact it just compiles everything into native language when it accesses methods. This is the whole purpose of the JIT (Just Int Time compiler).

If you feel the JIT compiler is a possible reason of slowness (it slows down the program startup), you can pre-compile your assemblys (mono –aot on linux). If you feel the mono files are a drawback, you can bundle everything in one file using mkbundle. You can read more about this here.

The only real drawback you have in the .Net framework is the garbage collector. It prevents the C# .Net it from being a realtime language/framework. But in real life usage it doesn’t slow anything down.

The purpose of the following test is to show that C# isn’t slower than C++. But if you’re searching for arguments to choose C# .Net over native C++, you should also consider this :

• Mono (and presumely the .Net framework) does some really interesting optimizations :

  1	# mono --list-opt

  I put some link so that you can learn about these really interesting optimization. This is like design patterns, you should really know these things.

  peephole	Peephole postpass
  branch	Branch optimizations
  inline	Inline method calls
  cfold	Constant folding
  consprop	Constant propagation
  copyprop	Copy propagation
  deadce	Dead code elimination
  linears	Linear scan global reg allocation
  cmov	Conditional moves
  shared	Emit per-domain code
  sched	Instruction scheduling
  intrins	Intrinsic method implementations
  tailc	Tail recursion and tail calls
  loop	Loop related optimizations
  fcmov	Fast x86 FP compares
  leaf	Leaf procedures optimizations
  aot	Usage of Ahead Of Time compiled code
  precomp	Precompile all methods before executing Main
  abcrem	Array bound checks removal
  ssapre	SSA based Partial Redundancy Elimination
  exception	Optimize exception catch blocks
  ssa	Use plain SSA form
  treeprop	Tree propagation
  sse2	SSE2 instructions on x86
  gshared	Share generics
  simd	Simd intrinsics

• Multithreading in .net is much easier than any other language. You even have the parrallel loop (like Parallel.For operations.

SO ! Here is the test…

I wrote these two sample programs :

One in C++ :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

#include <iostream>
using namespace std;
 
long stupidThing( long nb ) {
	long out = 1;
	while( nb > 0 )
		out *= nb--;
 
 
	return out;
}
 
int main() {
 
	long total = 0;
 
	for( int i = 0; i < 1000000; ++i )
		for( long l = 0; l < 100; ++l )
			total += stupidThing( l );
 
	cout << "Total : " << total << endl;
 
	return 0;
}

One in C# :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

using System;
 
namespace test {
        class Program {
                static long stupidThing( long nb ) {
                        long ret = 1;
                        while ( nb > 0 )
                                ret *= nb--;
                        return ret;
                }
 
                static void Main( string[] args ) {
 
                        long total = 0;
 
                        for ( int i = 0; i < 1000000; ++i )
                                for ( long l = 0; l < 100; ++l )
                                        total += stupidThing( l );
                        Console.WriteLine( "Total : {0}", total );
                }
        }
}

First of all, I open a shell in real-time priority, because I don’t want my other processses to mess with my tests :

1

# rtprio 99 bash

Then I compile the two programs :

1
2

# gmcs test.cs
# g++ -O4 test.cpp -o test

And then I launch my test :

On a 64 bits host :

1

Kernel : 2.6.9-023stab051.3-smp #1 SMP Wed Nov 4 18:36:34 MSK 2009 x86_64 x86_64 x86_64 GNU/Linux

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# time ./test ; time ./test ; time ./test ; time mono test.exe ; time mono test.exe ; time mono test.exe
Total : -6192109806162068864
 
real    0m12.433s
user    0m12.394s
sys     0m0.049s
Total : -6192109806162068864
 
real    0m12.415s
user    0m12.411s
sys     0m0.014s
Total : -6192109806162068864
 
real    0m12.430s
user    0m12.411s
sys     0m0.026s
Total : -6192109806162068864
 
real    0m10.311s
user    0m10.287s
sys     0m0.029s
Total : -6192109806162068864
 
real    0m10.254s
user    0m10.247s
sys     0m0.011s
Total : -6192109806162068864
 
real    0m10.250s
user    0m10.255s
sys     0m0.012s

C# clearly beats C++ here. Well

On a 32 bits host :

1

Kernel : 2.6.30-2-686 #1 SMP Fri Dec 4 00:53:20 UTC 2009 i686 GNU/Linux

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# time ./test ; time ./test ; time ./test ; time mono test.exe ; time mono test.exe ; time mono test.exe
Total : 100461056
 
real    1m10.927s
user    1m7.376s
sys     0m0.056s
Total : 100461056
 
real    1m12.590s
user    1m8.976s
sys     0m0.020s
Total : 100461056
 
real    1m13.279s
user    1m9.532s
sys     0m0.056s
Total : -6192109806162068864
 
real    2m22.492s
user    2m15.260s
sys     0m0.136s
Total : -6192109806162068864
 
real    2m23.002s
user    2m15.760s
sys     0m0.104s
Total : -6192109806162068864
 
real    2m25.102s
user    2m17.709s
sys     0m0.144s

C++ beats C# here, but in 32 bits C++ use other types whereas C# use the same. In C# long is always 64 bits, in C++ it can be 64 bits or 32 bits (depending on the current architecture).

This is just a very short post about this language I recently discovered : the D language. As described on its little website : “D is a systems programming language. Its focus is on combining the power and high performance of C and C++ with the programmer productivity of modern languages like Ruby and Python. Special attention is given to the needs of quality assurance, documentation, management, portability and reliability.”.

It has most of the features of the java and the C# (best language ever), but it is fully compiled (no JVM or JIT stuff).

You should have a quick loot of its features on the Digital mars website (they’re the who made it).

I wonder if this language really has a future. C# (with the .Net framework) compiles and runs on every platform (thanks to mono), even on the iPhone, so it only remove the burden of the libraries required to run .net apps.
So, will it really beats its performance ? Because in C# is very fast (my next post will bring a stupid C++/C# performance comparison), and in C# avoiding time or memory consuming mistakes or multi-threading our code is very easy.

If like me you’re wondering what exactly does SPSecurity.RunWithElevatedPrivileges, this is what I understand of it :

1
2
3
4
5

SPSecurity.RunWithElevatedPrivileges( () => {
    using( var newSite = new SPSite( site.Url ) ) {
        // your stuff
    }
});

Is equivalent to :

1
2
3

using( var newSite = new SPSite( site.Url, site.SystemAccount.Token ) ) {
    // your stuff
}

So if this is just to open an elevated privilege SPSite, it will be equivalent. But if you do some more complex things, as soon as you load a new SPSite it will open it with the original UserToken. And you lose…

When you choose to automatically generate a new version of your assembly at compilation, you have to set in the AssemblyInfo.cs file, this line :

1
2
3
4
5
6
7
8
9
10

// Version information for an assembly consists of the following four values:
//
//      Major Version
//      Minor Version 
//      Build Number
//      Revision
//
// You can specify all the values or you can default the Build and Revision Numbers 
// by using the '*' as shown below:
[assembly: AssemblyVersion("1.0.*")]

But what do these automatically generated numbers mean ? Well, chances are you don’t care. But one day after using this features for years I decided that I would.

• The build number is the number of days since 01/01/2000
• The revision number if the number of 2 seconds periods since 00:00 of this day

So… To get the compilation date of an assembly from its version, you would have to do :

1
2

var version = Assembly.GetExecutingAssembly().GetName().Version;
var date = new DateTime( 2000, 01, 01 ).AddDays( version.Build ).AddSeconds( version.Revision * 2 );

This feature is great if your need some atomic versionning of a particular library for instance. Each program using your library has its own version and you don’t risk to break working feature in future release of your library. And if you want to force the use of a newer version of a library within a particular application, you can use some assembly redirection.
But you should also note that if you deploy your assembly in the GAC (like it’s required to do with Sharepoint), you will have tons of “garbage” versions of your assembly. And there’s not “easy fix” for that.

When you are using threads on sharepoint, you should take care of some little things :

• The method used to launch the thread must have the “[STAThread]” attribute, it avoids some huge memory leaks.
• If you need to use an SPSite from the calling site, you should copy its URL and UserToken to build a new SPSite in the new thread. If you’re not using that “design pattern”, things will get really wrong.
• And of course you should dispose any sharepoint SPSite (SPWebs come from SPSite) allocated within the thread.

I met Gerôme Guillemin at work (well yes his personal page is totally outdated). He invented and supported this language, its surrounding development tools and some sample programs.

What is quite amazing with this script language is that it only requires a simple 450 KB exe file to run the FSBL scripts and all scripts perform very well (even with the OpenGL samples). The executed programs look like they are some native Win32 applications.

You can get the 3.32.2 version installer here.

The program is closed source but he’s ok to give the sources to people ready to get involved in the project.

By the way, Gerôme Guillemin is the first person I know to have played with the TI-74. I’ve spent a lot of time with this amazing single line calculator that my father owns.

I have played with Google Buzz on the web and on Google Maps 4.0. I’ve always been a huge fan of location based services, Google Maps w/Buzz is the ultimate one.

I like the world Google is building for us. There’s soo much data they allow us to access and build around everything.

Not related in anyway note : but well as nobody’s reading (even my girlfriend, isn’t that sad), I guess I can say useless things : I discovered the perfect “computer guy”’s girlfriend : Leah Culver. She is pretty, smart, loves to code and she even likes hamburgers. And she doesn’t seem disturbed by crappy cars.
There must be something wrong with this girl, maybe she’s a succubus from hell.
I reached my quota of useless links.

One or two weeks ago, I made a simple AB benchmarking test on a PHP site I built, it was ok. Then I did the same test on this blog and well… It was freaking slow… On 100 pages with 10 concurrent access, it took 3.5 to 10s to render. Well, I thought I should remove all these plugins I installed to make me and my blog famous (they didn’t perform well). It reduced the generation time by something like 100 ms.

So I had the brilliant idea to put everything in an APC cache, someone had the same idea.

So I approximatively did the same thing :

In the beginning of the index.php file :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

if ( $_SERVER['REQUEST_METHOD'] == 'GET' ) {
 
        // This key will identify the page used
        $cacheKey = 'WP_'.$_SERVER['HTTP_HOST'].md5( $_SERVER['REQUEST_URI'] );
 
        // We try to get this page
        $cacheData = apc_fetch($cacheKey);
 
        // If we got something
        if (is_array( $cacheData )) {
 
                // We give each header
                foreach( $cacheData[0] as $h )
                        header( $h );
 
                // And the content
                die($cacheData[1]);
        }
 
        // If we're still here, we have no cache, so we'll start caching right now
        ob_start();
}

In the end of the index.php file :

1
2
3

// We have a cache key, it means we have something to cache
if ( $cacheKey )
        apc_store( $cacheKey, array( headers_list(), ob_get_contents() ), 900 );

And once it’s cached it performs well, to make it short it can serve in mean time of 1 ms and gives a general bandwidth of 200 MB/s. As you can guess, these tests are made locally and this is normal.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

[root@s15342968 ~]# ab -n 100  "http://florent.clairambault.fr/"
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
 
Benchmarking florent.clairambault.fr (be patient).....done
 
 
Server Software:        Apache/2.2.8
Server Hostname:        florent.clairambault.fr
Server Port:            80
 
Document Path:          /
Document Length:        161165 bytes
 
Concurrency Level:      1
Time taken for tests:   0.119899 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      16156100 bytes
HTML transferred:       16116500 bytes
Requests per second:    834.04 [#/sec] (mean)
Time per request:       1.199 [ms] (mean)
Time per request:       1.199 [ms] (mean, across all concurrent requests)
Transfer rate:          131585.75 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     1    1   0.0      1       1
Waiting:        0    0   0.1      0       1
Total:          1    1   0.0      1       1
 
Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      1
  90%      1
  95%      1
  98%      1
  99%      1
 100%      1 (longest request)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

[root@s15342968 ~]# ab -n 100 -c 10  "http://florent.clairambault.fr/"
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
 
Benchmarking florent.clairambault.fr (be patient).....done
 
 
Server Software:        Apache/2.2.8
Server Hostname:        florent.clairambault.fr
Server Port:            80
 
Document Path:          /
Document Length:        161165 bytes
 
Concurrency Level:      10
Time taken for tests:   0.47897 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      16156100 bytes
HTML transferred:       16116500 bytes
Requests per second:    2087.81 [#/sec] (mean)
Time per request:       4.790 [ms] (mean)
Time per request:       0.479 [ms] (mean, across all concurrent requests)
Transfer rate:          329394.31 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     1    4   1.3      4      10
Waiting:        0    1   1.0      1       4
Total:          1    4   1.3      4      10
 
Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      5
  80%      5
  90%      6
  95%      6
  98%      8
  99%     10
 100%     10 (longest request)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

[root@s15342968 ~]# ab -n 10000 -c 10  "http://florent.clairambault.fr/"
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
 
Benchmarking florent.clairambault.fr (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Finished 10000 requests
 
 
Server Software:        Apache/2.2.8
Server Hostname:        florent.clairambault.fr
Server Port:            80
 
Document Path:          /
Document Length:        161165 bytes
 
Concurrency Level:      10
Time taken for tests:   7.898135 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1616094683 bytes
HTML transferred:       1612133495 bytes
Requests per second:    1266.12 [#/sec] (mean)
Time per request:       7.898 [ms] (mean)
Time per request:       0.790 [ms] (mean, across all concurrent requests)
Transfer rate:          199821.47 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.9      0       7
Processing:     2    6   1.6      7      13
Waiting:        0    1   1.4      1       8
Total:          2    7   1.5      7      16
 
Percentage of the requests served within a certain time (ms)
  50%      7
  66%      8
  75%      8
  80%      8
  90%      9
  95%     10
  98%     11
  99%     11
 100%     16 (longest request)

By the way, APC 3.0.x really has a deadlock bug. It totally crashes Apache making it accept but never reply connections. Everyone should update to APC 3.1.