Florent Clairambault

I find the concept of prioritization very interesting. It just enables you to do more with less. Doesn’t that sound great ?

Let’s say you want to be able to respond to user requests as fast as possible but update your data in a low priority manner :
You can set the process CPU priority from -20 (high priority) to 19 (low priority) by using the command :

1

nice -n <priority> <command>

You can set the process IO priority in 4 classes (0: none, 1: realtime, 2: best-effort, 3: idle) with some priorities within these classes (0-7, lower being higher prio). But you have to enable the CFQ (Complete Fair Queueing) scheduler first by typing something like that :

1

echo cfq >/sys/block/hda/queue/scheduler

So the ultimate low priority command will be

1

ionice -c 3 nice -n 20 <command>

But sometimes changing the CPU and IO priority won’t change much because the problem you might have might occur within the SQL server for say. So what you do ? Well, you could slow down your low priority program. If you have a low priority php script, you could do it like that :

1
2
3
4
5
6
7

<?php
while( true ) {
    list( $load ) = explode(' ', file_get_contents('/proc/loadavg') );
    echo 'sleeping '.$load."s\n";
    usleep( $load * 1000000);
}
?>

This program will make a loop that will slow down with the increasing load. It means that this program will always keep space of the system, even if it’s run hundreads of time.

So yeah, you can now manage efficiently CPU and disk. Are you done ? Not really, there’s still the memory issue. Memory (RAM) is always fast unless there’s no memory left, then it’s paged and everything becomes thousand times slower. You can only disable the virtual memory (swap on Linux), set some memory allocation limits (with ulimit), but that’s pretty much it.
I would personnally recommend to disable swap and always take more RAM than needed. On servers swap prevent them from having to kill process, but they are so much slowed by it that the whole system is slowed down. And then even ssh server is so slow you have an ssh timeout before reaching the shell.

So, Linux has a great scheduling capacity. But what about NOT scheduling ? Well, Linux is also great like with that. You can put some process in real-time mode, these processes won’t be interupted by anything unless they are sleeping or waiting for an I/O event.
You can use the Real-Time (RT) mode using the rtprio command :

1

rtprio 99 <command>

I recently did a simple and stupid comparison test between C++ (g++) and C# .Net (mono).

My point here is that C# can actually be considered as a very fast (as fast as C++) language. It allows automatic hardware-specific optimization.
Lot of people think it’s like a java, it has a virtual machine. But in fact it just compiles everything into native language when it accesses methods. This is the whole purpose of the JIT (Just Int Time compiler).

If you feel the JIT compiler is a possible reason of slowness (it slows down the program startup), you can pre-compile your assemblys (mono –aot on linux). If you feel the mono files are a drawback, you can bundle everything in one file using mkbundle. You can read more about this here.

The only real drawback you have in the .Net framework is the garbage collector. It prevents the C# .Net it from being a realtime language/framework. But in real life usage it doesn’t slow anything down.

The purpose of the following test is to show that C# isn’t slower than C++. But if you’re searching for arguments to choose C# .Net over native C++, you should also consider this :

• Mono (and presumely the .Net framework) does some really interesting optimizations :

  1	# mono --list-opt

  I put some link so that you can learn about these really interesting optimization. This is like design patterns, you should really know these things.

  peephole	Peephole postpass
  branch	Branch optimizations
  inline	Inline method calls
  cfold	Constant folding
  consprop	Constant propagation
  copyprop	Copy propagation
  deadce	Dead code elimination
  linears	Linear scan global reg allocation
  cmov	Conditional moves
  shared	Emit per-domain code
  sched	Instruction scheduling
  intrins	Intrinsic method implementations
  tailc	Tail recursion and tail calls
  loop	Loop related optimizations
  fcmov	Fast x86 FP compares
  leaf	Leaf procedures optimizations
  aot	Usage of Ahead Of Time compiled code
  precomp	Precompile all methods before executing Main
  abcrem	Array bound checks removal
  ssapre	SSA based Partial Redundancy Elimination
  exception	Optimize exception catch blocks
  ssa	Use plain SSA form
  treeprop	Tree propagation
  sse2	SSE2 instructions on x86
  gshared	Share generics
  simd	Simd intrinsics

• Multithreading in .net is much easier than any other language. You even have the parrallel loop (like Parallel.For operations.

SO ! Here is the test…

I wrote these two sample programs :

One in C++ :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

#include <iostream>
using namespace std;
 
long stupidThing( long nb ) {
	long out = 1;
	while( nb > 0 )
		out *= nb--;
 
 
	return out;
}
 
int main() {
 
	long total = 0;
 
	for( int i = 0; i < 1000000; ++i )
		for( long l = 0; l < 100; ++l )
			total += stupidThing( l );
 
	cout << "Total : " << total << endl;
 
	return 0;
}

One in C# :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

using System;
 
namespace test {
        class Program {
                static long stupidThing( long nb ) {
                        long ret = 1;
                        while ( nb > 0 )
                                ret *= nb--;
                        return ret;
                }
 
                static void Main( string[] args ) {
 
                        long total = 0;
 
                        for ( int i = 0; i < 1000000; ++i )
                                for ( long l = 0; l < 100; ++l )
                                        total += stupidThing( l );
                        Console.WriteLine( "Total : {0}", total );
                }
        }
}

First of all, I open a shell in real-time priority, because I don’t want my other processses to mess with my tests :

1

# rtprio 99 bash

Then I compile the two programs :

1
2

# gmcs test.cs
# g++ -O4 test.cpp -o test

And then I launch my test :

On a 64 bits host :

1

Kernel : 2.6.9-023stab051.3-smp #1 SMP Wed Nov 4 18:36:34 MSK 2009 x86_64 x86_64 x86_64 GNU/Linux

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# time ./test ; time ./test ; time ./test ; time mono test.exe ; time mono test.exe ; time mono test.exe
Total : -6192109806162068864
 
real    0m12.433s
user    0m12.394s
sys     0m0.049s
Total : -6192109806162068864
 
real    0m12.415s
user    0m12.411s
sys     0m0.014s
Total : -6192109806162068864
 
real    0m12.430s
user    0m12.411s
sys     0m0.026s
Total : -6192109806162068864
 
real    0m10.311s
user    0m10.287s
sys     0m0.029s
Total : -6192109806162068864
 
real    0m10.254s
user    0m10.247s
sys     0m0.011s
Total : -6192109806162068864
 
real    0m10.250s
user    0m10.255s
sys     0m0.012s

C# clearly beats C++ here. Well

On a 32 bits host :

1

Kernel : 2.6.30-2-686 #1 SMP Fri Dec 4 00:53:20 UTC 2009 i686 GNU/Linux

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# time ./test ; time ./test ; time ./test ; time mono test.exe ; time mono test.exe ; time mono test.exe
Total : 100461056
 
real    1m10.927s
user    1m7.376s
sys     0m0.056s
Total : 100461056
 
real    1m12.590s
user    1m8.976s
sys     0m0.020s
Total : 100461056
 
real    1m13.279s
user    1m9.532s
sys     0m0.056s
Total : -6192109806162068864
 
real    2m22.492s
user    2m15.260s
sys     0m0.136s
Total : -6192109806162068864
 
real    2m23.002s
user    2m15.760s
sys     0m0.104s
Total : -6192109806162068864
 
real    2m25.102s
user    2m17.709s
sys     0m0.144s

C++ beats C# here, but in 32 bits C++ use other types whereas C# use the same. In C# long is always 64 bits, in C++ it can be 64 bits or 32 bits (depending on the current architecture).

Sometimes, you just need to redirect a port on Linux. The solution I could find is to add an entry into xinetd. Here is a sample /etc/xinetd.d file I have, it just redirects the 587 (tcp) port to the 993 port of gmail’s servers. I have to do this because Virgin Mobile France blocks the 993 tcp port.
If you’re in the same situation, you can use my server to access you gmail IMAP access. You just have to set the server name to “webingenia.com” and the port to “587″.

1
2
3
4
5
6
7
8
9
10
11
12
13

service submission
{
        disable = no
        flags = REUSE
        port = 587
        socket_type = stream
        protocol = tcp
        user = root
        redirect = 74.125.45.109 993
        #server = /usr/bin/nc
        #serer_args = 74.125.45.109 993
        wait = no
}

I talked about this earlier.

The mono tools for Visual Studio is a commercial product that easily enables you to test and port your Windows applications to mono.

As I told before this is really a great idea. Because I think Visual Studio is the best IDE and Linux is the best server OS (in my opinion, it’s more efficient and more robust than Windows Server). So, I think it’s the perfect gateway between this two very different (but very close since Mono appeared) environments to achieve greatness.

And if you don’t want to buy it, well, you don’t have to. Mono is free and you can very easily test your Windows programs on it. You just have to install mono, copy your exe (plus its required DLLs) to the Linux host and type something like “mono SampleProgram.exe”.

Ricardo Schmidt made a great multi-platform TC65 control and file management tool called JOBextFTP. This can be used for anyone whose Module Exchange Suite (MES) doesn’t work or doesn’t work correctly.

The biggest effort is put on managing the OBEX file transmission protocol, but it also does offer some simple methods that can be integrated in your development process, like “turnOn”, “turnOff”, “getTime”, “runApp”.

I’m really sorry but I didn’t take the time to test it as I’m currently not making some TC65 development. But it looks very operational to me.

Maybe I should start a list of useful TC65 applications. I can already see this one to help you deploy programs and files locally and my SMSOTAP to help you deploy programs remotely.

Here is the email Ricardo Schmidt sent me :
I modified and formated some parts of the original mail.

If you encounter a “Transaction Check Error” on yum on a x64 system during an install, an update or an upgrade, you will find out that most of the time, you can’t remove the problematic packages. But it’s very likely that the problem comes from a i386 version of a package. The easiest way to proceed is just to remove the i386 version of each software or library as it appears on the Transaction Check Error.

I had a problem with postgresql-libs (i386) which was required by subversion (i386). I removed the postgresql-libs (and I had to remove subversion), but both of them were also in a x86_64 version, so it didn’t change anything. They were just garbage packages.

PHP 5.2 brings lots of little useful features and CentOS 5.3 comes with PHP 5.1. So most of my PHP apps failed. The easiest way to solve this is to :

• Edit /etc/yum.repos.d/CentOS-Testing.repo and put this :

  1 2 3 4 5 6

  [c5-testing] name=CentOS-5 Testing baseurl=http://dev.centos.org/centos/5/testing/$basearch/ enabled=1 gpgcheck=1 gpgkey=http://dev.centos.org/centos/RPM-GPG-KEY-CentOS-testing

• Launch :

  1	yum update php

Source :
http://www.freshblurbs.com/install-php-5-2-centos-5-2-using-yum

I switched from Debian to CentOS because I had the choice between an old 32 bits Debian 4.0 or a brand new 64 bits CentOS 5.3. And I have some scripts that use the great “start-stop-daemon” tool, which isn’t available on CentOS.

The easiest way to solve this problem is to get dpkg from Debian and then try to compile it. It’s likely that it will fail because libselinux (and it’s subsidiary library libsepol) won’t be registered in the pkgconfig dir. But we don’t really care as we only need start-stop-daemon, not dpkg.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

# wget http://ftp.de.debian.org/debian/pool/main/d/dpkg/dpkg_1.14.25.tar.gz
# tar -xf dpkg_1.14.25.tar.gz
# cd dpkg-1.14.25/
# ./configure >/dev/null
configure: WARNING: x86_64 not found in cputable
configure: WARNING: linux-gnu not found in ostable
Package libselinux was not found in the pkg-config search path.
Perhaps you should add the directory containing `libselinux.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libselinux' found
configure: WARNING: no curses library found
# make >/dev/null
processarc.o: In function `process_archive':
/root/dpkg-1.14.25/src/processarc.c:166: warning: the use of `tmpnam' is dangerous, better use `mkstemp'
archives.o: In function `tarobject':
/root/dpkg-1.14.25/src/archives.c:634: undefined reference to `matchpathcon'
/root/dpkg-1.14.25/src/archives.c:640: undefined reference to `setfscreatecon'
/root/dpkg-1.14.25/src/archives.c:802: undefined reference to `setfscreatecon'
/root/dpkg-1.14.25/src/archives.c:804: undefined reference to `freecon'
/root/dpkg-1.14.25/src/archives.c:826: undefined reference to `setfscreatecon'
/root/dpkg-1.14.25/src/archives.c:618: undefined reference to `is_selinux_enabled'
collect2: ld returned 1 exit status
make[2]: *** [dpkg] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
# cd utils
# make install
gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -DLOCALEDIR=\"/usr/local/share/locale\" -idirafter ../libcompat -I../lib    -g -O2 -MT start-stop-daemon.o -MD -MP -MF .deps/start-stop-daemon.Tpo -c -o start-stop-daemon.o start-stop-daemon.c
mv -f .deps/start-stop-daemon.Tpo .deps/start-stop-daemon.Po
gcc -std=gnu99  -g -O2  -Wl,-O1 -o start-stop-daemon start-stop-daemon.o ../libcompat/libcompat.a
make[1]: Entering directory `/root/dpkg-1.14.25/utils'
test -z "/usr/local/sbin" || /bin/mkdir -p "/usr/local/sbin"
  /usr/bin/install -c 'start-stop-daemon' '/usr/local/sbin/start-stop-daemon'
make[1]: Nothing to be done for `install-data-am'.
make[1]: Leaving directory `/root/dpkg-1.14.25/utils'

And you should have it !

I’ve switched from my two three years old dedicated servers to one brand new virtual server. Reasons are : These servers costed me too much and they were becoming old (risk of failure increases). It wasn’t worth it.

I spent last night doing that because I didn’t want to interrupt anybody using these servers.

My two servers were running some Debian and I’m now switching to a CentOS virtual server. I was a little bit worried at first that CentOS would have a crappy package management system, but its yum is in fact working the same way as Debian’s apt-get and OpenSuse’s zypper. The oool thing is that these three package management systems roughly work the same way : install , you don’t have to learn a new “ultimate” way to upgrade your software (like on FreeBSD). By the way, the faster package management system is yum, and the slowest one is zypper.

The biggest problem I had so far was to move all the databases. Previously, I was always using some Debian hosts, I was using exactly the same version of MySQL, so I just had to copy the MySQL datadir and logs files. Here, I had to export/import everything.

CentOS has an antique 1.2.4 version of Mono (when the current version is 2.4). I couldn’t manage to compile from the tarball but compiling from the SVN worked fine.

1
2
3
4
5
6
7
8
9

]# /usr/local/bin/mono --version
Mono JIT compiler version 2.5 (/trunk/mono r140917 Sat Aug 29 05:29:19 CEST 2009)
Copyright (C) 2002-2008 Novell, Inc and Contributors. www.mono-project.com
        TLS:           __thread
        GC:            Included Boehm (with typed GC and Parallel Mark)
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none

I talked some time ago about a library I made to take advantage of the kernel network events. I now release it and explain how to use it. It can help people to do some little network softwares without knowing where to start from.

I built it for network servers made to communicate with remotely connected embedded chips. I wanted to be able to always stay in touch with a huge number of chips without any real cost. So, my very personal goal was to built a server network layer for massive M2M applications.
I also made a little web server with it supporting Keep-Alive and partial file download (wit the “Range” header) and an other little library to send serialized objects.

I made this little network library to accomplish two main goals :

• Simplify network server development
• Be able to support a lot of connections

It is actually able to support a lot of connections : On a little Linux server using Mono (with 512 MB memory with swap deactivated), I easily managed to listen to 60 000 simultaneous connections without consuming more than 40% of the server’s memory.

And it allows to create network servers in only few line of code :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84

using System;
using System.Collections.Generic;
using System.Text;
using SoftIngenia.NetEventServer;
 
namespace TestServer {
 
	/// <summary>
	/// My little server
	/// </summary>
	class MyServer {
 
		public static String BytesToString( Byte[] data ) {
			var sb = new StringBuilder();
			sb.Append( String.Format( "[ {0} ] {{ ", data.Length ) );
			for ( int i = 0; i < data.Length; ++i )
				sb.Append( String.Format( " 0x{0:X02}", data[ i ] ) );
			sb.Append( " }" );
			return sb.ToString();
		}
 
		/// <summary>
		/// My server view of the client
		/// </summary>
		class MyClient {
			public MyClient( uint id ) {
				Id = id;
			}
 
			public uint Id { get; private set; }
 
			public int NbMessagesReceived { get; set; }
 
			public void Treat( byte[] data ) {
				Console.WriteLine( "{0}.Treat( {1} );", this, BytesToString( data ) );
				NbMessagesReceived++;
			}
 
			public override string ToString() {
				return String.Format( "Client{{Id={0}}}", Id );
			}
		}
 
		private readonly TcpEventServer _server;
		private readonly Dictionary<uint, MyClient> _clients = new Dictionary<uint, MyClient>();
 
		public MyServer( int portNumber ) {
			_server = new TcpEventServer( portNumber );
			_server.ClientConnected += server_ClientConnected;
			_server.ClientDisconnected += server_ClientDisconnected;
			_server.BinaryDataReceivedFromClient += server_BinaryDataReceivedFromClient;
		}
 
		public void StartListening() {
			_server.StartListening();
		}
 
		void server_BinaryDataReceivedFromClient( uint clientId, byte[] data ) {
			_clients[ clientId ].Treat( data );
		}
 
		void server_ClientDisconnected( uint clientId ) {
			Console.WriteLine( "Client {0} disconnected !", clientId );
			_clients.Remove( clientId );
		}
 
		void server_ClientConnected( uint clientId ) {
			Console.WriteLine( "Client {0} connected from {1} !", clientId, _server.RemoteEndPoint( clientId ) );
			_clients.Add( clientId, new MyClient( clientId ) );
		}
 
 
	}
 
	class Program {
		static void Main() {
			var myserver = new MyServer( 3000 );
			myserver.StartListening();
 
			Console.WriteLine( "Listening..." );
			Console.ReadLine();
		}
	}
}

This app launched gives you something like that :

1
2
3
4
5

Listening...
Client 1 connected from 127.0.0.1:53792 !
Client{Id=1}.Treat( [ 5 ] { 0x68 0x65 0x6C 0x6C 0x6F } ); // "hello"
Client{Id=1}.Treat( [ 2 ] { 0x0D 0x0A } ); // '<CR>' '<LF>'
Client 1 disconnected !

The library also enables you to receive data as text. You just have to subscribe to the “ReceivedLine” event. There’s no performance cost if you don’t subscribe to the event.

For network server, you still need to do some frame recognition. I usually instantiate a FrameParsing class into every client on the server side.

You can download the NetEventServer library with its XML and PDB files.