Florent Clairambault

This is now on the wiki.

On my spare time, I manage a handful of servers. And even if it’s not really my job, I try to do it well and efficiently. All of them work on Debian because it’s simple to manage. I started using cron-apt a few years ago. I started by upgrading everything automatically, this was a big mistake. I switched to only sending mails on available upgrades and doing the upgrade manually. But this is also quite painful because 95% of the time, it consists in typing “apt-get dist-upgrade -y” and waiting and I have lots more interestings things to do.

So here is my cron-apt configuration, I like it a lot:

In /etc/apt:
- I removed the sources.list file
- I put the content of my sources.list into sources.list.d/main.list, it should look something like that:

1
2

deb http://http.us.debian.org/debian stable main contrib non-free
deb-src http://http.us.debian.org/debian stable main contrib non-free

- I created a directory sources.security.list.d
- I put the following content:

1
2

deb http://security.debian.org/ stable/updates main contrib non-free
deb-src http://security.debian.org/ stable/updates main contrib non-free

Then I added the repositories with packages I want to manually upgrade to /etc/apt/sources.list.d/ and the ones that I want to automatically upgrade (which means that they can’t require any user interaction) to /etc/apt/sources.security.list.d/.

The interesting part is here, in /etc/cron-apt/action.d, this what I have:

0-update

1
2

update -o quiet=2
update -o quiet=2 -o Dir::Etc::sourceparts=/etc/apt/sources.security.list.d -o Dir::State::lists="security-lists"

We launch an update of the two kinds of repositories. For the sources.security.list.d one, we use also a different Dir::State::lists parameter (which is the directory the cache file) so that we don’t to re-download the content of the index files every time.

2-install-security

1

dist-upgrade -y -o quiet=1 -o Dir::Etc::sourceparts=/etc/apt/sources.security.list.d -o Dir::State::lists="security-lists" -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold"

For the –force-conf* options, I found the solution on Raphaël Hertzog’s blog.

We launch the upgrade (dist-upgrade actually) only on the repositories defined in /etc/apt/sources.security.list.d.

3-download

1

dist-upgrade -d -y -o APT::Get::Show-Upgraded=true

Then we only download files for the upgrade of the non-security packets.

6-clean

1

autoclean -y

And we finally delete all the old packets (the ones that will never be used).

If you want to play with the apt settings yourself, you should use apt-config to see what can change to fit your needs.

This made me save a lot of time. Because Debian produces quite a lot of security updates. Here is the frequency of the updates for one of my servers:

I’ve been testing btrfs for some months now. One of the most interesting features of this file-system is its snapshoting capabilities. Before that I was using rsnapshot. The issue with rsnapshot is that its lowest atomic level for snapshotting is the files themselves using hard-links. So any database table where one row is changed is copied completely. Btrfs as you might guess will only copy the modified chunks (I don’t know the atomicity of them [but who cares?]).

Here is a simple I’ve been using during these last months to backup my laptop and (remotely hosted) servers.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139

#!/bin/sh
 
# === ARGUMENTS PARSING ===
 
# We don't want to define a default period
PERIOD=
 
while echo $1 | grep ^- > /dev/null; do
 
    if [ "$1" = "--daily" ]; then
        PERIOD=daily
    fi
 
    if [ "$1" = "--monthly" ]; then
        PERIOD=monthly
    fi
 
    if [ "$1" = "--period" ]; then
        PERIOD=$2
        shift
    fi
 
    shift
done
 
if [ "${PERIOD}" = "" ]; then
        echo "You have to define a period  with the --period arg !" >&2
        exit 1
fi
 
# === END OF ARGUMENTS PARSING ===
 
# === PARAMETERS ===
 
# * Device we will use
DISK=/mnt/externe3
 
# * Subvolume used for the backup
SUBVOLUME=${DISK}/servers-backup
 
# * Current date (you could limit the date to +%Y-%m-%d)
DATE=`/bin/date +%Y-%m-%d_%H-%M-%S`
 
# * snapshot directory that will be used
SNAPDIR=${DISK}/snap/servers-backup
 
# * snapshot volume that will be used
SNAPVOL=${SNAPDIR}/${PERIOD}-${DATE}
 
# * max days to keep daily backups
MAX_DAYLY=60
 
# * max days to keep monthly backups
MAX_MONTHLY=365
 
# * Alert limit
LIMIT_ALERT=95
 
# * High limit
LIMIT_HIGH=90
 
# * Low limit
LIMIT_LOW=85
 
# === END OF PARAMETERS ===
 
# We get the space used over the total allocated space and the total percentage use.
# This is NOT the device total size but it's a lot more reliable than "df -h"
DISK_USED=`/sbin/btrfs filesystem df ${DISK}|grep Data|grep -Po "used=([0-9]*)"|cut -d= -f2`
DISK_TOTAL=`/sbin/btrfs filesystem df ${DISK}|grep Data|grep -Po "total=([0-9]*)"|cut -d= -f2`
DISK_PERC=`echo 100*${DISK_USED}/${DISK_TOTAL}|bc`
 
# We create the snapshot dir if it doesn't exist
if [ ! -d ${SNAPDIR} ]; then
        mkdir -p ${SNAPDIR}
fi
 
cd ${SNAPDIR}
 
# If we are over the low free space limit,
# we delete two days of daily backup.
if [ $DISK_PERC -gt $LIMIT_LOW ]; then
        echo "LOW LIMIT reached: $DISK_PERC > $LIMIT_LOW : Deleting 2 days" >&2
 
        OLDEST_FILES=`ls --sort=time -r|grep "daily-.*"|head -2`
        for file in $OLDEST_FILES; do
                /sbin/btrfs subvolume delete $file;
        done
 
fi
 
# If we are over the high free space limit,
# we delete a month of monthly backup
if [ $DISK_PERC -gt $LIMIT_HIGH ]; then
        echo "HIGH LIMIT reached: $DISK_PERC > $LIMIT_HIGH : Deleting 1 month" >&2
 
        OLDEST_FILES=`ls --sort=time -r|grep "monthly-.*"|head -1`
        for file in $OLDEST_FILES; do
                /sbin/btrfs subvolume delete $file;
        done
 
fi
 
# If we are over the alert free space limit,
# we delete the first two oldest files we can find
if [ $DISK_PERC -gt $LIMIT_ALERT ]; then
        echo "ALERT LIMIT reached: $DISK_PERC > $LIMIT_ALERT : Deleting the 2 oldest" >&2
 
        OLDEST_FILES=`ls --sort=time -r|head -2`
        for file in $OLDEST_FILES; do
                /sbin/btrfs subvolume delete $file;
        done
fi
 
 
# We touch the subvolume to change the modification date
touch ${SUBVOLUME}
 
# We do a snapshot of the subvolume
if [ ! -d "${SNAPVOL}" ]; then
        /sbin/btrfs subvolume snapshot ${SUBVOLUME} ${SNAPVOL}
fi
 
# We delete the backups older than MAX_DAYLY
find ${SNAPDIR} -mindepth 1 -maxdepth 1 -mtime +${MAX_DAYLY} -name "daily-*" -exec /sbin/btrfs subvolume delete {} \;
 
# We delete the backups older than MAX_MONTHLY
find ${SNAPDIR} -mindepth 1 -maxdepth 1 -mtime +${MAX_MONTHLY} -name "monthly-*" -exec /sbin/btrfs subvolume delete {} \;
 
 
# This is the actual backup code
# You need to save your data into the ${SUBVOLUME} directory
 
# We will only do the actual backup for the daily task
if [ "${PERIOD}" = "daily" ]; then
 
rsync -auv /usr/local/bin ${SUBVOLUME}/localhost/usr/local
 
fi

Then this is how you can use it by adding these cron-tasks :

1
2

0 12 * * *  user /usr/local/bin/backup-servers --period daily   >/var/log/backup-servers-daily.log
55 10 1 * * user /usr/local/bin/backup-servers --period monthly >/var/log/backup-servers-monthly.log

Debian released a new version of their system. I updated it on the server that powers this blog, it took me something like one hour to do the whole system upgrade. There was only a little glitch with mysql’s my.cnf file that had an unsupported “skip-bdb” line. Everything else went fine…

The very good thing in this new release is the new kfreebsd version (available in i386 and x86_64). It brings the power of the FreeBSD kernel to the great Debian OS. If you don’t see the point, read this. To put in a nutshell: a more stable kernel with less legal issues, better vendors support and the same softwares.

XRDP is a very interesting project. It allows to create a terminal server on Linux hosts using the RDP protocol. I think the RDP protocol is a great protocol (better than VNC) but that’s not why I’m writing this. The important thing about this tool is: It’s freaking simple to use.

On Debian, just type:

apt-get install xrdp -y

On Red Hat Enterprise Linux, just type:

yum install xrdp

Then, you have to start it:

/etc/init.d/xrdp start

And this is it, you can use it. You can connect dozens of clients without any difficulty.

Most of the projects depend on a single man. The man who gave this project a rebirth is Jay Sorg. He seems to be an expert of RDP and I can clearly understand why he got so interested by the protocol, just look at the its features.
Under the hood, it uses VNC (Xvnc) to actually get something from it.

“Ok, but I can do that with VNC.”. Well no, RDP allows you to :

• Setup a terminal server very quickly
• Choose the size of the terminal (640×480, 800×600, full screen, etc.)
• Use an encrypted connection (still without any setup, it’s even done without your knowledge)
• Use only one port (3389) for any number of simultaneous sessions

There’s only one pretty big limitation: It looks really ugly with KDE.

You should maybe change some of the settings, I personally changed the /etc/xrdp/sesman.ini file “Sessions” section to :

[Sessions]
MaxSessions=100
KillDisconnected=0
IdleTimeLimit=0
DisconnectedTimeLimit=3600

Here is a quick look of what happens in a 640×480 remote desktop connection login:

I often hear people talking about which technology is better between C / C++ / C# .Net / java. Most of the time, it’s more a political/brotherhood/community thing (like football, even thought frenchies like me aren’t so proud of their team now) than a technical talk.
I find it absurd. Computer science is about efficiency and making more money. You can take into account the pleasure you have to work on a technology (as you will be more likely to be more efficient with it), but it should still remain about efficiency.

The following video is about java & C# .Net. I’ve done quite a lot of C# .Net, Java and C, and some ObjectiveC (while developing iPhone applications) and a little bit of C++. The only thing I feel is that Sun with java had the good ideas, Microsoft with .Net took them and finished the work. Still, I often code in java, mostly for the TC65 chip, it’s quite comfortable. But for instance, it was longer to implement the M2MP protocol on the TC65 chip than the C# .net server. Mostly because of the unsigned types I had to “emulate” and the lack of generics (which are not available on the TC65′s JVM).

By the way, I’m running .Net apps on Linux everyday using Mono.

Here is the best example of what I’m talking about :

For all these lame people who don’t have an HTML5 compatible browser, you can go to the original video. The reason I downloaded it here in my blog is that you are forced to register to YouTube to watch it.

I find the concept of prioritization very interesting. It just enables you to do more with less. Doesn’t that sound great ?

Let’s say you want to be able to respond to user requests as fast as possible but update your data in a low priority manner :
You can set the process CPU priority from -20 (high priority) to 19 (low priority) by using the command :

1

nice -n <priority> <command>

You can set the process IO priority in 4 classes (0: none, 1: realtime, 2: best-effort, 3: idle) with some priorities within these classes (0-7, lower being higher prio). But you have to enable the CFQ (Complete Fair Queueing) scheduler first by typing something like that :

1

echo cfq >/sys/block/hda/queue/scheduler

So the ultimate low priority command will be

1

ionice -c 3 nice -n 20 <command>

But sometimes changing the CPU and IO priority won’t change much because the problem you might have might occur within the SQL server for say. So what you do ? Well, you could slow down your low priority program. If you have a low priority php script, you could do it like that :

1
2
3
4
5
6
7

<?php
while( true ) {
    list( $load ) = explode(' ', file_get_contents('/proc/loadavg') );
    echo 'sleeping '.$load."s\n";
    usleep( $load * 1000000);
}
?>

This program will make a loop that will slow down with the increasing load. It means that this program will always keep space of the system, even if it’s run hundreads of time.

So yeah, you can now manage efficiently CPU and disk. Are you done ? Not really, there’s still the memory issue. Memory (RAM) is always fast unless there’s no memory left, then it’s paged and everything becomes thousand times slower. You can only disable the virtual memory (swap on Linux), set some memory allocation limits (with ulimit), but that’s pretty much it.
I would personnally recommend to disable swap and always take more RAM than needed. On servers swap prevent them from having to kill process, but they are so much slowed by it that the whole system is slowed down. And then even ssh server is so slow you have an ssh timeout before reaching the shell.

So, Linux has a great scheduling capacity. But what about NOT scheduling AT ALL ? Well, Linux is also very able to do that. You can put some process in real-time mode, these processes won’t be interupted by anything unless they are sleeping or waiting for an I/O event.
You can use the Real-Time (RT) mode using the rtprio command :

1

rtprio 99 <command>

I found out that there is real test than the little one i did here. In fact the main interest of the post is this spreadsheet :

I recently did a simple and stupid comparison test between C++ (g++) and C# .Net (mono).

My point here is that C# can actually be considered as a very fast language. It allows automatic hardware-specific optimization. The only real drawback you have in the .Net framework is the garbage collector. It prevents the C# .Net it from being a realtime language/framework. But does it really matter for you ?

The purpose of the following test is to show that C++ isn’t faster than C# “as is”. I KNOW C# can not be faster than C++ because C# can do few hardware specific optimizations when you can do all of them in C++, because all the systems calls have to be done in C/C++ and because there’s no such thing like Template and forced inlining in C#.

But if you’re searching for arguments to choose C# .Net over native C++, you should also consider these :

• C# is much more simple. No headers, the languages specifications are simple and clear
• You don’t have to manage the memory in .Net (no memory leak, optimal memory consumption, etc.)
• C# code is as portable as java, it doesn’t need to be manually ported to a new architecture like C++
• .Net don’t just crash, they always throw an exception and they only crash if nobody catches it
• Mono (and presumely the .Net framework) does some really interesting optimizations :

  1	# mono --list-opt

  I put some link so that you can learn about these really interesting optimization. This is like design patterns, you should really know these things.

  peephole	Peephole postpass
  branch	Branch optimizations
  inline	Inline method calls
  cfold	Constant folding
  consprop	Constant propagation
  copyprop	Copy propagation
  deadce	Dead code elimination
  linears	Linear scan global reg allocation
  cmov	Conditional moves
  shared	Emit per-domain code
  sched	Instruction scheduling
  intrins	Intrinsic method implementations
  tailc	Tail recursion and tail calls
  loop	Loop related optimizations
  fcmov	Fast x86 FP compares
  leaf	Leaf procedures optimizations
  aot	Usage of Ahead Of Time compiled code
  precomp	Precompile all methods before executing Main
  abcrem	Array bound checks removal
  ssapre	SSA based Partial Redundancy Elimination
  exception	Optimize exception catch blocks
  ssa	Use plain SSA form
  sse2	SSE2 instructions on x86
  gshared	Share generics
  simd	Simd intrinsics

• Multithreading in .net is much easier than any other language. You even have the parrallel loop operations, like Parallel.For.

SO ! Here is the test…

I wrote these two sample programs :

One in C++ :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

#include <iostream>
using namespace std;
 
long stupidThing( long nb ) {
	long out = 1;
	while( nb > 0 )
		out *= nb--;
 
 
	return out;
}
 
int main() {
 
	long total = 0;
 
	for( int i = 0; i < 1000000; ++i )
		for( long l = 0; l < 100; ++l )
			total += stupidThing( l );
 
	cout << "Total : " << total << endl;
 
	return 0;
}

One in C# :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

using System;
 
namespace test {
        class Program {
                static long stupidThing( long nb ) {
                        long ret = 1;
                        while ( nb > 0 )
                                ret *= nb--;
                        return ret;
                }
 
                static void Main( string[] args ) {
 
                        long total = 0;
 
                        for ( int i = 0; i < 1000000; ++i )
                                for ( long l = 0; l < 100; ++l )
                                        total += stupidThing( l );
                        Console.WriteLine( "Total : {0}", total );
                }
        }
}

First of all, I open a shell in real-time priority, because I don’t want my other processses to mess with my tests :

1

# rtprio 99 bash

Then I compile the two programs :

1
2

# gmcs test.cs
# g++ -O4 test.cpp -o test

And then I launch my test :

On a 64 bits host :

1

Kernel : 2.6.9-023stab051.3-smp #1 SMP Wed Nov 4 18:36:34 MSK 2009 x86_64 x86_64 x86_64 GNU/Linux

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# time ./test ; time ./test ; time ./test ; time mono test.exe ; time mono test.exe ; time mono test.exe
Total : -6192109806162068864
 
real    0m12.433s
user    0m12.394s
sys     0m0.049s
Total : -6192109806162068864
 
real    0m12.415s
user    0m12.411s
sys     0m0.014s
Total : -6192109806162068864
 
real    0m12.430s
user    0m12.411s
sys     0m0.026s
Total : -6192109806162068864
 
real    0m10.311s
user    0m10.287s
sys     0m0.029s
Total : -6192109806162068864
 
real    0m10.254s
user    0m10.247s
sys     0m0.011s
Total : -6192109806162068864
 
real    0m10.250s
user    0m10.255s
sys     0m0.012s

C# clearly beats C++ here. Well

On a 32 bits host :

1

Kernel : 2.6.30-2-686 #1 SMP Fri Dec 4 00:53:20 UTC 2009 i686 GNU/Linux

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# time ./test ; time ./test ; time ./test ; time mono test.exe ; time mono test.exe ; time mono test.exe
Total : 100461056
 
real    1m10.927s
user    1m7.376s
sys     0m0.056s
Total : 100461056
 
real    1m12.590s
user    1m8.976s
sys     0m0.020s
Total : 100461056
 
real    1m13.279s
user    1m9.532s
sys     0m0.056s
Total : -6192109806162068864
 
real    2m22.492s
user    2m15.260s
sys     0m0.136s
Total : -6192109806162068864
 
real    2m23.002s
user    2m15.760s
sys     0m0.104s
Total : -6192109806162068864
 
real    2m25.102s
user    2m17.709s
sys     0m0.144s

C++ beats C# here, but in 32 bits C++ use other types whereas C# use the same. In C# long is always 64 bits, in C++ it can be 64 bits or 32 bits (depending on the current architecture).

Sometimes, you just need to redirect a port on Linux. The solution I could find is to add an entry into xinetd. Here is a sample /etc/xinetd.d file I have, it just redirects the 587 (tcp) port to the 993 port of gmail’s servers. I have to do this because Virgin Mobile France blocks the 993 tcp port.
If you’re in the same situation, you can use my server to access you gmail IMAP access. You just have to set the server name to “webingenia.com” and the port to “587″.

1
2
3
4
5
6
7
8
9
10
11
12
13

service submission
{
        disable = no
        flags = REUSE
        port = 587
        socket_type = stream
        protocol = tcp
        user = root
        redirect = 74.125.45.109 993
        #server = /usr/bin/nc
        #serer_args = 74.125.45.109 993
        wait = no
}

I talked about this earlier.

The mono tools for Visual Studio is a commercial product that easily enables you to test and port your Windows applications to mono.

As I told before this is really a great idea. Because I think Visual Studio is the best IDE and Linux is the best server OS (in my opinion, it’s more efficient and more robust than Windows Server). So, I think it’s the perfect gateway between this two very different (but very close since Mono appeared) environments to achieve greatness.

And if you don’t want to buy it, well, you don’t have to. Mono is free and you can very easily test your Windows programs on it. You just have to install mono, copy your exe (plus its required DLLs) to the Linux host and type something like “mono SampleProgram.exe”.