Free mobile

Je suis passé chez free mobile. Ce n’est pas vraiment le fait que j’économise 55% du prix de mon forfait actuel qui m’a motivé mais plutôt l’idée que je n’aurai plus (jamais) à surveiller ma consommation et que je n’aurai pas à prendre un téléphone fixe et recopier à la main (trop ringard) des numéros de téléphone à l’avenir.

Ma mère, elle, ne passera pas chez free mobile car elle serait peinée de participer au licenciement de milliers de télé-commerciaux chez les opérateurs de téléphonie mobile concurrents.

Pour l’instant pas de soucis. Tout fonctionne bien.

cron-apt and the perfect update system

On my spare time, I manage a handful of servers. And even if it’s not really my job, I try to do it well and efficiently. All of them work on Debian because it’s simple to manage. I started using cron-apt a few years ago. I started by upgrading everything automatically, this was a big mistake. I switched to only sending mails on available upgrades and doing the upgrade manually. But this is also quite painful because 95% of the time, it consists in typing “apt-get dist-upgrade -y” and waiting and I have lots more interestings things to do.

So here is my cron-apt configuration, I like it a lot:

In /etc/apt:
– I removed the sources.list file
– I put the content of my sources.list into sources.list.d/main.list, it should look something like that:

1
2
deb http://http.us.debian.org/debian stable main contrib non-free
deb-src http://http.us.debian.org/debian stable main contrib non-free

– I created a directory sources.security.list.d
– I put the following content:

1
2
deb http://security.debian.org/ stable/updates main contrib non-free
deb-src http://security.debian.org/ stable/updates main contrib non-free

Then I added the repositories with packages I want to manually upgrade to /etc/apt/sources.list.d/ and the ones that I want to automatically upgrade (which means that they can’t require any user interaction) to /etc/apt/sources.security.list.d/.

The interesting part is here, in /etc/cron-apt/action.d, this what I have:

0-update

1
2
update -o quiet=2
update -o quiet=2 -o Dir::Etc::sourceparts=/etc/apt/sources.security.list.d -o Dir::State::lists="security-lists"

We launch an update of the two kinds of repositories. For the sources.security.list.d one, we use also a different Dir::State::lists parameter (which is the directory the cache file) so that we don’t to re-download the content of the index files every time.

2-install-security

1
dist-upgrade -y -o quiet=1 -o Dir::Etc::sourceparts=/etc/apt/sources.security.list.d -o Dir::State::lists="security-lists" -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold"

For the –force-conf* options, I found the solution on Raphaël Hertzog’s blog.

We launch the upgrade (dist-upgrade actually) only on the repositories defined in /etc/apt/sources.security.list.d.

3-download

1
dist-upgrade -d -y -o APT::Get::Show-Upgraded=true

Then we only download files for the upgrade of the non-security packets.

6-clean

1
autoclean -y

And we finally delete all the old packets (the ones that will never be used).

If you want to play with the apt settings yourself, you should use apt-config to see what can change to fit your needs.

This made me save a lot of time. Because Debian produces quite a lot of security updates. Here is the frequency of the updates for one of my servers:

btrfs for a simple and powerful backup system

I’ve been testing btrfs for some months now. One of the most interesting features of this file-system is its snapshoting capabilities. Before that I was using rsnapshot. The issue with rsnapshot is that its lowest atomic level for snapshotting is the files themselves using hard-links. So any database table where one row is changed is copied completely. Btrfs as you might guess will only copy the modified chunks (I don’t know the atomicity of them [but who cares?]).

Here is a simple I’ve been using during these last months to backup my laptop and (remotely hosted) servers.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
#!/bin/sh
 
# === ARGUMENTS PARSING ===
 
# We don't want to define a default period
PERIOD=
 
while echo $1 | grep ^- > /dev/null; do
 
    if [ "$1" = "--daily" ]; then
        PERIOD=daily
    fi
 
    if [ "$1" = "--monthly" ]; then
        PERIOD=monthly
    fi
 
    if [ "$1" = "--period" ]; then
        PERIOD=$2
        shift
    fi
 
    shift
done
 
if [ "${PERIOD}" = "" ]; then
        echo "You have to define a period  with the --period arg !" >&2
        exit 1
fi
 
# === END OF ARGUMENTS PARSING ===
 
# === PARAMETERS ===
 
# * Device we will use
DISK=/mnt/externe3
 
# * Subvolume used for the backup
SUBVOLUME=${DISK}/servers-backup
 
# * Current date (you could limit the date to +%Y-%m-%d)
DATE=`/bin/date +%Y-%m-%d_%H-%M-%S`
 
# * snapshot directory that will be used
SNAPDIR=${DISK}/snap/servers-backup
 
# * snapshot volume that will be used
SNAPVOL=${SNAPDIR}/${PERIOD}-${DATE}
 
# * max days to keep daily backups
MAX_DAYLY=60
 
# * max days to keep monthly backups
MAX_MONTHLY=365
 
# * Alert limit
LIMIT_ALERT=95
 
# * High limit
LIMIT_HIGH=90
 
# * Low limit
LIMIT_LOW=85
 
# === END OF PARAMETERS ===
 
# We get the space used over the total allocated space and the total percentage use.
# This is NOT the device total size but it's a lot more reliable than "df -h"
DISK_USED=`/sbin/btrfs filesystem df ${DISK}|grep Data|grep -Po "used=([0-9]*)"|cut -d= -f2`
DISK_TOTAL=`/sbin/btrfs filesystem df ${DISK}|grep Data|grep -Po "total=([0-9]*)"|cut -d= -f2`
DISK_PERC=`echo 100*${DISK_USED}/${DISK_TOTAL}|bc`
 
# We create the snapshot dir if it doesn't exist
if [ ! -d ${SNAPDIR} ]; then
        mkdir -p ${SNAPDIR}
fi
 
cd ${SNAPDIR}
 
# If we are over the low free space limit,
# we delete two days of daily backup.
if [ $DISK_PERC -gt $LIMIT_LOW ]; then
        echo "LOW LIMIT reached: $DISK_PERC > $LIMIT_LOW : Deleting 2 days" >&2
 
        OLDEST_FILES=`ls --sort=time -r|grep "daily-.*"|head -2`
        for file in $OLDEST_FILES; do
                /sbin/btrfs subvolume delete $file;
        done
 
fi
 
# If we are over the high free space limit,
# we delete a month of monthly backup
if [ $DISK_PERC -gt $LIMIT_HIGH ]; then
        echo "HIGH LIMIT reached: $DISK_PERC > $LIMIT_HIGH : Deleting 1 month" >&2
 
        OLDEST_FILES=`ls --sort=time -r|grep "monthly-.*"|head -1`
        for file in $OLDEST_FILES; do
                /sbin/btrfs subvolume delete $file;
        done
 
fi
 
# If we are over the alert free space limit,
# we delete the first two oldest files we can find
if [ $DISK_PERC -gt $LIMIT_ALERT ]; then
        echo "ALERT LIMIT reached: $DISK_PERC > $LIMIT_ALERT : Deleting the 2 oldest" >&2
 
        OLDEST_FILES=`ls --sort=time -r|head -2`
        for file in $OLDEST_FILES; do
                /sbin/btrfs subvolume delete $file;
        done
fi
 
 
# We touch the subvolume to change the modification date
touch ${SUBVOLUME}
 
# We do a snapshot of the subvolume
if [ ! -d "${SNAPVOL}" ]; then
        /sbin/btrfs subvolume snapshot ${SUBVOLUME} ${SNAPVOL}
fi
 
# We delete the backups older than MAX_DAYLY
find ${SNAPDIR} -mindepth 1 -maxdepth 1 -mtime +${MAX_DAYLY} -name "daily-*" -exec /sbin/btrfs subvolume delete {} \;
 
# We delete the backups older than MAX_MONTHLY
find ${SNAPDIR} -mindepth 1 -maxdepth 1 -mtime +${MAX_MONTHLY} -name "monthly-*" -exec /sbin/btrfs subvolume delete {} \;
 
 
# This is the actual backup code
# You need to save your data into the ${SUBVOLUME} directory
 
# We will only do the actual backup for the daily task
if [ "${PERIOD}" = "daily" ]; then
 
rsync -auv --inplace /usr/local/bin ${SUBVOLUME}/localhost/usr/local
 
fi

Then this is how you can use it by adding these cron-tasks :

1
2
0 12 * * *  user /usr/local/bin/backup-servers --period daily   >/var/log/backup-servers-daily.log
55 10 1 * * user /usr/local/bin/backup-servers --period monthly >/var/log/backup-servers-monthly.log

Kindle: Sync is magic

I bought the Kindle. It might sound strange considering I already have an iPad, but I couldn’t help myself to buy a “e-ink” enabled device.

The user experience around the Kindle is OK but not great, because it’s not a touch screen. The real amazing thing is the sync between amazon, the kindle and all the other possible devices.

  • There’s the sync when you buy a book. It appears instantly on the device. This feeling is so great. You do the “one click buy” and plop, you can read it. I know you can download most of the ebooks for free. But it takes at least 10 times more time. And I think we should prefer to download things legally when it’s at a fair price and faster to download than illegally.
  • And there’s the sync between your devices: I can read the same book on my iPad (better at night), my iPhone (I always have it) or the kindle. And every time I open any of these devices, it tells me “You were last reading on page X on this device, do you want to go there ?”.

So thank you amazon for making a better world.

On a side node, you should know that you can’t cancel ebooks you bought on amazon, even with the “one click buy”. You have to send them a mail to ask for the cancellation of your order. But every time I contacted them they replied within 20 minutes, even at 2 AM.

TC65 Development document updated

Same title. I wish I was more original.

I’ve been missing writing on this blog. It’s very fulfilling to write stuff around subjects we love.

What’s new in this document ?

  • Netbeans 7.0 with the TC65i
  • The use of pre-processor
  • Small thougths around the EGS5.
  • Personnal advices on project/product management

This can be considered as a draft.
I’m waiting for your comments, even (or especially) negative ones, to help me improve this document.

same place to
<a href="http://www.webingenia.com/static/doc/TC65Dev.pdf" onclick="_gaq prix viagra 25 milligrams.push([‘_trackEvent’,’download’,’http://www.webingenia.com/static/doc/TC65Dev.pdf’]);” >[ DOWNLOAD IT ]

C++ optimization

I’ve been working in a company for two years now (which is the longest time I have ever worked in the same company). I wasn’t supposed to do a lot of C++ but this is what I ended up doing and on a calculation intensive software. I ended up having to learn a thing or two about improving performances.

I wrote these lines for new comers to the C++ performance optimizing problems.

Don’t optimize too soon

Don’t forget that prior optimization is the root of all evils. Get the logic right first.

If you are not quite experienced with C++ and you don’t have a lot of time, in many cases you will build faster program that you will build faster if you do it in other languages like java. The “C++ is the fastest language, I must use it” is in many case the best way to produce bullshit. It is way easier to build, debug and profile code for java or C# and the CPU or memory overhead of these languages isn’t relevant in most applications.

Read what is available

These are the most interesting pages on the subject, you should absolutely read them.

Get the right attitude

This is already described here. Basically, you have to test –> profile –> understand –> fix.

CPU profiling gives perfect results. Some people think it’s just a tool that gives some wrong measures because of its intrusive way of working, but gprof / ppprof profilers do some sampling profiling. So they actually give very very insightful (but not real CPU time) results.

Change your programming paradigm

Paradigm is great word. I recently discovered it (because I don’t know that many words) and I like it a lot. Mostly because it explains a lot of things (like failures which are very interesting).

Think like a CPU

If you need to achieve very good performances, you need to think like a CPU. And I mean at the lowest level of modern CPU. And then, once you think you found the perfect way to make things fast, you should TEST it. And you should test it in situations as closes as possible as reality. If you make a very simple use of memory, it could easily get optimized by the CPU scheduler (and cache) and then you could reach 20x times better performance than what you reach in reality.

Use memory in a smart way

If you need to read things sequencially, the best performances are reached with arrays. Mostly because the CPU can prefetch the array you are reading and cache it’s content. If you need to often move things around, you should use pointers because you don’t endup copying a lot of data. Don’t forget pointers consume a lot of data, and if they aren’t necessary you can use offsets.

Make it compiler friendly

Your C++ compiler will do as much as it can to optimize your code. Every time your do call on a virtual class, or a function’s pointer, you will prevent him from doing some optimization. So think carefully before you do them. Use templates as much as you can.

Google Performance Tools

Google-PerfTools is set of tools to create more robust and efficient applications.

The tcmalloc (TC stands for “thread caching”) library overloads the malloc/realloc/free functions and new/delete operators to make them use a virtual heap that garbage collects and recycles memory chunks. This avoids a lots of system requests. This also has a cost, it consumes a little bit more memory (but it’s so reasonnable that you won’t even notice it).
In our program that has its own memory garbage-collection mechanism (for the most important parts), we had an execution speed improvement of 30%.

A colleague (RON) did some comparative tests of memory allocation libraries on an other software. In our simple tests, tcmalloc outperformed all the others in terms of speed (we had a 40% speed improvement, so it’s pretty hard to beat) but also reduced the memory consumption by 25% when we didn’t see any change for with the other libraries.

The CPU and memory profiling tools are both very good. The CPU profiling and heap profiling can be enabled either by changing an environment variable either by enabling/disabling it at runtime. In my case, we have our own scripting language (I would have preferred we didn’t have it, wrapper are always a better option) so I added some functions to start and stop the profiling at any time.

The complete software is a static built so even call of our internal calculation libraries are inlined by the compiler. It makes a lot harder to clearly understand where we actually spent time. To prevent this from happening, you can use “__attribute__ (noline)” in your function signature. In our case, this allows to do a simple grep with the API namespace to get the percentage of time using in each function of the API.

The google performance library also has a heap / memory leak checking tool. And this one is also started by defining a environment variable.

By the way: Why is is so great to be able to heap-check, cpu-profile and/or heap-profile from anywhere? Because you can enable it in production and/or on end-user computers easily. And the most important thing: You have the same version everywhere. You don’t have to maintain different versions and provide it to your users.

Few things

I’d like to add few things:

In my tests, adding some memory prefetching code was very messsy and rarely gave some performance improvements. This is the kind of optimization that should be forgotten (at least on x86).

The compiler you will use changes a lot of things. MSVC, G++/Linux, G++/Cygwin/Windows have totally different performances for the same code.

By the way

All that stuff also applies to other languages. In Java or C#, profiling gets much simpler.
I tend to think that people who have a good understanding of C/C++ optimization are the most likely to make efficient software in other languages, because they all understand what could cost at a CPU/Memory level.

Mon iPad 2


Je suis allé samedi matin à 9h30 devant l’apple store d’Opéra et j’ai eu mon iPad 2 une heure plus tard. Ce qui m’a le plus surpris c’est l’attitude des vendeurs, on vous vend la cool/mac attitude en même temps que l’iPad.

J’ai pris une version Wifi donc pas de géoloc par GPS. L’iPad ne me géolocalisait pas du tout, c’est assez gênant pour les applis comme Allociné qui proposent des séances “à proximité”, ou google maps pour les commerces “à proximité”. Après une mise à jour de 4.3 vers 4.3.1 plus aucun soucis, j’étais localisé à 50m près. Le plus fort c’est que la géolocalisation évolue quand je bouge dans l’appart. Donc la géoloc par wifi bien que peu précise est tout à fait utilisable.

Pour ceux qui trouvent que l’iPad ne sert à rien, je peux juste vous présenter comment je vois les choses: Dans google docs je mets des dossiers contenant des articles autour de ce que j’ai lu ou qu’ils me restent à lire dans des dossiers que j’appelle “brain input”. L’iPad c’est pour moi le “mega brain input”, on consomme ses mails, des articles de divers sites, journaux et magazines. On reste à un stade de consommation, mais c’est dans des moments où on ne ferait sans doute rien de mieux et surtout c’est dans état de confort: tout est fluide, on accède à ce qu’on veut en quelques secondes.

Tout ça pour répondre à la question d’un amis: “Allez Flo dis nous concrètement ce que tu peux faire avec cet iPad que tu ne pouvais pas faire avec ton ordi (qui se situe à 1 mètre)”. Je peux faire quelques trucs en plus et beaucoup de choses en moins, mais je suis naturellement incité à le faire dans beaucoup plus de situations.

Et sinon pour apporter mon grain de sel à la soupe de commentaires:

Surprises positives
– Le son tout à fait correct. Mettre un peu de musique classique en lisant des articles est très agréable
– La smart cover: super pratique à la verticale, verouille l’iPad quand on la referme
– MobileRSS + “Read It Later” : MobileRSS est synchro avec Google Reader, on peut envoyer sur “Read It Later” les articles qu’on veut lire et ensuite les lire dans l’agréable interface de ReadItLater
– L’appli LePoint: La présentation des articles est très agréable et on peut télécharger tous les numéros de plus d’un mois.
– L’appli l’express: Une parfaite intégration des articles dans l’interface iPad (et dans les deux orientations)

Surprises négatives
– J’ai un peu de l’effet mura dont pas mal de gens parlent, j’ai râlé pendant 10 minutes jusqu’à ce qu’ils m’expliquent qu’ils ne le prenaient pas en compte, que c’était pour ma pomme jusqu’à nouvel ordre. C’est pour être honnête invisible au quotidien.
– Le poids: Au lit ça fait nettement plus lourd qu’un livre.
– J’ai eu le son qui s’est bloqué parce que j’avais mis le bouton de blocage de son en blocage d’orientation de l’écran.
– Beaucoup de livres sont absents des bibliothèques en lignes.
– J’ai acheté beaucoup d’applications (plus de 70€ en 3 jours). Les applications sont le coeur de l’iPad, donc on a envie de prendre les meilleures.

iOS 4.3 – iPhone 3G internet WiFi Sharing

I just tested the sharing of the iPhone 3G internet connection that comes with iOS 4.3. And it works great. I think Apple figured out that sharing 3G over WiFi was the main reason for jailbreaking iPhones.

Until now I was wondering if I would take the 3G or Wifi iPad 2. Now I’m pretty sure I will take the WiFi version. The good news here is also that Bouygues doesn’t seem to lock it (I accepted the mobile provider settings from the iTunes just after I updated the iPhone).

Sorry for the blog inactivity, I’m quite busy. And thank you for being so many (4000 /month in Analytics) to come here for reasons I can’t really explain (except I’m a very interesting person).

Debian 6.0

Debian released a new version of their system. I updated it on the server that powers this blog, it took me something like one hour to do the whole system upgrade. There was only a little glitch with mysql’s my.cnf file that had an unsupported “skip-bdb” line. Everything else went fine…

The very good thing in this new release is the new kfreebsd version (available in i386 and x86_64). It brings the power of the FreeBSD kernel to the great Debian OS. If you don’t see the point, read this. To put in a nutshell: a more stable kernel with less legal issues, better vendors support and the same softwares.