Cassandra on droid.io

droid.io is an other travis clone.

I gave it a try to execute some automated tests on top of cassandra. Unfortunately it doesn’t support cassandra out of the box. But adding support for it is in fact quite easy:

Here is a script to load cassandra and wait for its startup:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
#!/bin/bash
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre
 
curl -LO http://archive.apache.org/dist/cassandra/2.0.5/apache-cassandra-2.0.5-bin.tar.gz
tar xf apache-cassandra-2.0.5-bin.tar.gz
ln -s apache-cassandra-2.0.5 cassandra
cd cassandra
 
sudo mkdir -p /var/lib/cassandra /var/log/cassandra
sudo chown `whoami` /var/lib/cassandra /var/log/cassandra
 
bin/cassandra
 
for i in {0..30}; do echo "Waiting server ($i)..." ; nc localhost 9042 </dev/null && exit 0 ; sleep 1; done;
 
exit 1

Let’s GO

Why should you spend time learning go ?

The language is great

  • Simple: Languages syntax is quite small and easy to learn. Complex features that other languages provide have been omitted: inheritance, generics and pointer arithmetic.
  • Elegant: No semicolons. The scope of variables has been thoughtfully designed, it’s a huge win for readability of code. I personally add (often useless) brackets in my C to make it more readable. The defer keyword allows some clean and simple resource freeing.
  • Easy to use: Having a garbage collector makes it easy to use.
  • Built for concurrency: Making concurrent code has never been so easy. All IO calls are blocking but in a go-routine kind of way, meaning that as soon an I/O or time operation occurs, the current running thread will execute an other go-routime. It’s the java’s Future mechanism made easy.
  • Comes batteries included: Like python, it has some great standard packages.
  • Go on C libs: cgo makes it simple to integrate C code and C libraries. But for all the I/O related task, go internal code will perform much better (it shares a threads).
  • Unicode support: It’s one thing you won’t have to worry about.

But it’s not just the language

It is given a complete environment. You can can type “go...

  • build: Builds your code. No use for a Makefile or complex stuff like that. “go build” takes care of everything.
  • run: Runs your code if you don’t want to build it.
  • test: Alows some very easy (and fast) unit testing of your code.
  • get: Fetches any kind of package.
  • fmt: Formats your code. You don’t have to define some code formatting rules. Your code is reformated following the go requirements.
  • doc: Documents your code. No doxygen or javadoc, go has it.
  • vet: It inspects your code to help you detect wrong behaviors.
  • fix: Fixes your code by replacing old APIs calls by new ones.
  • install: Installs your packages (libraries). Installing a package on a host is easy as hell.
  • tool: Contains a set of useful tools. Most important one in my opinion is the pprof CPU/memory profiler.

Still…

To be exhaustive, I’d like to mention few issues you can expect from this languages at this stage.

  • Huge binaries: Mostly because produced go binaries are almost completely static but not only, they are huge (like 2MB big hello world). The problem is known and likely to be reduced in the future.
  • No big GUI libraries: The language is still young (Go 1.0 was released in Mars 2012) and focus has been put on server side code. So there’s not much librairies or librairies environment built out there.
  • No real localization: In python, we usually use gettext with _('stuff') but here we don’t. In Go you have solutions, but none of them is provided by the standard packages at this stage.
  • You might be disturbed by Go’s interfaces: If you come from java with its interfaces inheritances and its powerful namespaces. You might be disturbed that some interfaces included other interfaces because they include their functions.

Can we bet on it ?

Is it matture enough ? Is is a real disruption ? Will the language actually take-off ?
It’s definitely a major evolution. Because you have many good and exclusive feature while sill having some good old C language bindings and a LOT of libraries have already been ported or binded on it. The "go get" command is also a great recipe for boosting community support.

Getting started

To get you started, I would recommend making the golang tour, then installing go and using a good IDE like LiteIDE or SublimeText + GoBlime.

If (or when should I say) you are hooked, read the effective go page. There are many things like the getter/setter conventions that you will only see here or in the thousands of open-source Go projects outhere.

Just as a quick note: It’s static language. If you come from PHP, ruby or python with little C/C++/java experience, you might find the variable scoping a little bit hard to understand at first.

If you want to start making webapps, I can recommend trying revel or martini (+ fresh for automatic reloading)

Links

I’ll try to update this page later.

To all my dear pure C developers

I’ve recently spent quite some time integrating a lot of C code to turn it into production-ready software. It is surprisingly interesting, but the first few days are usually quite painful for these reasons:

Logs

Many C developers have done hundreds of projects, they always need logging but when an error occurred they still like to do some standard fprintf(stderr, ... );. I use something similar to this. But you should at least insert the __FILE__ and __LINE__ macros. It will save lots of people hours of debugging.

Use the warnings

On gcc, you should always have -Wall -Werror enabled. This will avoid a lot of stupid and horrible things.

Error handling

Check the return codes, log the abnormal ones.
Sample code that can make you lose quite some time:

1
2
3
FILE * f = fopen("logs.txt","w");
if ( ! f )
   goto end;

THIS IS WRONG, we should have something like this:

1
LL_WARNING("Could not open log file !");

using this kind of macro.

Dirty numbers

You know how to count in bytes, how pointer and offsets work, that’s great. But you should also know that you can always replace this by a sizeof(). That way, we won’t have to be as smart as you to read your code (and fix it).

Great POSIX standard headers and libraries

The standard errors code already cover a lot of cases you could encounter.
You can use booleans including stdbool.h, it’s even standard.
Argument parsing is standard, don’t make your own strange argument parsing code that won’t even (fully) document.

Great non standard libraries

Like json-c to serialize your data, or zeromq to handle network communication.

Use a debugger

You might have forgotten it. But debugger will make you save a lot of time especially when they are well integrated in IDEs like netbeans. You can just immediately know why something crashed or why the programmed stalled for no apparent reasons.

Optimization

In most of the cases, your compiler will do a great job. Don’t use inline, prefetching or any other crazy optimization unless you really know what you do.

Don’t re-create unreadable network protocols

Use something simple readable like JSON over zeromq. It’s easier to create, debug, monitor. Please don’t create a new binary protocol.

Learn other languages

Oops ? Did I just say that. YEAH I DID. C is fine but in many cases you can use something else that would give you a huge productivity boost, it will probably run slower and produce bigger binaries but in most of the cases it really doesn’t matter. When making network servers, python, go, node.js (java in some ways) make the job a lot easier.

Bad memory management

Leaks and buffer overflow can happen. And so as many other issues out there. This is why you should use valgrind quite frequently.

In C, “it worked fine on my computer” is not a good enough excuse. There are many memory mistakes that you will miss on a 64bits Linux with 8GB of RAM and that you will definitely see on a 32bits Linux with 128MB of RAM.

Makefile

Use standard tools like autoconf, automake or create a simple and easy to maintain Makefile like this one.

Malloc is not your (only) friend

In most cases, allocating things on the stack is enough. A lot of developers seem to forget it. And it’s usually the same who forget to free their mallocated pointers.

It’s outside the scope of this rant but still:

IDEs

It often feels like a big waste. Most Linux developers I know are pretty fast at doing C programs and yet they can’t seem to appreciate the use of a good IDE like netbeans. At least you should use something like sublime which is not an IDE but still gives a productivity boost.

Cassandra CQL3 internal data structure

I’m a huge fan of cassandra, I’ve been playing with it since 0.7 and I’ve never stopped using it. It would say it’s most amazing features are: Always working and simple replication + predictable performances.

I was very happy when it went from a key-value store to a well structured database with CQL. With CQL you can focus on your data, and less on how you should organize your own structure to handle it properly. Still, behind the wheels, it works the same (it’s still a KV store). That’s why it’s very important to understand how the internal structure is done:

it’s not a perfect replacement yet. For example, you can’t get the collection elements timestamp (called writetime in CQL). “SELECT map['value'] FROM table;” doesn’t exist (not CQL compatible), so “SELECT writetime( map['value'] ) FROM table;” doesn’t either unfortunately.

This problem is known by Cassandra’s dev team but there’s indeed a syntax issue to solve first.

Pieter Hintjens is a smart guy

I’m very interested by everything around internet of things technologies (M2M). And the components I like to follow closely are the distributed databases which allow to make timeseries without single point of failures like cassandra and hbase and the message broker like RabbitMQ and ZeroMQ.

This little video isn’t really about ZeroMQ, it’s more about how build a community around a project. Personal creativity is great thing, we can all experience it some way or an other. Creating in a team gives you some new perspectives on how you can make better things (and, usually, be a better person). But creating as part of a community of people you don’t known and share your ideas/experiments/knownledges/insights with them is amazing. How to compile everyone’s mind into something better.
I’ve personnaly had the best fun building things that evolved with time, on a daily basis. You know the agile methodology when you use it for real.

My favorite part was:

We discovered, by trial and errors, – as always, it’s always by making mistakes and see what goes wrong – that smart people make big mistakes. And the smarter you are, the more you make big mistakes.

But the rest of the talk is also great. Around 32:00, he talks about how he prefers to merge bad code to let people engage with the community and become part of it then passing on them (and their dumb code). It’s definitely valuable to any kind of orgzniation trying to build communities around quality content. It’s always better to let people engage, even in a bad way, it makes other people engage as well and then you have to build the tools so that the outcome always is positive (you want a good community around/AND a good content in the end).

You can do any kind of project with any language

Our language helps us express our ideas and emotions. Computer science languages help us build things following our ideas.

There are hundreds of computer science languages currently available to help do pretty much any kind of work. They were all built because (at least) one person once thought “this language isn’t good enough”.

I often like to compare the capabilities of languages like C/C++, C#(.Net)/java, scala/python/ruby, javascript/coffeescript, etc. And I’m always amazed how the smallest changes that were brought to existing languages allowed to gain in efficiency and clarity.

And yet, I can still find people telling me: But you can do this in this language as well.

Deploying your maven web apps on glassfish with jenkins

Simple deployment

If your jenkins server is installed on the same host as the glassfish server it’s quite simple.
You just have to let maven do its thing and then add this shell command:

1
/usr/local/glassfish/bin/asadmin --echo=true --host=localhost --port=4848 --user=admin --passwordfile=/secure/place/for/passwords/domain1_password --secure=false deploy --force=true --name=myproject --contextroot=/myproject target/*.war

The /secure/place/for/passwords/domain1_password should contain this:

1
AS_ADMIN_PASSWORD=the_password

Deploying different versions

With parametrized build you can go a little further and decide, for instance, if you want to deploy the current version as the production one or not. You can switch the build as a parametrized one and add a “stable” parameter (with a default value of false) for example.

The the only thing you have to do is to switch the previous code to:

1
2
3
4
5
if $stable; then
    /usr/local/glassfish/bin/asadmin --echo=true --host=localhost --port=4848 --user=admin --passwordfile=/secure/place/for/passwords/domain1_password --secure=false deploy --force=true --name=myproject --contextroot=/myproject target/*.war
else
    /usr/local/glassfish/bin/asadmin --echo=true --host=localhost --port=4848 --user=admin --passwordfile=/secure/place/for/passwords/domain1_password --secure=false deploy --force=true --name=myproject-test --contextroot=/myproject-test target/*.war
fi

Using the jabber plugin

Then you can use the jabber plugin and you will have interesting conversations with your jenkins build system, like:
!build myproject

And when you feel you are ready to go live:
!build myproject stable=true

Notes

  • It’s even simpler with other servers like tomcat or jetty.
  • Glassfish is often considered as very heavy but as most of the OSGi based softwares, you can remove a lot of its parts and make it as light as tomcat.

Valgrind might be wrong

Sometimes valgrind can report the right problem with the wrong stack trace:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
make && sudo valgrind --db-attach=yes --leak-check=full --track-origins=yes --show-reachable=yes ./dist/Debug/cbadaemon 
==4423== Memcheck, a memory error detector
==4423== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==4423== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==4423== Command: ./dist/Debug/cbadaemon
==4423== 
==4423== Conditional jump or move depends on uninitialised value(s)
==4423==    at 0x4922C47: __pthread_mutex_unlock_usercnt (pthread_mutex_unlock.c:38)
==4423==    by 0x804E91A: ServerContext::ServerContext() (ServerContext.cpp:11)
==4423==    by 0x804EB92: ServerCore::ServerCore(int, char**) (ServerCore.cpp:15)
==4423==    by 0x804F627: main (main.cpp:57)
==4423==  Uninitialised value was created by a heap allocation
==4423==    at 0x48D273C: operator new(unsigned int) (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==4423==    by 0x804F60F: main (main.cpp:57)
==4423== 
==4423== 
==4423== ---- Attach to debugger ? --- [Return/N/n/Y/y/C/c] ---- y
1
2
3
4
5
6
7
(gdb) bt
#0  0x04922c47 in __pthread_mutex_unlock_usercnt (mutex=0x75e80d8, decr=1) at pthread_mutex_unlock.c:38
#1  0x0805d15b in RequestManager::RequestManager (this=0x75e8030) at src/transac/RequestManager.cpp:27
#2  0x0804e91b in ServerContext::ServerContext (this=0x75e802c) at src/server/ServerContext.cpp:11
#3  0x0804eb93 in ServerCore::ServerCore (this=0x75e8028, argc=1, argv=0xfef79b94) at src/server/ServerCore.cpp:15
#4  0x0804f628 in main (argc=1, argv=0xfef79b94) at src/main.cpp:57
(gdb) q

And it’s not an optimization problem as it was compiled with:

1
-c -m32 -g -rdynamic -O0 -fPIC -Wall -Werror

Plex

Plex is great piece of software. It’s a “complete media solution”. I installed it on my laptop (free) without editing a single file, I tested it on my iPad (it’s a paid app) and it was just working instantly. Then I also tested it on my Nexus 7 (paid app on play store as well) and it’s also working perfectly.

Then I discovered it can also do streaming through it’s web based interface, so it can stream to anything it doesn’t yet support. We could stop here but no: It can be installed on Debian stable hosts as well (which I love because they are easy to maintain [no daily updates and still pretty updated]).

As described here, the installation instruction for plex on Debian/stable is:

1
2
3
echo "deb http://shell.ninthgate.se/packages/debian squeeze main" | sudo tee -a /etc/apt/sources.list.d/plexmediaserver.list
wget http://shell.ninthgate.se/packages/shell-ninthgate-se-keyring.key -O - -q | sudo apt-key add -
sudo apt-get update && sudo apt-get install plexmediaserver