Tag Archives: java

Some good reasons for learning python

Python

President Obama thinks required programming language learning in high school is a great idea. So do I, and I think we should all start with python.

  • Writing code with it is very fast. When software engineers tell you “I can do it in 10 minutes”, in C/C++ they mean 4h, in java they mean 2h and in python they mean it.
  • You can really do anything, I’ve done some serial communication, bit level manipulation, network level event-based servers, multithreading, webservice providing and consuming, SQL and cassandra client faster than what I’ve been doing in any other language.
  • It’s easy to learn viagra pfizer 50 mg. You can start your first program right now and be good at it in 2 weeks.
  • It comes “batteries included”. You don’t have to install third-party libraries. Contrary to ruby, you don’t have to choose between the thousands of gems available, there’s almost always one official way to do things. Which leads to the next point:
  • It’s simple to read someone else’s code. This is because it’s high level language and you quickly know all the librairies.
  • It now has some IDEs. I know some people like to code in vi, but this is ugly and unproductive. Pydev is simple to install and supports a pretty good (or not so bad) auto-completion.

This leads me to two opposing ideas (but you’ll understand where I stand):

  • On a software architecture level, I think java (or C#) is the right choice for any complex or performance requiring system. IDE can really do there magic and most of the problems (mistakes, API change, etc.) are found at compilation.
  • But on a pragmatic/real-life level, I think python is especially relevant for companies (I’m really thinking about startups) who want to build and launch something quick, make it evolve easily and obviously don’t need performance. Your engineer brain might think “Yes, but it’s scripting, this sucks”. But who cares? In 3 years, your product will be probably obsolete, if not already dead and in noone’s hard-drive anyway.

If you feel python isn’t the right choice because you need to have a complex all-in-one-language architecture, you should have a look at message brokers. My favorite one is RabbitMQ. It works instantly (like any modern software should be), has client libraries in every language you can imagine and supports very interesting features like persistent queues, load balancing and replication. Load balancing means that if python happened to be a bottleneck in your system, you could just duplicate the instances and server two times more.

Technology brotherhoods


I often hear people talking about which technology is better between C / C++ / C# .Net / java. Most of the time, it’s more a political/brotherhood/community thing (like football, even thought frenchies like me aren’t so proud of their team now) than a technical talk.
I find it absurd. Computer science is about efficiency and making more money. You can take into account the pleasure you have to work on a technology (as you will be more likely to be more efficient with it), but it should still remain about efficiency.

The following video is about java & C# .Net. I’ve done quite a lot of C# .Net, Java and C, and some ObjectiveC (while developing iPhone applications) and a little bit of C++. The only thing I feel is that Sun with java had the good ideas, Microsoft with .Net took them and finished the work. Still, I often code in java, mostly for the TC65 chip, it’s quite comfortable. But for instance, it was longer to implement the M2MP protocol on the TC65 chip than the C# .net server. Mostly because of the unsigned types I had to “emulate” and the lack of generics (which are not available on the TC65’s JVM).

By the way, I’m running .Net apps on Linux everyday using Mono.

Here is the best example of what I’m talking about :

For all these lame people who don’t have an HTML5 compatible browser, you can go to the original video. The reason I downloaded it here in my blog is that you are forced to register to YouTube to watch it.

TC65 : Slow UDP reception

Receiving UDP datagrams on the TC65 is easy but inefficient. You need to create a new thread if you want to receive data asynchronously. There’s no way of knowing if a new datagram has been received or not without hanging on the ::receive( Datagram ); method. This is quite weird considering you can do it in TCP.

The real problem is it’s freaking slow. The Cinterion documentation just tells you “This method blocks until a datagram is received.”. What it doesn’t say is that nearly each time, it blocks for 100 to 700 ms after UDP datagrams have actually been received. And worse, this slowness/sleeping avoids the program from treating data and finally throws a little java.io.IOException exception : “No buffer space available”. I tried to give the udp receiving thread a higher priority than the other threads, I tried to make the TCP reception thread sleep a lot just in case it would lock some kind of network object, I looked on how could this method be implemented but couldn’t find a solution and don’t believe there’s one.

By the way, most of the codes on UDP data reception are lame. They all show the same stupid synchronous code. The real code you will need is something like that :

You can remove the Logger class calls. It enables me to make some build-specific (mostly logging) code. The “if” false conditions are remove by the java compiler (which is actually mostly a pre-compiler). I will (quickly this time) talk about this someday…

This program isn’t TC65 specific, it should work with any MIDP enabled J2ME device.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
package Network.UDPReceiver;
 
import Common.Logger;
import java.io.IOException;
import javax.microedition.io.Datagram;
import javax.microedition.io.DatagramConnection;
 
/**
 * UDP event-like receiving class
 * @author Florent Clairambault
 */
public class UDPReceiver implements Runnable {
 
    private IUDPReceive _receivingClass;
    private DatagramConnection _udpConn;
    private Datagram _udpDatagram;
    private Thread _thread;
    private boolean _loop;
 
    /**
     * Creates and launchs the receiver
     * @param conn The connection
     * @param receivingClass The receiving class of the UDP receive event
     */
    public UDPReceiver(DatagramConnection conn, IUDPReceive receivingClass) {
        _udpConn = conn;
        _receivingClass = receivingClass;
        init();
        start();
    }
 
    /**
     * Prepares the datagram used to receive data
     */
    private void init() {
        if (Logger.E_VERBOSE)
            Logger.Log( 32, "UDPReceiver.init();" );
        try {
            int maxLength = _udpConn.getMaximumLength();
            if (Logger.E_DEBUG) {
                Logger.Log( 39, "maxLength=" + maxLength );
            }
            _udpDatagram = _udpConn.newDatagram( maxLength );
        } catch (IOException ex) {
 
            // This should NEVER happen !
            if (Logger.E_CRITICAL)
                Logger.Log( 37, "UDPReceiver.init", ex );
        }
    }
 
    /**
     * The actual data reception
     * @return the data received
     * @throws java.io.IOException When something fais, it means we have to stop
     */
    public synchronized byte[] receiveUdpFrame() throws IOException {
        if (Logger.E_DEBUG)
            Logger.Log( 46, "UDPReceiver.receiveUdpFrame();" );
 
        _udpConn.receive( _udpDatagram );
        int size = _udpDatagram.getLength();
        if (Logger.E_DEBUG)
            Logger.Log( 49, "size = " + size );
 
        // We copy the data so that it can be used on an other thread
        byte[] data = new byte[size];
        System.arraycopy( _udpDatagram.getData(), 0, data, 0, size );
 
        // These two lines might seem weird but are the more efficient way
        // to prepare next datagram reception.
        _udpDatagram.reset();
        _udpDatagram.setLength( _udpDatagram.getData().length );
 
        if (Logger.E_DEBUG)
            Logger.Log( 63, "UDPReceiver.receiveUdpFrame : ok !" );
 
        return data;
    }
 
    /**
     * Starts the receiving thread
     */
    private void start() {
        _loop = true;
 
        _thread = new Thread( this, "udp" );
        _thread.start();
    }
 
    /**
     * Plan to stop
     * 
     * This method is in fact useless. The thread will stop when the
     * DatagramConnection (_udpConn) will be closed.
     */
    public void stop() {
        _loop = false;
    }
 
    /**
     * The thread method
     */
    public void run() {
        try {
            while (_loop) {
                if (Logger.E_DEBUG)
                    Logger.Log( 70, "UDPReceiver : running..." );
 
                byte[] data = receiveUdpFrame();
 
                // We "throw" an event-like method
                _receivingClass.UdpDataReceived( data );
            }
        } catch (IOException ex) {
            if (Logger.E_DEBUG)
                Logger.Log( 54, "UDPReceiver.run", ex );
 
            // The connection must have been closed, we have to stop !
            _loop = false;
        } catch (Exception ex) {
            if (Logger.E_CRITICAL)
                Logger.Log( 119, "UDPReceiver.run", ex );
        }
    }
}

The “mother” class has implements this IUDPReceiver interface :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
package Network.UDPReceiver;
 
/**
 * Interface an asynchronous UDP receiving class must implement
 * @author Florent Clairambault
 */
public interface IUDPReceive {
 
    /**
     * Called when a datagram is received on a connection
     * @param data Content of the datagram
     * 
     * We assume that the receiving class already knows the sender of the
     * datagram because it's a connection established with an "UDP server".
     */
    void UdpDataReceived( byte[] data );
}

The “mother” class should open the connection like that :

1
2
_udpConn = (UDPDatagramConnection) Connector.open( "datagram://" + server );
_udpReceiver = new UDPReceiver( _udpConn, this );

where server is an adresse like that “x.x.x.x:y”. It can also be used as a UDP server with “:y”.

And has to implement the interface IUDPReceiver interface.

Note on receiving UDP over GPRS

Most GPRS connections (every one of them unless you subscribe to special options), are “NATed” (using a NAT router). They are behind a private network (10.x.x.x/8, 192.168.x.x/16, 172.16.x.x/12). That prevents the equipments from being directly accessible (by other means than their phone number).

On TCP, you have to establish a connection with a remote host (that you’re quite likely to call a “server”), and then the server can answer on the link. You have a reliable bi-directional stream you can use to transmit anything you’d like. You can keep the connection open running forever, you just have to send data or set the KeepAlive option. NAT routers break TCP connections when nothing has been transmitted on it for 1 hour to 4 days. This has even been considered like a “bug” by some people because the NAT router reach its memory limit or the max number of TCP connections it can translate (from private to public).
On the TC65, it means you can communicate over a TCP connections for months. You just have to make sure that it’s still alive by providing a little ping mechanism in your protocol (you can send pings every 15/30 minutes for instance).

On UDP, it’s roughly the same thing except there’s no connection. In fact, UDP is minimalist (it only has a 8 bytes header). You have no mechanism for connection establishment, you just throw your data (and by chance, it will be received). The router can’t possibly know when the transmission has ended. So when the router supports bi-directional UDP, it just tries to keep track of the last connections established for a limited time. My personal tomato firmware enabled router, keeps track of the connections when you sent data from LAN to WAN in the last 30 seconds, and that goes to 180 seconds when the other side replies.

So, basically what you need to do to set up a bi-directionnal UDP connection is sending some data from the equipment to the server each 25 seconds. My guess is that even empty UDP packets (8 bytes sized packet) should work.

The most simple/secure way to do this (don’t forget that UDP datagrams can be dropped on their way to the server), is to establish a communication with the server in TCP. Then, ask for a unique identifier (two bytes for instance) and then use it each time you send data on UDP.

If you want to send data between two NATed network, it’s also possible. This little RFC explains it very well. If this seems a little bit to abstract, you should try out wireshark. I always use it when I have a doubts about some low-level network transmission (I like to investigate a lot, just to make sure I’m not missing anything).

I didn’t say it sucks

It didn’t say the TC65 sucks. It’s just not as efficient as I thought it could be. The network stack doesn’t seem to have been built for performances. If you send TCP data or receive UDP data too often, there’s a pretty good chance you will face the same problems. If you found a solution to bypass these problems, please tell me how.

But remember, if you only need to send some data each second (or less often), which is the case in most of the M2M applications, you can be sure the chip will suit your needs.