Tag Archives: TC65

TC65 : Slow UDP reception

Receiving UDP datagrams on the TC65 is easy but inefficient. You need to create a new thread if you want to receive data asynchronously. There’s no way of knowing if a new datagram has been received or not without hanging on the ::receive( Datagram ); method. This is quite weird considering you can do it in TCP.

The real problem is it’s freaking slow. The Cinterion documentation just tells you “This method blocks until a datagram is received.”. What it doesn’t say is that nearly each time, it blocks for 100 to 700 ms after UDP datagrams have actually been received. And worse, this slowness/sleeping avoids the program from treating data and finally throws a little java.io.IOException exception : “No buffer space available”. I tried to give the udp receiving thread a higher priority than the other threads, I tried to make the TCP reception thread sleep a lot just in case it would lock some kind of network object, I looked on how could this method be implemented but couldn’t find a solution and don’t believe there’s one.

By the way, most of the codes on UDP data reception are lame. They all show the same stupid synchronous code. The real code you will need is something like that :

You can remove the Logger class calls. It enables me to make some build-specific (mostly logging) code. The “if” false conditions are remove by the java compiler (which is actually mostly a pre-compiler). I will (quickly this time) talk about this someday…

This program isn’t TC65 specific, it should work with any MIDP enabled J2ME device.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
package Network.UDPReceiver;
 
import Common.Logger;
import java.io.IOException;
import javax.microedition.io.Datagram;
import javax.microedition.io.DatagramConnection;
 
/**
 * UDP event-like receiving class
 * @author Florent Clairambault
 */
public class UDPReceiver implements Runnable {
 
    private IUDPReceive _receivingClass;
    private DatagramConnection _udpConn;
    private Datagram _udpDatagram;
    private Thread _thread;
    private boolean _loop;
 
    /**
     * Creates and launchs the receiver
     * @param conn The connection
     * @param receivingClass The receiving class of the UDP receive event
     */
    public UDPReceiver(DatagramConnection conn, IUDPReceive receivingClass) {
        _udpConn = conn;
        _receivingClass = receivingClass;
        init();
        start();
    }
 
    /**
     * Prepares the datagram used to receive data
     */
    private void init() {
        if (Logger.E_VERBOSE)
            Logger.Log( 32, "UDPReceiver.init();" );
        try {
            int maxLength = _udpConn.getMaximumLength();
            if (Logger.E_DEBUG) {
                Logger.Log( 39, "maxLength=" + maxLength );
            }
            _udpDatagram = _udpConn.newDatagram( maxLength );
        } catch (IOException ex) {
 
            // This should NEVER happen !
            if (Logger.E_CRITICAL)
                Logger.Log( 37, "UDPReceiver.init", ex );
        }
    }
 
    /**
     * The actual data reception
     * @return the data received
     * @throws java.io.IOException When something fais, it means we have to stop
     */
    public synchronized byte[] receiveUdpFrame() throws IOException {
        if (Logger.E_DEBUG)
            Logger.Log( 46, "UDPReceiver.receiveUdpFrame();" );
 
        _udpConn.receive( _udpDatagram );
        int size = _udpDatagram.getLength();
        if (Logger.E_DEBUG)
            Logger.Log( 49, "size = " + size );
 
        // We copy the data so that it can be used on an other thread
        byte[] data = new byte[size];
        System.arraycopy( _udpDatagram.getData(), 0, data, 0, size );
 
        // These two lines might seem weird but are the more efficient way
        // to prepare next datagram reception.
        _udpDatagram.reset();
        _udpDatagram.setLength( _udpDatagram.getData().length );
 
        if (Logger.E_DEBUG)
            Logger.Log( 63, "UDPReceiver.receiveUdpFrame : ok !" );
 
        return data;
    }
 
    /**
     * Starts the receiving thread
     */
    private void start() {
        _loop = true;
 
        _thread = new Thread( this, "udp" );
        _thread.start();
    }
 
    /**
     * Plan to stop
     * 
     * This method is in fact useless. The thread will stop when the
     * DatagramConnection (_udpConn) will be closed.
     */
    public void stop() {
        _loop = false;
    }
 
    /**
     * The thread method
     */
    public void run() {
        try {
            while (_loop) {
                if (Logger.E_DEBUG)
                    Logger.Log( 70, "UDPReceiver : running..." );
 
                byte[] data = receiveUdpFrame();
 
                // We "throw" an event-like method
                _receivingClass.UdpDataReceived( data );
            }
        } catch (IOException ex) {
            if (Logger.E_DEBUG)
                Logger.Log( 54, "UDPReceiver.run", ex );
 
            // The connection must have been closed, we have to stop !
            _loop = false;
        } catch (Exception ex) {
            if (Logger.E_CRITICAL)
                Logger.Log( 119, "UDPReceiver.run", ex );
        }
    }
}

The “mother” class has implements this IUDPReceiver interface :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
package Network.UDPReceiver;
 
/**
 * Interface an asynchronous UDP receiving class must implement
 * @author Florent Clairambault
 */
public interface IUDPReceive {
 
    /**
     * Called when a datagram is received on a connection
     * @param data Content of the datagram
     * 
     * We assume that the receiving class already knows the sender of the
     * datagram because it's a connection established with an "UDP server".
     */
    void UdpDataReceived( byte[] data );
}

The “mother” class should open the connection like that :

1
2
_udpConn = (UDPDatagramConnection) Connector.open( "datagram://" + server );
_udpReceiver = new UDPReceiver( _udpConn, this );

where server is an adresse like that “x.x.x.x:y”. It can also be used as a UDP server with “:y”.

And has to implement the interface IUDPReceiver interface.

Note on receiving UDP over GPRS

Most GPRS connections (every one of them unless you subscribe to special options), are “NATed” (using a NAT router). They are behind a private network (10.x.x.x/8, 192.168.x.x/16, 172.16.x.x/12). That prevents the equipments from being directly accessible (by other means than their phone number).

On TCP, you have to establish a connection with a remote host (that you’re quite likely to call a “server”), and then the server can answer on the link. You have a reliable bi-directional stream you can use to transmit anything you’d like. You can keep the connection open running forever, you just have to send data or set the KeepAlive option. NAT routers break TCP connections when nothing has been transmitted on it for 1 hour to 4 days. This has even been considered like a “bug” by some people because the NAT router reach its memory limit or the max number of TCP connections it can translate (from private to public).
On the TC65, it means you can communicate over a TCP connections for months. You just have to make sure that it’s still alive by providing a little ping mechanism in your protocol (you can send pings every 15/30 minutes for instance).

On UDP, it’s roughly the same thing except there’s no connection. In fact, UDP is minimalist (it only has a 8 bytes header). You have no mechanism for connection establishment, you just throw your data (and by chance, it will be received). The router can’t possibly know when the transmission has ended. So when the router supports bi-directional UDP, it just tries to keep track of the last connections established for a limited time. My personal tomato firmware enabled router, keeps track of the connections when you sent data from LAN to WAN in the last 30 seconds, and that goes to 180 seconds when the other side replies.

So, basically what you need to do to set up a bi-directionnal UDP connection is sending some data from the equipment to the server each 25 seconds. My guess is that even empty UDP packets (8 bytes sized packet) should work.

The most simple/secure way to do this (don’t forget that UDP datagrams can be dropped on their way to the server), is to establish a communication with the server in TCP. Then, ask for a unique identifier (two bytes for instance) and then use it each time you send data on UDP.

If you want to send data between two NATed network, it’s also possible. This little RFC explains it very well. If this seems a little bit to abstract, you should try out wireshark. I always use it when I have a doubts about some low-level network transmission (I like to investigate a lot, just to make sure I’m not missing anything).

I didn’t say it sucks

It didn’t say the TC65 sucks. It’s just not as efficient as I thought it could be. The network stack doesn’t seem to have been built for performances. If you send TCP data or receive UDP data too often, there’s a pretty good chance you will face the same problems. If you found a solution to bypass these problems, please tell me how.

But remember, if you only need to send some data each second (or less often), which is the case in most of the M2M applications, you can be sure the chip will suit your needs.

Cinterion TC65 development tools work on Windows 7

I’ve installed Windows 7 on my PC. I made an upgrade from vista. Everything was saved except the deploy/debug connection to the Cinterion’s SDK. But a little repair operation with the setup installer made it work. It’s pretty cool because when I switched from XP to Vista, well… I switched back to XP due to that problem.

22/12/09 update :
MES doesn’t work fine on any other system than XP. You should try the JObexFTP tool instead.

I know I’m talking a lot about the TC65 these last days, but I’m finishing a little program (on my spare time) on this chip for someone. This little project is interesting because it requires to be fully optimized to match the required goals of performance. It’s pretty new for me, because a second delay time was ok in my last projects. Here, data must be sent at 5Hz (each 200 ms) and if possible at 20Hz (each 50ms), and that’s why I talked about the TC65’s crappy TCP stack.

Don’t forget you can download the SDK here.
I don’t provide the SDK anymore, I had to remove it.

SMS OTAP program for the TC65 updated

WARNING: All the Cinterion related content from this blog will be removed to go to the javacint wiki soon. Please get used to going there.

Program updated !

As I told you in the comment where I released this TC65 SMS OTAP program, it didn’t support serial communication. Well, I have added this feature. The program is now able to directly send OTAP SMS, using a GSM modem.

The “config.bin” with your settings file isn’t compatible anymore.

Please tell me if it works for you and/or if you find bugs. You can also ask me questions about the TC65 if you need some help.

So, to use this program :
1. Extract at least SMSOTAP.exe from SMSOTAP.zip
2. Launch “SMSOTAP.exe”
3. Connect your TC65/XT65 modem to your serial port (or virtual serial port using USB interface)
4. Build a short message (take care about the class and the PID)
5. Click on the “Send” button

Note : When your type your message, the “\n” and “\r” are replaced by ‘\n’ and ‘\r’ chars. The carrier-return (CR) chars typed in the textbox are not sent. No other chars than ‘\n’,’\r’ are converted (but I might change that if you need it).

Note 2 : You should always remember that the “AT^SJOTAP” parameters always override the SMS ones. So, if you want to be sure you will always be able to remotely update your chip, you should reset all parameters by sending the “AT^SJOTAP=” command.

Note 3 : There’s a pretty good chance that this program works on a lot of other GSM modems than the Cinterion TC65 chip. It has never been tested on any other chips but it uses standard AT commands (you can see them in the window).

Cinterion TC65’s TCP stack limits

WARNING: All the Cinterion related content from this blog will be removed to go to the javacint wiki soon. Please get used to going there.

If you use the TC65 to transmit some frequent (with a transmission interval of less than 700 ms) data, you should know that the TC65’s has a poor TCP implementation considering the ACK management of sent messages.

When the TC65 sends a TCP packet, it waits for an ACK packet before sending any other packet. It means that if you have a delay 400 ms between the TC65 and your server (which is very quite common), the TC65 will wait for 800 ms (400 ms for the data packet from the TC65 to the server and 400 ms for the ACK packet to return) before sending any data.

I discovered it because I wanted to send real-time GPS tracking data, and at a rate of 20 positions per second it was way above the minimal delay of 800ms. My positions came by packet of 5 to 20.

I asked for help to the french Cinterion’s support with complete description of the problem, “AT^SCFG” complete view and some libpcap network captures. And they couldn’t give me any real solution to this problem.

I think this is a chosen restriction. The chip has a memory of 400 Kb, to send a packet before waiting the acknowledge of the previous packet, it would have to use a more complex TCP stack and to consume some RAM to store all the packets left to acknowledge. I might be wrong, there might be a way to activate this, but right now Cinterion isn’t able give me any answer.

The only solution I found is to send data from the TC65 using UDP and to receive data using TCP. And receiving in real-time (I mean short time) DOES work perfectly with the chip (no need for increasing the receive buffer). You just have to disable Nagle’s algorithm (TCP_NODELAY=TRUE) on the remote host (most probably the TCP server).

Why TC65 SMS OTAP software update is great

WARNING: All the Cinterion related content from this blog will be removed to go to the javacint wiki soon. Please get used to going there.

Update anything

Cinterion gives specifications on how to send SMS messages to launch a remote Over The Air Provisionning (OTAP) operation.

Recently, someone asked me : We have a little program on some TC65 chips that only send SMS, we would like to connect it to the software that you built. What would we have to do. Well that where the magic comes. You don’t have to touch any of the hardware. The only concrete thing you might have to do is enable GPRS on the sim cards of your M2M fleet.

Let’s say the current little program’s name is “little.jar” (+ its “little.jad”). You have to build a program called “little.jad” that :

  • Change the name of the starting program from a:/little.jad to a:/m2msoft.jad
  • launches a local OTAP operation (with the “AT^SJOTAP” command)

Then you publish it on your HTTP server, send an SMS to every TC65 chip with the address of your HTTP server, and HERE IT IS ! Your whole M2M equipments fleet is updated with your brand new software.

Note on the SMS

According to Cinterion’s specifications, it’s pretty easy to send the software update SMS. But, when the time comes where you actually have to do it, you might get stuck, because it has to be precisely forged. I built a little (english+french) program that enables me to send SMS update to any TC65 ship I like.

It looks like that :


I updated the program. More details here.

A little bit deeper

You should remember that whatever parameter is sent by SMS, it is overridden by the “AT^SJOTAP” setting. The only way to have a remote complete control over the chip is to send the “AT^SJOTAP=” command.

That means that if you want to just put your chips where they have to act and then do a little OTAP to install the first program on it, you have to set :

  • AT^SJOTAP
  • AT^SCFG=”Userware/Autostart”,””,”1″

Cinterion TC65 Chip

WARNING: All the Cinterion related content from this blog will be removed to go to the javacint wiki soon. Please get used to going there.

M2M projects are the type of project I like to work on. Connecting remote devices, making them work together, and controlling everything remotely are things i love !

I discovered the Cinterion TC65 (which belonged to Siemens then) 3 years ago. And it was a dream come true. I’ve been mostly working on it to build some powerfull GPS tracking software over GPRS. But I have also worked on some other projects making it communicate with some equipments using GPIO or serial port.

Some things you might like with the TC65 Chip :

  • Java Virtual Machine (J2ME)
  • Only requires a serial or a virtual serial link over a USB cable to program
  • You can easily build powerful, multi-threaded programs
  • Easy management of GPIO, SPI, I2C ports
  • Easy management of TCP/UDP connections
  • Quite powerful (400 KB of RAM, 1.7 MB of memory)
  • You can update your code remotely with at commands or specially formated SMS
  • It’s fast enough for most of your M2M usages (but it’s not well suited for sound or video transport)

I’m quite disappointed to see how only few people/companies are interested in this product. The product is great, developers tools are great and the over the air updating feature is great too ! I think it costs around 60/70 € ($95). One reason might be that the first version of the chip was pretty buggy. The only safe solution was to add a hardware watchdog.

There’s a pretty interesting feature that I would love to use : libraries that are downloaded and updated separately from the main program. But sadly, it isn’t compatible with some old v1 chips I still use.

Available TC65 software versions are :

  • TC65 v1 : First version. Very unstable. It could crash for days. It also happened that the software was suddenly deleted. Couldn’t get standard output to the virtual serial port on USB. Sometimes OTAP SMS could save the chip, sometimes we just had to wait for days before it decided to come back to life.
  • TC65 v2 : New hardware (HW v2). Seems more stable. Thought, I have one chip where it sometimes crash just 20 seconds after startup.
  • TC65 v3 : Can be upgraded from a software v2 (needs a HW v2). Offers a software watchdog.
  • TC65i v1 : Cinterion has bought it. Smaller, better power saving mode.

And for the XT65 chip :

  • XT65 v1 : TC65 v2 with GPS
  • XT65 v2 : TC65 v3 with GPS

Vendors selling their TC65 with a watchdog often don’t have one. They just rely on the software watchdog of the TC65 v3 chip.