The M2MP protocol

December 12, 2009 — Florent Clairambault

Today, I’m going to talk about a little protocol that I’ve implemented in few softwares (on TC65, .Net client and .Net server). It won’t be very interesting for most of you (99.99%). But still, I wanted to write it because it has been very useful and it might prevent you from writing the next stupid M2M protocol specifications.

Motivations

I made this simple protocol to solve very simple and common problems on M2M equipments, we want to :

  • Transmit and receive in real-time
  • Transmit as few data as possible
  • Send and receive any kind of data

So the basic ideas are to :

  • Keep a TCP connection open all the time by sending regular keep-alive frames.
  • Define named channels that are only transmitted once
  • Send byte array, and two-dimensionnal byte arrays on these channels

We rely on the TCP packet checksum, packet ordering and every little things it does very well.

Basically, we send these kind of frames :

  • Identification frames
  • Channel name to id definition
  • Data transmission of single and two-dimentionnal byte arrays.

The protocol

Identification
The client ask for authentication :

1
2
3

[ 1 ] { 0x01 } // Identification header
[ 1 ] { 0x05 } // Size of the identification
[ 5 ] { 'h', 'e', 'l', 'l', 'o' } // Identifier

The server replies “ok” or “not ok”

1
2

[ 1 ] { 0x01 } // Identification request
[ 1 ] { 0x01 } // OK, 0x00 for not ok

Ping / Keep alives
Client sends a ping request :

1
2

[ 1 ] { 0x02 } // client ping request header
[ 1 ] { 0x15 } // the number (here 0x15) can be incremented or random

Server answers the ping request :

1
2

[ 1 ] { 0x02 } // client ping server response header
[ 1 ] { 0x15 } // where 0x15 is the number of the ping request

Server sends a ping request :

1
2

[ 1 ] { 0x03 } // server ping request header
[ 1 ] { 0x16 } // the number (here 0x16) can be incremented or random

Client answers the ping request :

1
2

[ 1 ] { 0x03 } // server ping client response header
[ 1 ] { 0x16 } // where 0x16 is the 0x16 number of the ping request

Defining a named channel
This applies to both client to server and server to client communcation.

1
2
3
4

[ 1 ] { 0x20 } // channel definition frame header
[ 1 ] { 0x0D } // where 13 is the size of the following message
[ 1 ] { 0x03 } // where 3 is the id of the channel
[ 12 ] { 'c', 'h', 'a', 'n', 'n', 'e', 'l', ' ', 'n', 'a', 'm', 'e' } // the name of the channel : "channel name"

Sending some data
This applies to both client to server and server to client communcation.

For 0 to 254 sized messages :

1
2
3
4

[ 1 ] { 0x21 } // one byte sized data transmission frame header
[ 1 ] { 0x06 } // size of the data
[ 1 ] { 0x03 } // where 3 is the id of the channel
[ 5 ] { 0x01, 0x02, 0x03, 0x04, 0x05 } // data transmitted on the channel

For 255 to 65534 sized data messages :

1
2
3
4

[ 1 ] { 0x41 } // two bytes sized data transmission frame header
[ 2 ] { 0x00, 0x06 } // size of the data
[ 1 ] { 0x03 } // where 3 is the id of the channel
[ 5 ] { 0x01, 0x02, 0x03, 0x04, 0x05 } // data transmitted on the channel

For 65 535 octets to 4 294 967 294 sized data messages :

1
2
3
4

[ 1 ] { 0x61 } // two bytes sized data transmission frame header
[ 4 ] { 0x00, 0x00, 0x00, 0x06 } // size of the data
[ 1 ] { 0x03 } // where 3 is the id of the channel
[ 5 ] { 0x01, 0x02, 0x03, 0x04, 0x05 } // data transmitted on the channel

For data arrray transmission, it’s the same except frame header are 0×22, 0×42, 0×62 instead of 0×21, 0×41, 0×61. If you transmit data array like this one Byte[][] data = { { 0×01, 0×02 }, { 0×03, 0×04, 0×05 }, { 0×06, 0×07, 0×08, 0×09 } } :

1
2
3
4
5
6
7
8
9

[ 1 ] { 0x22 } // one byte sized data transmission frame header
[ 1 ] { 0x0D } // size of the data
[ 1 ] { 0x03 } // where 3 is the id of the channel
[ 1 ] { 0x02 } // size of the first array
[ 2 ] { 0x01, 0x02 } 
[ 1 ] { 0x03 } // size of the second array
[ 3 ] { 0x03, 0x04, 0x05 } 
[ 1 ] { 0x04 } // size of the third array
[ 4 ] { 0x06, 0x07, 0x08, 0x09 }

For bigger arrays, you need to define :

1
2
3
4
5
6
7
8
9

[ 1 ] { 0x42 } // one byte sized data transmission frame header
[ 1 ] { 0x10 } // size of the data (16)
[ 1 ] { 0x03 } // where 3 is the id of the channel
[ 2 ] { 0x00, 0x02 } // size of the first array
[ 2 ] { 0x01, 0x02 } 
[ 2 ] { 0x00, 0x03 } // size of the second array
[ 3 ] { 0x03, 0x04, 0x05 } 
[ 2 ] { 0x00, 0x04 } // size of the third array
[ 4 ] { 0x06, 0x07, 0x08, 0x09 }

and so on :

1
2
3
4
5
6
7
8
9

[ 1 ] { 0x62 } // one byte sized data transmission frame header
[ 1 ] { 0x16 } // size of the data (22)
[ 1 ] { 0x03 } // where 3 is the id of the channel
[ 4 ] { 0x00, 0x00, 0x00, 0x02 } // size of the first array
[ 2 ] { 0x01, 0x02 } 
[ 4 ] { 0x00, 0x00, 0x00, 0x03 } // size of the second array
[ 3 ] { 0x03, 0x04, 0x05 } 
[ 4 ] { 0x00, 0x00, 0x00, 0x04 } // size of the third array
[ 4 ] { 0x06, 0x07, 0x08, 0x09 }

Sample data types transmitted

Strings
Easy… Just the bytes value of the string

Position
when moving :

1
2
3
4
5

[ 4 ] Timestamp : UInt32 (since 1970 or 2009 if you wish)
[ 4 ] Longitude : Float
[ 4 ] Latitude : Float
[ 2 ] Speed : UInt16
[ 2 ] Altitude : UInt16

= 16 bytes

when stopped :

1
2
3

[ 4 ] Timestamp : UInt32 (since 1970 or 2009 if you wish)
[ 4 ] Longitude : Float
[ 4 ] Latitude : Float

= 12 bytes

Battery
Percentage :

1

[ 1 ] Percentage (Byte)

Voltage :

1

[ 2 ] Voltage in mV (UInt16)

Sample communication

This is just to show how is working a sample communication

1
2
3
4

--> [] { 0x01, 0x04, 0x01, 0x02, 0x03, 0x04 } // identification frame
<-- [] { 0x01, 0x01 } // identification accepted
--> [] { 0x20, 0x08, 0x00, 'b', 'a', 't', 't', 'e', 'r', 'y' } // We define the channel "battery" with id 0x00
--> [] { 0x21, 0x02, 0x00, 70 } // We send 70 (meaning 70%) on the "battery" channel.

On top of that

On top of that, I added some settings management and capacities specifications.

Capacities
If the server send “?” on the “_cap” (like capacities) channel, the client replies on the same channel with an data array containing the capacities it has (like “positionTracking”, “smsSending”, “sms” ).

Settings management
The server can get and set some settings on the “_set” channel.
If it’s a two-dimentionnal array and the first element is “s” (like “set”) it means that it sets some settings, the next elements will be “setting1=value1″, “setting2=value2″, etc.
If it’s a two-dimentionnal array and the first element is “g” (like “get”) it means that it needs the client to report the value of some settings, like “setting1″, “setting2″
If it’s a single dimentionnal array that contains “ga” (like “get all”) it means that the server wants the client to report the value of every settings.

Gateway acting and sub equipements
Any equipment can act as a gateway by encapsulating data of an sub-equipment in a channel name like this “_eq/[sub equipment identifier]” but this has never been used for the moment.

Stupid protocols

I’ve seen a lot of protocols from big companies doing some really stupid things like :

  • Fixed size frames for not fixed size data.
  • 4 zero filled bytes preamble. Why the hell would we need preamble in a TCP connection ?
  • 8 bytes timestamp in millisecond when data has a 1 second precision.
  • 4 bytes integer for specifying some number that never exceed 32.
  • Checksums on top of the TCP checksumming mechanism.
  • Redundant data at the beginning and the end.
  • Disconnecting very frequently (TCP establishment + identification costs a lot).
Posted in English. Tags: , , , , . No Comments »

TC65 : Settings management

December 11, 2009 — Florent Clairambault

Someone recently asked me and the javacint group how we should handle settings. So I’ll give you two answers :

  • ejw’s reply : You should use some simple DataInputStream and DataOutputStream objects.
  • Mine : If you only store simple values (numbers and text), you should use a simple text file. It enables you to easily see and modify settings outside the TC65.

So, my little gift of today will be a simple settings management class. The idea is that this file is in a very simple format (that can be read in any PC) and it only stores settings that have changed. This is very important, it allows you to change the default behavior at next software update but also enable you to override some of the settings for each chip (like the last IMSI identifier of the SIM card).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227

package Common;
 
import java.io.*;
import java.util.*;
import javax.microedition.io.*;
import com.siemens.icm.io.file.*;
 
/**
 * Settings management class
 * @author Florent Clairambault
 */
public class Settings {
 
        // The cached settings
	private Hashtable _settings;
 
        // File of the settings
	private final String _fileName = "a:/settings.txt";
 
        // Singleton instance of the settings
        private static Settings _instance;
 
	/**
	 * Get Default instance of the Settings class
	 * @return The default instance of the Settings class
	 */
	public static synchronized Settings getInstance() {
		if ( _instance == null )
			_instance = new Settings();
		return _instance;
	}
 
	/**
	 * Free the singleton instance
	 */
	public static synchronized void freeInstance() {
		_instance = null;
	}
 
	/**
	 * Load settings
	 */
	public synchronized void Load() {
		StringBuffer buffer = new StringBuffer();
		Hashtable settings = getDefaultSettings();
		try {
 
 
			FileConnection fc = (FileConnection) Connector.open( "file:///" + _fileName, Connector.READ );
			InputStream is = fc.openInputStream();
 
			while ( is.available() > 0 ) {
				int c = is.read();
 
				if ( c == '\n' ) {
					Load_TreatLine( settings, buffer.toString() );
					buffer = new StringBuffer();
				} else
					buffer.append( (char) c );
			}
			is.close();
			fc.close();
 
		} catch ( IOException ex ) {
			// The exception we shoud have is at first launch : 
			// There shouldn't be any file to read from
 
			if ( Logger.E_CRITICAL )
				Logger.Log( 19, "Settings.Load", ex );
		}
		_settings = settings;
	}
 
	/**
	 * Treat each line of the file
	 * @param def Default settings
	 * @param line Line to parse
	 */
	private static void Load_TreatLine( Hashtable settings, String line ) {
		if ( Logger.E_VERBOSE )
			Logger.Log( 78, "Load_TreatLine( [...], \"" + line + "\" );" );
		String[] spl = Common.strSplit( '=', line );
		String key = spl[0];
		String value = spl[1];
 
		// If default settings hashTable contains this key
		// we can use this value
		if ( settings.containsKey( key ) ) {
			settings.remove( key );
			settings.put( key, value );
		}
 
	}
 
 
	/**
	 * Get default settings
	 * @return Default settings Hashtable
	 */
	private Hashtable getDefaultSettings() {
		Hashtable defaultSettings = new Hashtable();
 
		// General M2MSoft settings :
		defaultSettings.put( "code", "8888" );
		defaultSettings.put( "servers", "87.106.206.30:3000" );
		defaultSettings.put( "apn", "gprs,m2minternet,\"\",\"\",,0" );
		defaultSettings.put( "imsi", "0000" );
		defaultSettings.put( "watchdogtimer", "20" );
		defaultSettings.put( "version", "0" );
		defaultSettings.put( "phoneManager", "+33686955405" );
 
		return defaultSettings;
	}
 
	/**
	 * Reset everything
	 */
	public synchronized void ResetErything() {
		try {
			FileConnection fc = (FileConnection) Connector.open( "file:///" + _fileName, Connector.READ_WRITE );
			if ( fc.exists() )
				fc.delete();
                        _settings = new Hashtable();
		} catch ( Exception ex ) {
			if ( Logger.E_CRITICAL )
				Logger.Log( 16725, "Settings.ResetEverything", ex );
		}
	}
 
	/** 
	 * Save setttings
	 */
	public synchronized void Save() {
 
		// If there's no settings, we shouldn't have to save anything
		if ( _settings == null )
			return;
 
		try {
			Hashtable defSettings = getDefaultSettings();
			Enumeration e = defSettings.keys();
			FileConnection fc = (FileConnection) Connector.open( "file:///" + _fileName, Connector.READ_WRITE );
			if ( fc.exists() )
				fc.delete();
			//fc = (FileConnection) Connector.open("file:///" + _fileName, Connector.READ_WRITE);
			fc.create();
			OutputStream os = fc.openOutputStream();
 
			while ( e.hasMoreElements() ) {
				String key = (String) e.nextElement();
				String value = (String) _settings.get( key );
				String defValue = (String) defSettings.get( key );
 
				if ( // if there is a default value
					defValue != null && // and
					// the value isn't the same as the default value
					defValue.compareTo( value ) != 0 ) {
					String line = key + "=" + value + '\n';
 
					if ( Logger.E_DEBUG )
						Logger.Log( 131, "Settings.Save.line = \"" + line + "\"" );
 
					os.write( line.getBytes() );
				}
 
			}
			os.flush();
			os.close();
			fc.close();
		} catch ( Exception ex ) {
			if ( Logger.E_CRITICAL )
				Logger.Log( 131, "Settings.Save", ex );
		}
 
	}
 
	/**
	 * Init (and ReInit) method
	 */
	private void CheckLoad() {
		if ( _settings == null )
			Load();
	}
 
	/**
	 * Get a setting's value as a String
	 * @param key Key Name of the setting
	 * @return String value of the setting
	 */
	public synchronized String GetSetting( String key ) {
		CheckLoad();
		if ( _settings.containsKey( key ) )
			return (String) _settings.get( key );
		else
			return null;
	}
 
	/**
	 * Sets a setting
	 * @param key Setting to set
	 * @param value Value of the setting
	 */
	public synchronized void SetSetting( String key, String value ) {
		CheckLoad();
		if ( _settings.containsKey( key ) )
			_settings.remove( key );
 
		_settings.put( key, value );
 
		OnSettingChanged( key );
	}
 
	/**
	 * Get a setting's value as an int
	 * @param key Key Name of setting
	 * @return Integer value of the setting
	 * @throws java.lang.NumberFormatException When the int cannot be parsed
	 */
	public int GetSettingInt( String key ) throws NumberFormatException {
		String value = GetSetting( key );
 
		if ( value == null )
			return -1;
 
		return Integer.parseInt( value );
	}
}

Latter on, I added some settings consumer capacities. It allows to use this class in some sort of “third party” developed components. Each component had to register itself to the settings management class, the settings management class could then request the default value settings it will provide, and it could set/get settings. The settings event class also launched an event when a setting was changed.

Posted in English. Tags: , , , , . No Comments »
«
»