The M2MP protocol

Today, I’m going to talk about a little protocol that I’ve implemented in few softwares (on TC65, .Net client and .Net server). It won’t be very interesting for most of you (99.99%). But still, I wanted to write it because it has been very useful and it might prevent you from writing the next stupid M2M protocol specifications.

Motivations

I made this simple protocol to solve very simple and common problems on M2M equipments, we want to :

  • Transmit and receive in real-time
  • Transmit as few data as possible
  • Send and receive any kind of data

So the basic ideas are to :

  • Keep a TCP connection open all the time by sending regular keep-alive frames.
  • Define named channels that are only transmitted once
  • Send byte array, and two-dimensionnal byte arrays on these channels

We rely on the TCP protocol for packet checksuming, packet ordering and every little things it does very well.

Basically, we send these kind of frames :

  • Identification frames
  • Channel name to id definition
  • Data transmission of single byte arrays (Byte[]).
  • Data transmission of two-dimentionnal byte arrays (Byte[][]).

The protocol

Identification
The client ask for authentication :

1
2
3
[ 1 ] { 0x01 } // Identification header
[ 1 ] { 0x05 } // Size of the identification
[ 5 ] { 'h', 'e', 'l', 'l', 'o' } // Identifier

The server replies “ok” or “not ok”

1
2
[ 1 ] { 0x01 } // Identification request
[ 1 ] { 0x01 } // OK, 0x00 for not ok

Ping / Keep alives
Client sends a ping request :

1
2
[ 1 ] { 0x02 } // client ping request header
[ 1 ] { 0x15 } // the number (here 0x15) can be incremented or random

Server answers the ping request :

1
2
[ 1 ] { 0x02 } // client ping server response header
[ 1 ] { 0x15 } // where 0x15 is the number of the ping request

Server sends a ping request :

1
2
[ 1 ] { 0x03 } // server ping request header
[ 1 ] { 0x16 } // the number (here 0x16) can be incremented or random

Client answers the ping request :

1
2
[ 1 ] { 0x03 } // server ping client response header
[ 1 ] { 0x16 } // where 0x16 is the 0x16 number of the ping request

Defining a named channel
This applies to both client to server and server to client communcation.

1
2
3
4
[ 1 ] { 0x20 } // channel definition frame header
[ 1 ] { 0x0D } // where 13 is the size of the following message
[ 1 ] { 0x03 } // where 3 is the id of the channel
[ 12 ] { 'c', 'h', 'a', 'n', 'n', 'e', 'l', ' ', 'n', 'a', 'm', 'e' } // the name of the channel : "channel name"

Sending some data
This applies to both client to server and server to client communcation.

For 0 to 254 sized messages :

1
2
3
4
[ 1 ] { 0x21 } // one byte sized data transmission frame header
[ 1 ] { 0x06 } // size of the data
[ 1 ] { 0x03 } // where 3 is the id of the channel
[ 5 ] { 0x01, 0x02, 0x03, 0x04, 0x05 } // data transmitted on the channel

For 255 to 65534 sized data messages :

1
2
3
4
[ 1 ] { 0x41 } // two bytes sized data transmission frame header
[ 2 ] { 0x00, 0x06 } // size of the data
[ 1 ] { 0x03 } // where 3 is the id of the channel
[ 5 ] { 0x01, 0x02, 0x03, 0x04, 0x05 } // data transmitted on the channel

For 65 535 octets to 4 294 967 294 sized data messages :

1
2
3
4
[ 1 ] { 0x61 } // two bytes sized data transmission frame header
[ 4 ] { 0x00, 0x00, 0x00, 0x06 } // size of the data
[ 1 ] { 0x03 } // where 3 is the id of the channel
[ 5 ] { 0x01, 0x02, 0x03, 0x04, 0x05 } // data transmitted on the channel

For data arrray transmission, it’s the same except frame header are 0x22, 0x42, 0x62 instead of 0x21, 0x41, 0x61. If you transmit data array like this one Byte[][] data = { { 0x01, 0x02 }, { 0x03, 0x04, 0x05 }, { 0x06, 0x07, 0x08, 0x09 } } :

1
2
3
4
5
6
7
8
9
[ 1 ] { 0x22 } // one byte sized data transmission frame header
[ 1 ] { 0x0D } // size of the data
[ 1 ] { 0x03 } // where 3 is the id of the channel
[ 1 ] { 0x02 } // size of the first array
[ 2 ] { 0x01, 0x02 } 
[ 1 ] { 0x03 } // size of the second array
[ 3 ] { 0x03, 0x04, 0x05 } 
[ 1 ] { 0x04 } // size of the third array
[ 4 ] { 0x06, 0x07, 0x08, 0x09 }

For bigger arrays, you need to define :

1
2
3
4
5
6
7
8
9
[ 1 ] { 0x42 } // one byte sized data transmission frame header
[ 1 ] { 0x10 } // size of the data (16)
[ 1 ] { 0x03 } // where 3 is the id of the channel
[ 2 ] { 0x00, 0x02 } // size of the first array
[ 2 ] { 0x01, 0x02 } 
[ 2 ] { 0x00, 0x03 } // size of the second array
[ 3 ] { 0x03, 0x04, 0x05 } 
[ 2 ] { 0x00, 0x04 } // size of the third array
[ 4 ] { 0x06, 0x07, 0x08, 0x09 }

and so on :

1
2
3
4
5
6
7
8
9
[ 1 ] { 0x62 } // one byte sized data transmission frame header
[ 1 ] { 0x16 } // size of the data (22)
[ 1 ] { 0x03 } // where 3 is the id of the channel
[ 4 ] { 0x00, 0x00, 0x00, 0x02 } // size of the first array
[ 2 ] { 0x01, 0x02 } 
[ 4 ] { 0x00, 0x00, 0x00, 0x03 } // size of the second array
[ 3 ] { 0x03, 0x04, 0x05 } 
[ 4 ] { 0x00, 0x00, 0x00, 0x04 } // size of the third array
[ 4 ] { 0x06, 0x07, 0x08, 0x09 }

Sample data types transmitted

Strings
Easy… Just the bytes value of the string

Position
when moving :

1
2
3
4
5
[ 4 ] Timestamp : UInt32 (since 1970 or 2009 if you wish)
[ 4 ] Longitude : Float
[ 4 ] Latitude : Float
[ 2 ] Speed : UInt16
[ 2 ] Altitude : UInt16

= 16 bytes

when stopped :

1
2
3
[ 4 ] Timestamp : UInt32 (since 1970 or 2009 if you wish)
[ 4 ] Longitude : Float
[ 4 ] Latitude : Float

= 12 bytes

Battery
Percentage :

1
[ 1 ] Percentage (Byte)

Voltage :

1
[ 2 ] Voltage in mV (UInt16)

Sample communication

This is just to show how is working a sample communication

1
2
3
4
--> [] { 0x01, 0x04, 0x01, 0x02, 0x03, 0x04 } // identification frame
<-- [] { 0x01, 0x01 } // identification accepted
--> [] { 0x20, 0x08, 0x00, 'b', 'a', 't', 't', 'e', 'r', 'y' } // We define the channel "battery" with id 0x00
--> [] { 0x21, 0x02, 0x00, 70 } // We send 70 (meaning 70%) on the "battery" channel.

On top of that

On top of that, I added some settings management and status specifications.

Settings management
The server can get and set some settings on the “_set” channel.
If it’s a two-dimentionnal array and the first element is “s” (like “set”) it means that it sets some settings, the next elements will be “setting1=value1”, “setting2=value2”, etc.
If it’s a two-dimentionnal array and the first element is “g” (like “get”) it means that it needs the client to report the value of some settings, like “setting1”, “setting2”
If it’s a single dimentionnal array that contains “ga” (like “get all”) it means that the server wants the client to report the value of every settings.
If the client doesn’t know some settings that were sent to it, it can report it using “u”.
The client can also send “c” (in the same format as the get) asynchronously if a setting changed on the client side.

Sample transmission:
Setting values:

1
 --("_set")--> { "s", "setting1=value1", "setting2=value2" }

It’s the responsibility of the server to send an acknowledge request to know that the client has actually successfully received this settings.

Getting values:

1
2
 --("_set")--> { "g", "setting1", "setting2" }
<--("_set")--  { "g", "setting1=value1", "setting2=value2" }

Setting and gettings value :

1
2
 --("_set")--> { "sg", "setting1=value1", "setting2=value2" }
<--("_set")--  { "g", "setting1=value1", "setting2=value2" }

Gettings all the values :

1
2
 --("_set")--> "ga"
<--("_set")--  { "g", "setting1=value1", "setting2=value2" }

Sending a value that is unknown to the client :

1
2
 --("_set")--> { "sg", "setting1=value1", "setting98=value98", "setting99=value99" }
<--("_set")--  { "g", "setting1=value1", "setting98", "setting99" }

Same since here except we don’t specify the get. The client might decide to tell us the setting is unknown.

1
2
 --("_set")--> { "s", "setting1=value1", "setting98=value98", "setting99=value99" }
<--("_set")--  { "u", "setting98", "setting99" }

Status parameters management
The server can get some status parameters on the “_sta” channel. Status are pretty much like settings except that they describe parameters that cannot be changed from the server.

Getting a parameter:

1
2
 --("_sta")--> { "g", "version" }
<--("_sta")--  { "g", "version=0.0.1" }

Getting all parameters:

1
2
 --("_sta")--> { "ga" }
<--("_sta")--  { "g", "hw=tc65i/1.1", "sw=sampleprogram/0.0.1", "cap=sms,applicationA/1.0" }

Value changed on the client side

1
<--("_set")-- { "c", "setting1=value1" }

Capacities
Previously, capacities where requested by server with the “?” message on the “_cap” channel but they are now handled through the standard status message and the “cap” parameter.

1
2
 --("_sta")--> { "g", "cap" }
<--("_sta")--  { "g", "cap=sms_relay_send_1,gpio_1" }

Command execution

1
2
3
4
// To send a command
 --("_cmd")--> { "e", "<commandId>", "command", "arg1", "arg2" }
// To acknowledge the command
<--("_cmd")--  { "a", "<commandId>" }

In the future commands could algo be acknowledge as a group by sending an acknowledge request.

Files transmission
The server sends:

1
<--("_fi:u")-- { byte[5] = "file1", byte[4] = (uint32_t) file1Size, byte[16] = file1Hash, byte[5] = "file2", byte[4] = (uint32_t) file2Size, byte[16] = file2Hash }

If the client wants to get the detailed subhashes:

1
2
 --("_fi:h")--> { byte[5] = "file1", byte[4] = (uint32_t) size, byte[4] = (uint32_t) offset1, byte[4] = (uint32_t) offset2 }
<--("_fi:h")--  { byte[5] = "file1", byte[4] = (uint32_t) size, byte[4] = (uint32_t) offset1, byte[16] = hash, byte[4] = (uint32_t) size, byte[16] = hash }

Then if the client has a different value of the files:

1
2
 --("_fi:g:file1")--> { byte[4] = (uint32_t) offset, byte[4] = (uint32_t) size }
<--("_fi:g:file1")--  { byte[4] = (uint32_t) offset, byte[ x ] = data, byte[16] = file1Hash }

Publish/Subscribe (MQTT like)
You could subscribe to a channel “toto” using:

1
2
 --("_pubsub")--> { "s" , "toto" }
<--("_pubsub")--  { "sa", "toto", [ 0x01 ] }

You could unsubscribe to a channel “toto” using:

1
2
 --("_pubsub")--> { "u", "toto" }
<--("_pubsub")--  { "ua", "toto", [ 0x01 ] }

And the rest is left like this.

Gateway acting and sub equipements
Any equipment can act as a gateway by encapsulating data of an sub-equipment in a channel name like this “_eq/[sub equipment identifier]” but this has never been used for the moment.

Every packet coming from the sub-equipment is relayed to the server as a data packet.
If the equipment, identified locally by “mac:112233445566” sends an identification to the server, this will work like this :
Subequipment to Equipment:

1
[ 0x01 0x04 't' 'o' 't' 'o' ]

This will be relayed to the server as data message like this :

1
--("_eq/mac:112233445566")--> [ 0x01, 0x04, 't', 'o', 't', 'o' ]

Meaning in raw data :

1
[0x21 0x07 0x03 0x01 0x04 't' 'o' 't' 'o' ]

(where 0x03 is the generated channel id)

Stupid protocols

I’ve seen a lot of protocols from big companies doing some really stupid things like :

  • Fixed size frames for not fixed size data.
  • 4 zero filled bytes preamble. Why the hell would we need preamble in a TCP connection ?
  • 8 bytes timestamp in millisecond when data has a 1 second precision.
  • 4 bytes integer for specifying some number that never exceed 32.
  • Checksums on top of the TCP checksumming mechanism.
  • Redundant data at the beginning and the end.
  • Disconnecting very frequently (TCP establishment + identification costs a lot).

3 thoughts on “The M2MP protocol”

  1. Hi Florent,
    I am doing something similar but using the “Record format” described in http://cr.yp.to/cdb/cdbmake.html

    I chose it because it has an extremely simple definition (only two text lines), is very easy to parse (<30 lines of Java code, exception handling included), is human readable, has little protocol overhead, it can be sanity checked.

  2. Hi Marcello,

    Thank you. This is very interesting. But I’m not sure this format suits the same goals. It’s more a simple storage format than a transmission protocol. And it takes much more data to transmit messages.

    Nevertheless, this is might be a very good solution to store some array of data on the chip so that it can be easily read by a human.

Leave a Reply

Your email address will not be published. Required fields are marked *