PIC18 development

That’s something I wanted to do for quite a long time now : Creating some programs for PIC chips. I bougth the ICD3 programmer/debugger with the PICDEM Z development kit (for ZigBee communication). I can’t say it was chip (it costed me 500 €), but I’ve made a good use of it.

The thing is, it brings me back to my first C programs. It’s quite hard to come back to some unmanaged, memory limited, namespace less language. I thought the lack of objects would be the hardest part, but in fact it’s not (you can easily emulate it). The hardest part is the lack of namespace and the massive use of “#define”, I really would be happier with a some sort of limited version of C++. The MPLAB development environment is awful (compared to Visual Studio) : no reindenting, no refactoring, no smart-anything. This is WILD for me. It’s like being a citizen in the jungle.
I took back the old Doxygen to add some pretty doc.

I’m doing it step by step, 2 to 3 hours some evenings. I started by being able to use the leds, then the buttons, then the serial port (using the UARTIntC library). Recently I added MiWi protocol support. My two chips are connecting to each other and sending messages to each other just fine.

As you can guess, programming this kind of equipment is much less fun than doing some C# .Net. But it has the major advantage of running on very low-cost (less than 4€) chips.

There are some funny things with the Microchip PIC18 Compiler (MCC18).

If you take this code :
buttons.c :

1
2
3
4
#include "buttons.h"
void buttons_Task() {
    // Some stuff
}

buttons.h :

1
void buttons_Task();

It produces this error :

1
Warning [2058] call of function without prototype

The reason is that you must declare your prototypes with no arguments with a void. You have to declare it as “void buttons_Task(void);”. Well, thank you Microchip for the dumb pre-compiler error messages.

I also discovered that they are two memory (data and program memory).
In one file I had an : “extern unsigned char *var;” and in the other one : “unsigned char var[8];”. The thing is the compiler doesn’t throw you any warning but when you will copy data from this variable, you will get some other data. Because the C programs doesn’t use the same memory in the two cases. MCC18 doesn’t detect this and the debugger doesn’t help much because it chooses one variable.

TC65 HTTP POST request

This is just a quick code to show how to do a simple http POST request from a TC65 chip.

    private void httpCheck() {
        String data = "Parameter1=000009960623185&Parameter2=$GPWPL,4807.038,N,01131.000,E,WPTNME*5C";
        String url = "http://test.webingenia.com/postIdent";
        HttpConnection conn = null;
        InputStream is = null;
        OutputStream os = null;
        try { // Test HTTP connection

            // We prepare the POST request
            conn = (HttpConnection) Connector.open( url );
            conn.setRequestMethod( HttpConnection.POST );
            conn.setRequestProperty( "Content-Type", "application/x-www-form-urlencoded" );
            os = conn.openOutputStream();
            os.write( data.getBytes() );

            // We launch the request
            System.out.println( "Response code : " + conn.getResponseCode() );

            // We display the generated content
            is = conn.openInputStream();
            int ch;
            System.out.println( "Output : " );
            while ( (ch = is.read()) != -1 )
                System.out.print( (char) ch );

        } catch ( Exception ex ) {
            System.out.println( "Http : ex : " + ex.getClass() + " : " + ex.getMessage() );
            ex.printStackTrace();
        } // Whatever happens, we close everything
        finally {
            try {
                if ( conn != null )
                    conn.close();
                if ( is != null )
                    is.close();
                if ( os != null )
                    os.close();
            } catch ( Exception ex ) {
                System.out.println( "Http : ex2 : " + ex.getClass() + " : " + ex.getMessage() );
                ex.printStackTrace();
            }
        }
    }

Server-Side HTTP Request tester

I made this simple test website for those of you who would want to check if their HTTP request made from a simple equipment like the TC65 are sent correctly.

The idea is simple :

You can contact me if you don’t understand something, if something doesn’t work or if you would like me to add a feature.

Yum Transaction Check Error on x64 CentOS

If you encounter a “Transaction Check Error” on yum on a x64 system during an install, an update or an upgrade, you will find out that most of the time, you can’t remove the problematic packages. But it’s very likely that the problem comes from a i386 version of a package. The easiest way to proceed is just to remove the i386 version of each software or library as it appears on the Transaction Check Error.

I had a problem with postgresql-libs (i386) which was required by subversion (i386). I removed the postgresql-libs (and I had to remove subversion), but both of them were also in a x86_64 version, so it didn’t change anything. They were just garbage packages.

.Net, Flex and WebORB.Net

I’ve been working on a project where we had to use Flex on a .Net environment and it had to be realtime. I was a little worried at first that it could be tricky to set up. But with WebORB.Net it’s pretty easy. We used the integrated RTMP (Real Time Messaging Protocol) messaging server. It’s almost like using WCF. The most important differences are that, by default, objects are transmitted as Hashtable and calls can’t be synchronous. We can bind the object to right .Net object within the WebOrb management console but we decided to do it ourself using reflection (because we don’t like to depend too much on the management console).

RTMP
The result is pretty impressive as it makes a really powerful real-time management system. And it’s freaking fast. Like WCF, it can handle concurrent calls. It’s in fact easier because everything is asynchronous but still, it does manage that. These calls are received on parallel threads.

The only problem with WebORB.Net is that you don’t have too much examples in the documentation. Our biggest problem was that we didn’t even knew how to detect what was the current calling connection on a method. This is pretty easy but you can’t exactly guess it. This is a dummy sample to show you how it’s done :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
public class FlexConnectionsServer:ApplicationAdapter {
	private int _connectionIdCounter = 0;
	private Dictionary<IConnection,int> _connectionsToId = new Dictionary<IConnection,int>();
 
	// Returns the CURRENT connection (must be used from a method called by the flex client)
	private IConnection CurrentConnection { get { return ConnectionHub.getConnectionLocal(); } }
	private int CurrentID { get { return _connectionsToId[ CurrentConnection ]; } }
 
	// When the RTMP service is loaded, we start the time dispatcher thread
	public override bool appStart( IScope app ) {
		new Thread( Thread_TimeDispatcher ).Start();
	}
 
	// This method is called when a client connects
	public override bool appConnect( IConnection conn, object[] parms ) {
		lock ( _connectionsToId ) {
			_connectionsToId.Add( conn, _connectionIdCounter++ );
		}
 
		return true;
	}
 
	// This method is called when a client disconnects
	public override void appDisconnect( IConnection conn ) {
		lock ( _connectionsToId ) {
			_connectionsToId.Remove( conn );
		}
	}
 
	// This is a sample method call
	public String GetMyName() {
		return String.Format( "Client{0}", CurrentId );
	}
 
	// This thread dispatches the current time to every client
	private void Thread_TimeDispatcher() {
		while( true ) {
			var t = DateTime.UtcNow;
			SendCommand( "CurrentTime", new Object[] { t } );
			Thread.Sleep( 1000 );
		}
	}
 
	// Call a remote method on each client connection
	private void SendCommand( String command, Object[] params ) {
		lock ( _connectionsToId ) {
			foreach( var kvp in _connectionsToId )
				( kvp.Key as IServiceCapableConnection ).Invoke( command, params );
		}
	}
}

It’s a little bit like using WCF in Single instance mode.

IIS 7.0 Setup
There was something really weird that happened with my IIS 7.0 on my Windows 7 x64. Using the RTMP connection, I frequently had null return values on .Net method calls. We firstly find a quick way to solve it : Recall each time we get a null value. It’s really dirty, but It worked as a very temporary solution.
Then, when debugging on the IIS process, I noticed a lot of NullReferenceException exceptions displayed on the debugger console. They were about the WebORB “LicenseManager” class, and I had as much exceptions as I had null values on the Flex client. I tried a lot of things that didn’t do anything. And then, I changed the “Managemd pipeline mode” of the Application Pool from “Integrated” to “Classic”, and the problem was solved. The really weird stuff here is that WebORB recommends to use the “Integrated” mode.

MidnightCoders support
We asked MidnightCoders to send us a community edition license, and they instantly answered. We got our license within the 48 hours after our first mail (I was the one slowing it down).

Quick note on the project
To give you a better view of the project I’m talking about :
GPRS Equipment <--(M2MP protocol)–> [Generic M2M Server] <--(WCF)--> [Business logic software] <--(WCF)--> [Flex connections server] <--(RTMP)--> Flex
We’ve got a little equipment (TC65 based as you can guess) that connects to a generic server using our own (real-time and bandwidth optimized) protocol. A business logic software connects to this server using a WCF service. And then we have the WebORB.Net / IIS that connects to this business logic software.

WCF helps make very interesting .Net applications because it simplifies all the communication but can also create deadlocks you might not have seen. Take this example : You have a lock on the WCF Server on the clients list object, you make a call to the WCF client and this client makes a call to the WCF Server that also requires a lock. If you were doing all this client/server calls on the same process this wouldn’t be a problem. But as you use WCF client-to-server and server-to-client calls are made on differents threads. You create a deadlock (unless you set the OperationContract attribute to IsOneWay and/or you fix the deadlock source).

If you’re doing real-time applications, I would recommend to define all the frequently used methods with the “IsOneWay=true” option.

06/11/2009 Update : I added a sample server-to-client code because some people were interested by it.

LinQ : Join with multiple conditions

I wanted to some simple join in LinQ on multiple criteria. And coming from standard SQL it’s not as straightforward as I thought it would be. The little disturbing thing is that you lose auto-completion.

Here is my database schema :

1
2
3
4
5
6
7
8
9
10
11
12
13
User:
* UserId          int           (PK)
* Login           String
* DateOfBirth     DateTime
 
Parameter:
* ParameterId     int           (PK)
* Name            String
 
User_Parameter:
* UserId          int           (FK User:Userid)
* ParameterId     int           (FK Parameter:ParameterId)
* Value           String

If you want to fetch the “PreferredSuperHero” parameter, you have to execute :

1
2
3
4
5
6
7
8
9
10
11
12
13
var query = 
from user in Db.User
join preferredSuperHero in Db.User_Parameter on 
	new { 
		UserId = user.UserId, 
		ParamName = "PreferredSuperHero" 
	} 
	equals new { 
		UserId  = preferredSuperHero.UderId, 
		ParamName = preferredSuperHero.Parameter.Name 
	}
orderby user.Name
select new { User = user, PreferredSuperHero = preferredSuperHero };

In SQL, you would execute :

1
2
3
4
5
SELECT `u`.*, `up`.*
FROM `user` AS `u`
JOIN `Parameter` AS `p`
JOIN `User_Parameter` AS `up` ON `up`.`UserId`=`u`.`UserId` AND `up`.`ParameterId`=`p`.`ParameterId`
WHERE  `p`.`ParamName`='PreferredSuperHero'

On the LinQ Query, the creation of an anonymous type is necessary for the multiple join condition you can see on line 4.

My own little StreamRipper

This is a stream ripping program I did in order to read and rip streams (like Frequence 3) sending frequent advertising messages that usually prevent users from ripping them. It’s somehow like a protection bypass ripper.

What it does :

  • Saves shoutcast streams in MP3 files
  • Saves shoutcast streams in one MP3 file with its cue sheet file
  • Doesn’t consider text information that have seen recently (this is why I did it)
  • Adds MP3 ID3 tags (ID3v1 and ID3v2) using TagLib#
  • Reads M3U playlist streams

Here are the command line options :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
 
StreamRipper - Help
===================
 
-h    --help               This help
-u    --url                URL to specify (stream or playlist file)
-d    --output-directory   Output directory (Default: output)
-a    --user-agent         User-Agent to specify (Default: "FlorentSR/0.2.5.16365 - http://florent.clairambault.fr/")
-e    --encoding           Text encoding (Default: iso-8859-1)
-r    --reconnect-attempts Maximum reconnect attempts (Default: 60)
-m    --min-file-size      Minimum file size (Format: 2M, 1G, 100K; Default: 2M)
-M    --max-file-size      Maximum file size (Format: 2M, 1G, 100K; Default: unlimited)
-c    --cue-sheet          Rip one big MP3 with a cue file
-x    --exclude            Exclude some pattern

As always, it can work on Windows with .Net and Linux/Mac Os X with Mono.

[ download streamripper ]

Note : This program was used to build WebRadioWatcher (only in french right now) which doesn’t do much than indexing all the played song by the webrados.

Edit : 11/11/2009 : I removed the other posts around this program. This is the only post about it. It describes the latest version of the stream ripping program I made.

Edit : 23/02/2010 : Updated the program to 0.2.5 to add the “-x” option. With it you can easily filter the “next song :” stream titles.

LinQ to Entities on ADO.Net vs LinQ to SQL

To make it short :

  • LinQ to SQL only works with SQL Server but it’s working great. It’s simple, supports all LinQ operations and optimizes the generated SQL queries.
  • LinQ to Entities on ADO.Net works with lots of database but it doesn’t behave like LinQ to SQL and doesn’t support some key LinQ features.

Now the long story :
For the last project I’ve been working on, we wanted to use LinQ and still be able to switch to one database to an other (MSSQL to MySQL for instance). So we decided we would use LinQ to entity. The generated object are roughly the same, the code to Create/Read/Update/Delete data is roughly the same. So we were quite happy.

Deffered loading
When you do a LinQ request, the data from the associated table isn’t fetched. There’s in fact no other way, as you would get your whole database if you always fetched all the associated objects of your current query.
Deferred loading consists in loading the data coming from an associated table when you request it (when you access your object property).
Eager loading consists in loading the data when you request it.

In LinQ to entities, deferred loading isn’t supported. You have to use eager loading with the .Include(“Property”) LinQ method.

1
var query = from u in Db.User.Include( "Group" ) select u;

But in fact, you can do some deferred loading very easily in LinQ to Entities :

1
2
3
4
5
6
7
8
var query = from u in Db.User select u;
 
foreach( var user in query ) {
    if ( ! user.GroupReference.IsLoaded )     
        user.GroupReference.Load();
 
    Console.WriteLine( "User={0}, Group={1}", user.Name, user.Group.Name );
}

Non supported LinQ key features
Well, now I only want to select the users 1, 2, 3 :

1
2
var users = new int[] { 1, 2, 3 };
var query = from u in Db.User.Include( "Group" ) where users.Contains( u.UserId ) select u;

Oups, that would work on LinQ to SQL, but on LinQ to Entities it doesn’t. LinQ to Entities doesn’t support the .Contains or .Any methods. (Note : these methods are supported since .Net 4.0, see below).

There’s in fact a solution but it requires your request to be typed in a method based LinQ Query :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
		static Expression<Func<TElement, bool>> BuildContainsExpression<TElement, TValue>(
			Expression<Func<TElement, TValue>> valueSelector,
			IEnumerable<TValue> values
		) {
			if ( null == valueSelector ) { throw new ArgumentNullException( "valueSelector" ); }
			if ( null == values ) { throw new ArgumentNullException( "values" ); }
			ParameterExpression p = valueSelector.Parameters.Single();
			// p => valueSelector(p) == values[0] || valueSelector(p) == ...
			if ( !values.Any() ) {
				return e => false;
			}
			var equals = values.Select( value => (Expression) Expression.Equal( valueSelector.Body, Expression.Constant( value, typeof( TValue ) ) ) );
			var body = equals.Aggregate<Expression>( ( accumulate, equal ) => Expression.Or( accumulate, equal ) );
			return Expression.Lambda<Func<TElement, bool>>( body, p );
		}

And then it looks like this :

1
2
var users = new int[] { 1, 2, 3 };
var query = Db.User.Where( BuildContainsExpression<User, int>( u=> u.UserId, users ) );

Now, let’s say you want to use the Include method in the method based query. Well you can’t. But someone added an extension method to do this :

1
2
3
4
5
	public static class ObjectQueryExtension {
		public static ObjectQuery<T> Include<T>( this ObjectQuery<T> mainQuery, Expression<Func<T, object>> subSelector ) {
			return mainQuery.Include( ( ( subSelector.Body as MemberExpression ).Member as System.Reflection.PropertyInfo ).Name );
		}
	}

You have to use it like that :

1
2
var users = new int[] { 1, 2, 3 };
var query = Db.User.Include( u => u.Group ).Where( BuildContainsExpression<User, int>( u=> u.UserId, users ) );

I hope this could help you start or switch to LinQ to entities in case you hesitated.

LEFT JOIN in LinQ to entities
You know how LEFT JOIN works in LinQ, you need a join and DefaultIfEmpty :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
var query = 
from user in Db.User
join preferredSuperHero in Db.User_Parameter on 
	new { 
		UserId = user.UserId, 
		ParamName = "PreferredSuperHero" 
	} 
	equals new { 
		UserId  = preferredSuperHero.UderId, 
		ParamName = preferredSuperHero.Parameter.Name 
	}
	into joinedPreferredSuperHero
from preferredSuperHero in joinedPreferredSuperHero.DefaultIfEmpty()
orderby user.Name
select new { User = user, PreferredSuperHero = preferredSuperHero };

Except… DefaultIfEmpty doesn’t works with LinQ-to-entities. You generally can find other solutions, but it will add some difficulties. (Note : this method is supported since .Net 4.0, see below).

Check what is supported
In this page, you can check if the methods you might require are available. The two non supported methods I talked about are supported with the .Net 4.0 (currently only available in bêta). So most of the problem related to the missing methods will be solved.

About switching from one base to an other
Switching from one base to the other can be pretty easy if you have the right conversion tool. If not, you have to do all the work yourself. If you wish to switch to MySQL for instance, you can use the MySQL conversion tool. It converts the table structure, the keys, the foreign keys, the indexes and the data from MSSQL to MySQL.

After that, the generated code is exactly the same but I’m not sure you can use one EDMX model file for multiple databases. It might work by using an external SSDL (instead of an resource embedded file) that you select in the connection String for each database. But I haven’t tested it yet.

My little conclusion
LinQ to entities works pretty well. The non automatic deferred loading is really not a problem, especially since it totally kills performances (it fetches data row by row instead of loading everything from a single request). My biggest problem was the lacking support of some LinQ methods, you can solve most of them easily (it just makes an ugly code).

With the .Net 3.5, I think the multi-database feature offered by LinQ to Entities on ADO.Net worth this little extra effort to make it work. But if you don’t care about this (because you only build a little short-life program), you should stick to LiNQ to (MS)SQL as everything works instantly on it.

With the .Net 4.0, LinQ to entities becomes even more interesting. And that’s a reason to start using it with the .Net 3.5 framework.

Sources :

TC65 Module Exchange Suite (MES) problems

WARNING: All the Cinterion related content from this blog will be removed to go to the javacint wiki soon. Please get used to going there.

Update (2011-06-28):

You should look at the jOBEXFtp program if you don’t want to spend hours fixing a problem with a crappy software.

When you want to locally deploy software on your TC65 chip, you need the Module Exchange Suite (MES). If it doesn’t work, it won’t tell you why, finding out why can be quite tricky.

If you want to find out why the communication with your TC65 failed, the best way is to use the portmon tool and see what is exchanged between the MESServer.exe and the chip.

I collected 4 portmon captures that could help you diagnose your MES communication problems :

  • Here is a successful MES files listing at 19200 bps. You can see that it sends the “AT+CGMM” at different bit rates unless it gets a good response. Then it sets some serial communication options. And then it sends the “AT^SQWE=100” commands which fails and then establish a successful link using the “AT^SQWE=3” command.
  • Here is a failed MES file listing. Here, after the “”AT^SQWE=3″” is sent, MES sends a 26 bytes message and receives a 41 bytes message from the chip. Then it should request the file listing. But here, MES doesn’t request anything. Maybe MES doesn’t work well, or maybe the reply from the chip doesn’t satisfy his requirements.
    Anyway, after that it should send a 27 bytes message containing : “x-obex/folder-listing”, and it should get the file listing. But it doesn’t. It reads on the serial port like if it was still waiting for something. And finally, it closes the connection (with “+++”).
  • Here is a totally failed communication. Here it’s because the modem gets stuck on waiting for the modem hardware flow control (“IOCTL_SERIAL_GET_MODEMSTATUS”) when it’s only a 3 wires (TX,RX,GND) communication. So MES never reads or writes anything.
  • Here is communication that failed by not receiving anything. Everything seems ok but no data was received. It’s quite likely that the RX cable is disconnected.

If you have the same kind of problem, you can send me your portmon capture file.