Google PowerMeter and sample C# .Net API usage

Update 2001-06-25
A lot of people still contact me about this project. I received this mail today, just in case you’re getting interested by the project, you should know that it will die pretty soon:

Dear PowerMeter User,

We first launched Google PowerMeter as a Google.org project to raise awareness about the importance of giving people access to their energy information. Since our launch, there’s been more attention brought to this issue, and we’re excited that PowerMeter has helped demonstrate the importance of access to energy data. However, our efforts have not scaled as quickly as we would have liked, so we have decided to retire PowerMeter.

You will continue to have access to the product until September 16, 2011, after which time you will no longer be able to access your PowerMeter account. We know that having access to your energy information has helped you save energy and money. There are many options available for you from our device and utility partners. Please visit this page to learn more: http://www.google.com/support/powermeter/bin/answer.py?hl=en&answer=1342532

We also understand that having your historical energy data is important to you. We’ve made it easy for you to download your data. To export your PowerMeter data to a CSV (Comma Separated Values) file, log in to your account and go to “Account Settings.” More information can be found here: http://www.google.com/support/powermeter/bin/answer.py?hl=en&answer=164264

We appreciate your understanding and hope that you’ve enjoyed using Google PowerMeter. If you have questions about this announcement, please visit our FAQ pages at http://www.google.com/powermeter/about/faqs.html

Thank you,
Google PowerMeter team

© 2011 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043

You’ve received this mandatory service announcement email to update you about important changes to your Google PowerMeter account.

Google PowerMeter offers you to collect your home’s / enterprise’s / specific equipment’s power usage and have some basic analysis on it.

There’s also a Google PowerMeter API so that everyone can build devices able to report their consumption and applications.

I would say one cheap and evolutive way to implement this API would be to use a Z-Wave power meter and have a Z-Wave central controller sent the data to Google servers. It could quickly get interesting because this could make you know the exact consumption of all your lights by calculating the result of the light-on/light-off order on individual lights. And this could even lead to define some max energy consumption scenarios so that you don’t allow more than X watts for all the lights of your house (this is just a idea, I would never put this in my home).

I know this isn’t the next BIG thing. Because everyone knows it’s all about the social right-now. But still, I think this could be the most green project of all time. Because it clearly shows people (or enterprises) how they consume energy and gives the useful analysis to find solutions. And for once it doesn’t try to make you feel guilty about not being green enough, it just brings facts.

By the way, you can have a negative energy consumption (power meter counting backward) in Google PowerMeter. I don’t think this is a bug, it must be for people selling their energy for their solar or wind energy producer equipments. But unfortunately you set the estimated cost of energy but you can’t set its estimated revenue price.

Sample API call

Here is a sample Google PowerMeter API call in C# .Net :
I just changed a little bit of the authentication parameters so that you can’t actually upload data on my account.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
using System;
using System.Globalization;
using System.IO;
using System.Net;
using System.Text;
using System.Threading;
 
/* Equipment was activated with this URL :
 * https://www.google.com/powermeter/device/activate?mfg=WebIngenia&model=TestEquipment1&did=100102030405&cvars=1&dvars=0&rurl=http://test.webingenia.com/powerMeter&snonce=123
 *
 * The resulting request was (available here : http://test.webingenia.com/read/powerMeter#id2473 ) :
 * $_POST = array(
 *		[snonce] => 123
 *		[hash] => _____6c2562aeb7bab4e24afc7beeb18c701c04ba,____8c9f62fa8a137eba7fa35b5f58802fca2e56,____f4db13304c49d36fb81b20cabbe147f8ce34
 *		[token] => ________FhCtzebg_P____8BGMWjtUs
 *		[path] => /user/____4578579014915368/____4578579014915368/variable/WebIngenia.TestEquipment1.100102030405
 * )
 *
 */
 
namespace test {
	class Program {
 
		private static readonly Random _random = new Random();
 
		public const String DeviceHash = // Useless
			"____bf8262aeb7bab4e24afc7beeb18c701c04ba,____8c9f62fa8a137eba7fa35b5f58802fca2e56,____f4db13304c49d36fb81b20cabbe147f8ce34";
 
		public const String DeviceToken = // Needed for authentication
			"________FhCtzebg_P____8BGMWjtUs";
 
		public const String DevicePath = // URL of the device
			"/user/____4578579014915368/_____4578579014915368/variable/WebIngenia.TestEquipment1.100102030405";
 
		public const String DeviceVar = "c1";
 
		public const String FeedUrl = // URL of the Google PowerMeter API
			"https://www.google.com/powermeter/feeds";
 
		public const String EventUrl = // URL where to send the data
			FeedUrl + "/event";
 
		public const int Period = 5000; // 5 to 10 seconds
 
 
		static double MeterValue {
			get {
				return Settings.Default.MeterValue;
			}
			set {
				Settings.Default.MeterValue = value;
				Settings.Default.Save();
			}
		}
 
		static void Send() {
 
			// We increment the counter (by 0 to 1 kWh)
			MeterValue += _random.NextDouble();
 
			Console.WriteLine( "MeterValue : {0}", MeterValue );
 
			// This is what we are going to send
			var strContent = String.Format(
			@"
<feed xmlns=""http://www.w3.org/2005/Atom"" xmlns:meter=""http://schemas.google.com/meter/2008"">
	<entry>
      <category scheme=""http://schemas.google.com/g/2005#kind"" term=""http://schemas.google.com/meter/2008#instMeasurement"" />
      <meter:subject>{0}</meter:subject>
      <meter:occurTime meter:uncertainty=""1.0"">{1}</meter:occurTime>
      <meter:quantity meter:uncertainty=""0.001"" meter:unit=""kW h"">{2}</meter:quantity>
    </entry>
</feed>",
				FeedUrl + DevicePath + "." + DeviceVar,						// Device URL
				DateTime.UtcNow.ToString( "yyyy-MM-dd'T'HH:mm:ss.fffK" ),	// Current UTC date
				MeterValue.ToString( CultureInfo.InvariantCulture )			// Meter value
			);
 
			// We convert it into raw content
			var content = Encoding.UTF8.GetBytes( strContent );
 
			// We create the POST request
			var req = WebRequest.Create( EventUrl );
			req.Headers.Add( "Authorization", "AuthSub token=\"" + DeviceToken + "\"" );
			req.Method = "POST";
			req.ContentLength = content.Length;
			req.ContentType = "application/atom+xml";
 
			try {
				// Opening the request stream means starting the request
				Console.Write( "Requesting..." );
				using ( var requestStream = req.GetRequestStream() ) {
 
					// We send the content
					requestStream.Write( content, 0, content.Length );
 
					// And we end the request by getting the response
					using ( var streamReader = new StreamReader( req.GetResponse().GetResponseStream() ) ) {
						Console.WriteLine( "Result : " + streamReader.ReadToEnd() );
					}
				}
			}
			catch ( WebException e ) { // If a webException occurs (should be 400)
				// We display the error
				var response = e.Response as HttpWebResponse;
				if ( response == null ) // This is not the exception that we want
					throw;
				using ( response ) {
					using ( var responseStream = response.GetResponseStream() ) {
						Console.WriteLine(
							"Error {0} : {1} ",
							response.StatusCode,
							new StreamReader( responseStream ).ReadToEnd()
						);
					}
				}
			}
		}
 
		static void Main() {
			while ( true ) {
				try {
					Send();
				}
				catch ( Exception ex ) {
					Console.WriteLine( "Exception : " + ex );
				}
				Thread.Sleep( Period + _random.Next( Period ) );
			}
		}
 
		// This code was generated by Visual Studio (but I prefer to put it here rather than having to publish the VS project)
		#region Auto-generated
		[System.Runtime.CompilerServices.CompilerGeneratedAttribute()]
		[System.CodeDom.Compiler.GeneratedCodeAttribute( "Microsoft.VisualStudio.Editors.SettingsDesigner.SettingsSingleFileGenerator", "9.0.0.0" )]
		internal sealed class Settings:System.Configuration.ApplicationSettingsBase {
 
			private static Settings defaultInstance = ( (Settings) ( Synchronized( new Settings() ) ) );
 
			public static Settings Default {
				get {
					return defaultInstance;
				}
			}
 
			[System.Configuration.UserScopedSettingAttribute()]
			[System.Diagnostics.DebuggerNonUserCodeAttribute()]
			[System.Configuration.DefaultSettingValueAttribute( "3000" )]
			public double MeterValue {
				get {
					return ( (double) ( this[ "MeterValue" ] ) );
				}
				set {
					this[ "MeterValue" ] = value;
				}
			}
		}
		#endregion
	}
}

I didn’t let it running too much time but still you can see that it works :

WP-Codebox fix so that it doesn’t prevent posts export

If you have a PHP configured with some open_basedir restriction (which is recommended), the WP-Codebox plugin will prevent you from exporting the posts of your blog.

Go to your blog, go in the “wp-content/plugins/wp-codebox” dir and edit the “wp-codebox.php” file :
On line 27, you should have :

1
2
include_once "../../../wp-config.php";
include_once "../../../wp-settings.php";

Well, it fails because the plugin doesn’t use the good practices, so it’s pretty easy to fix :

1
2
require_once( ABSPATH.'/wp-config.php' );
require_once( ABSPATH.'/wp-settings.php' );

I’d like to thank the developer of this plugin for doing such a great job. It brings the power of geshi (not the latest version by the way) into wordpress.

BTW, why change :
* include_once to require_once : Why should we accept that a file couldn’t be included on go on ?
* double quote to simple quote : php reads it faster because it doesn’t parse vars (“$” starting words).
* no parenthesis to parenthesis : Well, it’s a method isn’t it ?

Solving BSOD using crash dump files (.dmp)


Blue Screen Of Death are the most annoying things that can happen on Windows. Most of the people just think “my stupid windows just crashed” and sometimes they even get reinstalling it (what a waste of time). But in fact the windows kernel itself never crashes, the drivers do.

And from a general point of view, I think it’s better to understand the problems before solving them. It sometimes takes a little bit more time but it makes you save a lot of time in the long run (and everything in life is about time).

Extract data from the dumps

  • Download and install the debugging Tools from Microsoft.
  • Locate where is your dump file. It should be in “C:\windows\minidump”, but it comes from an other computer you need to copy it locally and know where is it.
  • Open a command prompt (execute “cmd”)
  • Go to the debugging tools directory.
    On a x86 host, type :

    1
    
    cd "C:\Program Files\Debugging Tools for Windows"

    On a x64 host, type :

    1
    
    cd "C:\Program Files\Debugging Tools for Windows (x64)"
  • Type something like that to open the debugger :

    1
    
    kd -z C:\WINDOWS\Minidump\051110-69280-01.dmp
  • Type this to open the logs

    1
    
    .logopen "C:\debuglogs.txt"
  • Type this to automatically load some symbols (humanely readable information corresponding to some raw data) from microsoft symbols servers :

    1
    
    .sympath srv*c:\symbols*http://msdl.microsoft.com/download/symbols
  • Type this to analyze everything :

    1
    
    .reload;!analyze -v;r;kv;lmnt
  • Type this to close logs file and quit :

    1
    
    .logclose;q
  • Open the log file (created in : C:\debuglogs.txt in that example), and then you can analyze the logs.

Analyze the dump

I’m not sure if there’s a good method to do this but I can give you two samples of tests I personally did some with this :

This is from my local computer (Windows 7 x64)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
DRIVER_POWER_STATE_FAILURE (9f)
A driver is causing an inconsistent power state.
Arguments:
Arg1: 0000000000000003, A device object has been blocking an Irp for too long a time
Arg2: fffffa8006675910, Physical Device Object of the stack
Arg3: fffff80004567748, Functional Device Object of the stack
Arg4: fffffa80057e3170, The blocked IRP
 
Debugging Details:
------------------
 
*** WARNING: Unable to verify timestamp for ftdibus.sys
*** ERROR: Module load completed but symbols could not be loaded for ftdibus.sys
 
DRVPOWERSTATE_SUBCODE:  3
 
IMAGE_NAME:  ftdibus.sys
 
DEBUG_FLR_IMAGE_TIMESTAMP:  4ae075df
 
MODULE_NAME: ftdibus
 
FAULTING_MODULE: fffff880015de000 ftdibus
 
CUSTOMER_CRASH_COUNT:  1
 
DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT
 
BUGCHECK_STR:  0x9F
 
PROCESS_NAME:  vmware-authd.e
 
CURRENT_IRQL:  2
 
STACK_TEXT:  
fffff800`045676f8 fffff800`02efde23 : 00000000`0000009f 00000000`00000003 fffffa80`06675910 fffff800`04567748 : nt!KeBugCheckEx
fffff800`04567700 fffff800`02e9afa6 : fffff800`04567830 fffff800`04567830 00000000`00000001 00000000`00000000 : nt! ?? ::FNODOBFM::`string'+0x29330
fffff800`045677a0 fffff800`02e9a326 : fffff800`0303d700 00000000`004951e8 00000000`00000000 00000000`00000000 : nt!KiProcessTimerDpcTable+0x66
fffff800`04567810 fffff800`02e9ae7e : 000000ae`87a1c8c5 fffff800`04567e88 00000000`004951e8 fffff800`0300bf88 : nt!KiProcessExpiredTimerList+0xc6
fffff800`04567e60 fffff800`02e9a697 : fffff800`03007ec1 fffffa80`004951e8 00000000`00000000 00000000`000000e8 : nt!KiTimerExpiration+0x1be
fffff800`04567f00 fffff800`02e95065 : 00000000`00000000 fffffa80`07729b60 00000000`00000000 fffff800`02fadc80 : nt!KiRetireDpcList+0x277
fffff800`04567fb0 fffff800`02e94e7c : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KxRetireDpcList+0x5
fffff880`09dcf4f0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiDispatchInterruptContinue
 
 
STACK_COMMAND:  kb
 
FOLLOWUP_NAME:  MachineOwner
 
FAILURE_BUCKET_ID:  X64_0x9F_3_ftser2k_IMAGE_ftdibus.sys
 
BUCKET_ID:  X64_0x9F_3_ftser2k_IMAGE_ftdibus.sys

Here we can see that it occurred in vmware and that the faulting “module” is ftdibus.sys. This is an USB to serial adapter I use to do some TC65 dev. So this basically means I have to update the driver of this adapter. This is what I did.

In that case this is related to a bad power management of the driver. My computer crashed each time I left VMWare with Windows XP f(or TC65 dev) while my computer was sleeping (I never stop my computer).

This comes from an other computer (a Windows XP x86) but was analyzed on my computer (Windows 7 x64)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 017f0304, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, bitfield :
	bit 0 : value 0 = read operation, 1 = write operation
	bit 3 : value 0 = not an execute operation, 1 = execute operation (only on chips which support this level of status)
Arg4: 804d9b64, address which referenced memory
 
Debugging Details:
------------------
 
 
READ_ADDRESS:  017f0304 
 
CURRENT_IRQL:  2
 
FAULTING_IP: 
nt!memcpy+12c
804d9b64 8b448efc        mov     eax,dword ptr [esi+ecx*4-4]
 
CUSTOMER_CRASH_COUNT:  3
 
DEFAULT_BUCKET_ID:  DRIVER_FAULT
 
BUGCHECK_STR:  0xA
 
PROCESS_NAME:  Idle
 
LAST_CONTROL_TRANSFER:  from f80e6112 to 804d9b64
 
STACK_TEXT:  
8054f544 f80e6112 8054f5a4 017f0304 00000004 nt!memcpy+0x12c
WARNING: Stack unwind information not available. Following frames may be wrong.
8054f594 f80a1cb2 82889268 017f0304 00000099 w22n51+0x6a112
8054f684 f80d999e 8282d028 8299f034 80540107 w22n51+0x25cb2
80550a48 f80d8f7d 82a37c70 8299f01c 8299f008 w22n51+0x5d99e
80550b80 f809b897 82a37c70 80550bac 00000001 w22n51+0x5cf7d
80550b94 f808bb1c 8282d000 80550bac 00000001 w22n51+0x1f897
80550ecc f808eae3 82ac8d20 82c18e50 f8084702 w22n51+0xfb1c
80550ef0 f8084c4b 82d10160 82c18e50 00000000 w22n51+0x12ae3
80550f04 f807cd1c 82c18e50 82ac9000 80550f2c w22n51+0x8c4b
80550f14 f8381e99 82ac9000 80559c20 805599c0 w22n51+0xd1c
80550f2c 804dbbd4 82d10178 82d10164 00000000 NDIS!ndisMDpcX+0x21
80550f50 804dbb4d 00000000 0000000e 00000000 nt!KiRetireDpcList+0x46
80550f54 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0x26
 
 
STACK_COMMAND:  kb
 
FOLLOWUP_IP: 
w22n51+6a112
f80e6112 ??              ???
 
SYMBOL_STACK_INDEX:  1
 
SYMBOL_NAME:  w22n51+6a112
 
FOLLOWUP_NAME:  MachineOwner
 
MODULE_NAME: w22n51
 
IMAGE_NAME:  w22n51.sys
 
DEBUG_FLR_IMAGE_TIMESTAMP:  404c5c4c
 
FAILURE_BUCKET_ID:  0xA_w22n51+6a112
 
BUCKET_ID:  0xA_w22n51+6a112

So here, the problem occurred in a module named “w22n51.sys” which is the driver of the wireless card. So I just had once again to update the driver to solve the problem.

In that case, it just seems that the driver had a nasty bug.

Update: 2011-03-01
I had an other bug when I switched my home network to 802.11n and I fixed by updating my wireless drivers, here is the bug content:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 0000000000000000, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000000, bitfield :
	bit 0 : value 0 = read operation, 1 = write operation
	bit 3 : value 0 = not an execute operation, 1 = execute operation (only on chips which support this level of status)
Arg4: fffff800033f8477, address which referenced memory
 
Debugging Details:
------------------
 
Unable to load image \SystemRoot\system32\DRIVERS\NETw5s64.sys, Win32 error 0n2
*** WARNING: Unable to verify timestamp for NETw5s64.sys
*** ERROR: Module load completed but symbols could not be loaded for NETw5s64.sys
 
READ_ADDRESS: GetPointerFromAddress: unable to read from fffff800030c10e0
 0000000000000000 
 
CURRENT_IRQL:  2
 
FAULTING_IP: 
hal!HalpDmaNextContiguousPiece+103
fffff800`033f8477 448b08          mov     r9d,dword ptr [rax]
 
CUSTOMER_CRASH_COUNT:  1
 
DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT
 
BUGCHECK_STR:  0xA
 
PROCESS_NAME:  System
 
TRAP_FRAME:  fffff8000475edc0 -- (.trap 0xfffff8000475edc0)
.trap 0xfffff8000475edc0
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000000 rbx=0000000000000000 rcx=0000000000000a50
rdx=0000000000000a50 rsi=0000000000000000 rdi=0000000000000000
rip=fffff800033f8477 rsp=fffff8000475ef58 rbp=0000000000000000
 r8=0000000000000054  r9=0000000000000000 r10=00000000000fffff
r11=fffffa8007d62ea0 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl nz ac po nc
hal!HalpDmaNextContiguousPiece+0x103:
fffff800`033f8477 448b08          mov     r9d,dword ptr [rax] ds:0001:00000000`00000000=????????
.trap
Resetting default scope
 
LAST_CONTROL_TRANSFER:  from fffff80002e88ca9 to fffff80002e89740
 
STACK_TEXT:  
fffff800`0475ec78 fffff800`02e88ca9 : 00000000`0000000a 00000000`00000000 00000000`00000002 00000000`00000000 : nt!KeBugCheckEx
fffff800`0475ec80 fffff800`02e87920 : fffffa80`092f8ee0 fffffa80`0a505720 fffffa80`0903d260 fffff880`04b608b2 : nt!KiBugCheckDispatch+0x69
fffff800`0475edc0 fffff800`033f8477 : fffffa80`0493d850 fffffa80`04f1e5b0 fffff800`033f8524 fffffa80`0a505720 : nt!KiPageFault+0x260
fffff800`0475ef58 fffff800`033f8524 : fffffa80`0a505720 fffffa80`07d62ea0 fffffa80`07d62c60 00000000`00000000 : hal!HalpDmaNextContiguousPiece+0x103
fffff800`0475ef70 fffff800`033fb4fb : fffffa80`0a505720 fffffa80`07d62ea0 fffffa80`0493d850 fffffa80`04e64fd8 : hal!HalpDmaMapScatterTransfer+0x34
fffff800`0475efc0 fffff800`033fb472 : fffffa80`0a505720 fffffa80`0a505718 00000000`00000054 00000000`00000000 : hal!HalpMapTransfer+0x7b
fffff800`0475f050 fffff800`033fa94f : 00000000`00000000 fffff800`033f7fb9 00000000`00000000 00000000`00000003 : hal!IoMapTransfer+0x8e
fffff800`0475f090 fffff800`033faafd : fffffa80`07366050 fffffa80`07d62ea0 00000000`00000001 fffffa80`00000000 : hal!HalpAllocateAdapterCallback+0xc7
fffff800`0475f130 fffff800`033fb156 : fffffa80`07d62ea0 00000000`00000500 fffffa80`07d62e00 00000000`00000004 : hal!IoFreeAdapterChannel+0x109
fffff800`0475f160 fffff800`033fa71f : fffffa80`06f51010 00000000`000005fc fffffa80`07d62ea0 fffffa80`089fcdc0 : hal!HalAllocateAdapterChannel+0x11a
fffff800`0475f1a0 fffff880`0187b7c0 : fffffa80`0449ced0 fffffa80`07ccf050 fffffa80`000000a0 fffff880`04d37c40 : hal!HalBuildScatterGatherList+0x2f3
fffff800`0475f210 fffff880`04af376b : fffffa80`073661a0 00000000`00000000 fffffa80`0492ec60 fffff880`04c98aa0 : ndis!NdisMAllocateNetBufferSGList+0x110
fffff800`0475f2b0 fffffa80`073661a0 : 00000000`00000000 fffffa80`0492ec60 fffff880`04c98aa0 00000000`00000000 : NETw5s64+0xb76b
fffff800`0475f2b8 00000000`00000000 : fffffa80`0492ec60 fffff880`04c98aa0 00000000`00000000 000005fc`00000100 : 0xfffffa80`073661a0
 
 
STACK_COMMAND:  kb
 
FOLLOWUP_IP: 
NETw5s64+b76b
fffff880`04af376b ??              ???
 
SYMBOL_STACK_INDEX:  c
 
SYMBOL_NAME:  NETw5s64+b76b
 
FOLLOWUP_NAME:  MachineOwner
 
MODULE_NAME: NETw5s64
 
IMAGE_NAME:  NETw5s64.sys
 
DEBUG_FLR_IMAGE_TIMESTAMP:  4aafedb7
 
FAILURE_BUCKET_ID:  X64_0xA_NETw5s64+b76b
 
BUCKET_ID:  X64_0xA_NETw5s64+b76b

Update: 2011-05-28
Here is a new BSOD I had on a new computer (Core i7 with SSD). It happened while I was out. No idea what caused it, this might come from an error of the paging file due to a hard-disk problem.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced.  This cannot be protected by try-except,
it must be protected by a Probe.  Typically the address is just plain bad or it
is pointing at freed memory.
Arguments:
Arg1: fffff900c9b33048, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffff96000296614, If non-zero, the instruction address which referenced the bad memory
	address.
Arg4: 0000000000000000, (reserved)
 
Debugging Details:
------------------
 
 
Could not read faulting driver name
 
READ_ADDRESS: GetPointerFromAddress: unable to read from fffff80002cb6100
 fffff900c9b33048 
 
FAULTING_IP: 
win32k!GreTransferSpriteStateToDwmState+198
fffff960`00296614 8b4020          mov     eax,dword ptr [rax+20h]
 
MM_INTERNAL_CODE:  0
 
CUSTOMER_CRASH_COUNT:  1
 
DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT
 
BUGCHECK_STR:  0x50
 
PROCESS_NAME:  dwm.exe
 
CURRENT_IRQL:  0
 
TRAP_FRAME:  fffff880087be740 -- (.trap 0xfffff880087be740)
.trap 0xfffff880087be740
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=fffff900c9b33028 rbx=0000000000000000 rcx=000000000000037a
rdx=fffff900c037f3e8 rsi=0000000000000000 rdi=0000000000000000
rip=fffff96000296614 rsp=fffff880087be8d0 rbp=0000000000000001
 r8=0000000000000001  r9=000000000000ff7f r10=000000000002fe7d
r11=fffff880087be8b0 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei ng nz na po nc
win32k!GreTransferSpriteStateToDwmState+0x198:
fffff960`00296614 8b4020          mov     eax,dword ptr [rax+20h] ds:fffff900`c9b33048=????????
.trap
Resetting default scope
 
LAST_CONTROL_TRANSFER:  from fffff80002a32a0c to fffff80002a86d00
 
STACK_TEXT:  
fffff880`087be5d8 fffff800`02a32a0c : 00000000`00000050 fffff900`c9b33048 00000000`00000000 fffff880`087be740 : nt!KeBugCheckEx
fffff880`087be5e0 fffff800`02a84e2e : 00000000`00000000 fffff900`c9b33048 fffff900`c0123000 00000000`00000000 : nt! ?? ::FNODOBFM::`string'+0x4621f
fffff880`087be740 fffff960`00296614 : 00000000`00000000 00000000`00000001 fffff900`c5037b50 00000000`00000000 : nt!KiPageFault+0x16e
fffff880`087be8d0 fffff960`00211bfe : fffff880`00000001 fffff800`00000001 00000000`000000b8 00000000`00000143 : win32k!GreTransferSpriteStateToDwmState+0x198
fffff880`087be9e0 fffff960`00212633 : 00000000`00000000 fffff880`087beb60 00000000`c0000001 fffff880`087beb60 : win32k!zzzComposeDesktop+0x52
fffff880`087bea80 fffff960`001a7333 : 00000000`00000000 00000001`0000a3b0 00000000`00300000 00000000`00000000 : win32k!zzzDwmStartRedirection+0xbf
fffff880`087beab0 fffff800`02a85f93 : fffffa80`0ed55b60 fffff880`087beb60 fffffa80`0e0ae3e0 00000000`00000000 : win32k!NtUserDwmStartRedirection+0x6b
fffff880`087beae0 00000000`77825f7a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`023af848 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x77825f7a
 
 
STACK_COMMAND:  kb
 
FOLLOWUP_IP: 
win32k!GreTransferSpriteStateToDwmState+198
fffff960`00296614 8b4020          mov     eax,dword ptr [rax+20h]
 
SYMBOL_STACK_INDEX:  3
 
SYMBOL_NAME:  win32k!GreTransferSpriteStateToDwmState+198
 
FOLLOWUP_NAME:  MachineOwner
 
MODULE_NAME: win32k
 
IMAGE_NAME:  win32k.sys
 
DEBUG_FLR_IMAGE_TIMESTAMP:  4d6f104c
 
FAILURE_BUCKET_ID:  X64_0x50_win32k!GreTransferSpriteStateToDwmState+198
 
BUCKET_ID:  X64_0x50_win32k!GreTransferSpriteStateToDwmState+198

Update: 2011-06-05

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced.  This cannot be protected by try-except,
it must be protected by a Probe.  Typically the address is just plain bad or it
is pointing at freed memory.
Arguments:
Arg1: fffff900c86ce7a0, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffff9600070dc51, If non-zero, the instruction address which referenced the bad memory
	address.
Arg4: 0000000000000000, (reserved)
 
Debugging Details:
------------------
 
Unable to load image \SystemRoot\System32\cdd.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for cdd.dll
*** ERROR: Module load completed but symbols could not be loaded for cdd.dll
 
Could not read faulting driver name
 
READ_ADDRESS: GetPointerFromAddress: unable to read from fffff80002cc6100
 fffff900c86ce7a0 
 
FAULTING_IP: 
cdd+dc51
fffff960`0070dc51 483b9080070000  cmp     rdx,qword ptr [rax+780h]
 
MM_INTERNAL_CODE:  0
 
CUSTOMER_CRASH_COUNT:  1
 
DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT
 
BUGCHECK_STR:  0x50
 
PROCESS_NAME:  csrss.exe
 
CURRENT_IRQL:  0
 
TRAP_FRAME:  fffff8800d2dcf10 -- (.trap 0xfffff8800d2dcf10)
.trap 0xfffff8800d2dcf10
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=fffff900c86ce020 rbx=0000000000000000 rcx=fffff900c59be4b0
rdx=fffffa800e9a6790 rsi=0000000000000000 rdi=0000000000000000
rip=fffff9600070dc51 rsp=fffff8800d2dd0a0 rbp=0000000000000001
 r8=0000000000000000  r9=fffff9600032e9f8 r10=0000000000009a68
r11=fffff8800d2dd0c0 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
cdd+0xdc51:
fffff960`0070dc51 483b9080070000  cmp     rdx,qword ptr [rax+780h] ds:6700:fffff900`c86ce7a0=????????????????
.trap
Resetting default scope
 
LAST_CONTROL_TRANSFER:  from fffff80002a42a0c to fffff80002a96d00
 
STACK_TEXT:  
fffff880`0d2dcda8 fffff800`02a42a0c : 00000000`00000050 fffff900`c86ce7a0 00000000`00000000 fffff880`0d2dcf10 : nt!KeBugCheckEx
fffff880`0d2dcdb0 fffff800`02a94e2e : 00000000`00000000 fffff900`c86ce7a0 fffff900`c4823e00 fffff900`c00f1010 : nt! ?? ::FNODOBFM::`string'+0x4621f
fffff880`0d2dcf10 fffff960`0070dc51 : 00000000`00000000 00000288`00000000 00003378`00000000 00000000`00000001 : nt!KiPageFault+0x16e
fffff880`0d2dd0a0 00000000`00000000 : 00000288`00000000 00003378`00000000 00000000`00000001 00000000`00000820 : cdd+0xdc51
 
 
STACK_COMMAND:  kb
 
FOLLOWUP_IP: 
cdd+dc51
fffff960`0070dc51 483b9080070000  cmp     rdx,qword ptr [rax+780h]
 
SYMBOL_STACK_INDEX:  3
 
SYMBOL_NAME:  cdd+dc51
 
FOLLOWUP_NAME:  MachineOwner
 
MODULE_NAME: cdd
 
IMAGE_NAME:  cdd.dll
 
DEBUG_FLR_IMAGE_TIMESTAMP:  0
 
FAILURE_BUCKET_ID:  X64_0x50_cdd+dc51
 
BUCKET_ID:  X64_0x50_cdd+dc51
 
Followup: MachineOwner
---------
 
rax=0000000000000000 rbx=54bfbec100000000 rcx=0000000000000050
rdx=fffff900c86ce7a0 rsi=0000000000000001 rdi=0000000000000000
rip=fffff80002a96d00 rsp=fffff8800d2dcda8 rbp=fffff8800d2dce00
 r8=0000000000000000  r9=fffff8800d2dcf10 r10=0000000000000000
r11=0000000000000000 r12=fffffa800e9a6790 r13=fffff900c86ce7a0
r14=fffff880049f3c00 r15=00000000000001f2
iopl=0         nv up ei ng nz ac po cy
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00000297
nt!KeBugCheckEx:
fffff800`02a96d00 48894c2408      mov     qword ptr [rsp+8],rcx ss:0018:fffff880`0d2dcdb0=0000000000000050
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffff880`0d2dcda8 fffff800`02a42a0c : 00000000`00000050 fffff900`c86ce7a0 00000000`00000000 fffff880`0d2dcf10 : nt!KeBugCheckEx
fffff880`0d2dcdb0 fffff800`02a94e2e : 00000000`00000000 fffff900`c86ce7a0 fffff900`c4823e00 fffff900`c00f1010 : nt! ?? ::FNODOBFM::`string'+0x4621f
fffff880`0d2dcf10 fffff960`0070dc51 : 00000000`00000000 00000288`00000000 00003378`00000000 00000000`00000001 : nt!KiPageFault+0x16e (TrapFrame @ fffff880`0d2dcf10)
fffff880`0d2dd0a0 00000000`00000000 : 00000288`00000000 00003378`00000000 00000000`00000001 00000000`00000820 : cdd+0xdc51

Source : IFoundAFix

Dom2Dom

UPDATE (25 July 2013) :
I closed the project because it was consuming a lot of resources for not special result. It was merely a test of cassandra. BTW, Cassandra didn’t have any problem handling the load generated by this project. It went very smoothly.

UPDATE (07 Nov 2011) :
I reworked the same project with a totally different architecture. I used Java/Servlet/Glassfish + Cassandra. It’s just a test project to see how I could apply this kind of NoSQL DB to other projects, so it’s really simple.

UPDATE (07 Nov 2011) :
I reworked the same project with a totally different architecture. I used Java/Servlet/Glassfish + Cassandra. It’s just a test project to see how I could apply this kind of NoSQL DB to other projects, so it’s really simple.

UPDATE (02 Oct 2010) :
Service is available here. Here is an example with Apple.

This is an experimental little project of mine. The goal is to be able to tell what domain are hosted on the same hosts as an other domain. Some services are already offering it but they do a very crappy job. This service will do only that.

The program just goes from links to links to find new domain names. It stays as less possible on each domain, it doesn’t store any information other than the domain name.

The only address I did put in the program is the address of this blog. All the other were discovered.

Please tell me if you would also be interested by this service.

06/07/10 :
In less than 1 week, the program already collected a little bit more than 400 000 hosts (and I think a little bit more than 350 000 domains), and there really are a lot of porn sites.
I think I will start the DNS requesting part in two weeks and the little webinterface two weeks later. I could do it sooner but results would be very crappy anyway.

10/07/10 :
We’ve now reached 1 million indexed hosts.

12/07/10 :
I’ve added the DNS requesting code. It’s working fine (it’s much easier to do and maintain). It has indexed 60 000 hosts. I’ve made a little web interface but I’m not giving it right away because the SQL requests aren’t optimized yet (but no worries, the database is).

15/07/10 :
1.650 M hosts indexed
420 k have been ip linked
136 k different IP addresses have been found

21/07/10 :
2.5 M hosts indexed
1.0 M hosts IP linked
340 k IP addresses

02/08/10 :
I fixed the program. It is now identified as “Dom2Dom/0.1.3865.36392_2010-08-01_20:13:04”. It can’t make too much requests on the same server now (I limited it explicitely) and it should crawl the web more efficiently.
Statistics are :
3.7 M hosts indexed
(almost) all hosts IP linked
900 k IP addresses

04/08/10 :
4.1 M hosts indexed
all hosts IP linked (and it’s starting to relink old hosts to their potentially changed IP adresses)
957 k IP adresses

20/08/10 :
5.8 M hosts indexed
all hosts IP linked
1.2 M IP adresses

28/08/10 :
I still can’t find time to do the webinterface but the program and the great comments continue.
I took ten minutes to make a little stats page if you’re interested :
http://dom2dom.webingenia.com/stats