Sweet Syntactic Sugar

SPAM email – spreading classical literature?

2011-06-25T04:20:00.001-07:00

SPAM email has gotten quite smart in bypassing various spam filters, the funniest being the use of snippets from works of literature to pad the text, to make it look more “real”. So it will start off by offering to increase the size of certain appendages, but then suddenly break into a snippet of Sherlock Holmes. Sometimes these make you go “I wonder what book this comes from, I would love to read it!” – e.g. today’s SPAM from my inbox comes courtesy of two classics of literature:

The first part apparently comes from “Dracula” (what CAN’T Google find?):

"You are quite right. I did not trust you because I did not know you. But I know you now, and let me say that I should have known you long ago. I know that Lucy told you of me. She told me of you too. May I make the only atonement in my power? Take the cylinders and hear them. The first half-dozen of them are personal to me, and they will not horrify you. Then you will know me better. Dinner will by then be ready. In the meantime I shall read over some of these documents, and shall be better able to understand certain things."

The other from “Gulliver’s Travels”:

“And, therefore, when he was again about to move, I screamed as loud as fear could make me. Whereupon the huge creature trod short, and looking round about under him for some time, at last espied me as I lay on the ground. He considered awhile, with the caution of one who endeavors to lay hold on a small dangerous animal in such a manner that it shall not be able either to scratch or to bite him, as I myself have sometimes done with a weasel in England.”

Now I imagine they’ve had go from doing full paragraphs ripped out of Project Gutenberg either to chopped sentences either because filters in places like GMail are smart enough to look for longer Gutenberg matches, or to save space. Or both. Would love to find a site charting the evolution of countermeasures employed in SPAM to bypass various filters. Maybe the final step in SPAM will be the evolution of AI to appear like a genuine unique email about a subject?

.NET decompilers–or “Attack of the (currently inferior) Clones”

2011-05-15T11:49:00.001-07:00

RedGate’s Reflector tool has been ridiculously popular amongst .NET developers for a long time. It has shed light on all kinds of dark .NET corners in both the shipped .NET assemblies and third party ones. Heck, I’ve even used it to try to make sense of poorly obfuscated ones, with surprisingly positive results (DevArt dotConnect, I’m looking at you!).

But then we were all cast out of Eden. Free Reflector no longer exists, and they want you to pony up $35 to get back in. I tried to hold on to the old versions for as long as possible:

But eventually decided to have a look at Reflector 7, which I’ve now reached the end of the 14-day trial of:

So let’s have a look at the contenders for the “.NET Decompiler King” crown, that have surfaced from the nerd rage that bathed the web after the announcement of the demise of free Reflector.

A good list of possibilities can be found in this StackOverflow discussion.

I’ll look at the, what appear to me, top 3 offerings: ILSpy, which is open source, JustDecompile from Telerik (free but closed), and dotPeek from JetBrains (free but closed):

ILSpy:

Interestingly, ILSpy lists .NET4 as a prerequisite. Meh. Whatever, I’m hoping it’s an indicator of use of more “interesting” coding patterns.

It’s a bit rough at the moment, but runs fast, and is pretty stable. Navigation is not as fluid as in Reflector.

Lambda support is as bad as it was in Reflector 6, but is clearly on the roadmap on the ILSpy website.

This has promise, but is lacking in the usability department.

JustDecompile:

Overall, the look is very much like Reflector - types are hyperlinked, you can expand all members of a class etc. Other assemblies are pulled in a needed if it can resolve them. One annoying thing was that calls to "base" constructors weren't hyperlinked - something that's annoying if there are a lot of constuctor overloads.

Overall impression is one of extreme slowness. Assemblies take ages to load, everything is quite painfully slow. But then again, it’s a young product.

Has nifty search, allowing to find usages. A nice improvement that can already be seen is the ability to filter by only Read or Write usages. Handy when you are trying to find possible culprits that are setting some member for example. Although it didn't quite work as expected - as in, it didn't work for me.

At the moment, the product seems a very sluggish, and a fair few crashes. Also, from the various errors and the presence of the DLL in the directory, we can tell that Mono.Cecil is being used. Refreshingly, at least for the moment, the application does not seem to be obfuscated, like Reflector was. Anyhow, the product has a long way to go. Until I see refinements like bookmarks, ability to create deeper navigation trees (in Reflector you could do deep investigations of using "used by/called by" to visualise a deep call tree you're interested in, expanding more and more nodes to reveal a long chain of callers to establish a relationship between two modules).

Lambda handling is poor as well:

IEnumerable<DOCUMENT> dOCUMENTs = ParameterExpression[] parameterExpressionArray3 = new ParameterExpression[1].Where(Expression.Lambda(parameterExpressionArray3[0] = parameterExpression, parameterExpressionArray3));

As for Add-Ins, which in my opinion endeared Reflector to the community, I posted a question on the Telerik forums (which are already full of bug reports and feature requests, only a few days after release) and quickly got the response that "This is definitely an option that we'll target for the feature version of the product". So their heart seems to be in the right place.

dotPeek:

JetBrains has just released a very early version of their dotPeek software, and it's a little, well, undercooked. They must be coming from the "release early, release often" school of development:

There appear to be various errors sprinkled through the code, and the treatment of lambdas is by far the least readable. Navigation and search are very functional, as I expect they’re almost straight out of ReSharper.

Conclusion and musings:

Looking at these three, it seems to me that so far we're quite far off getting a decent replacement for Reflector. Although the announcement of moving to a paid model has created this Cambrian explosion of .NET decompilers, the current offerings seem to indicate that it's quite difficult to get the level of polish that Reflector had.

An interesting differentiator in the near future for these products I think will be Lambda expressions - given how much more we're using Lambdas in day to day .NET development now, I'd say the final favourite decompiler will be the one that renders them properly, and so far it's a feature only available in Reflector 7 (even though it uses the "query" style syntax for rendering them):

We'll just have to wait and see what happens over the next few months. All of the above tools are quite early on in their development, and I’m guessing they’re all shooting for mimicking Reflector’s abilities one to one, before adding their own flavour on top, but they’re not there yet.

In the meantime, is $35 all that much to spend on a decent tool? My feelings towards this were quite mixed initially, first I thought, well $35 is fine for a trusted tool, but then the feeling of betrayal sets in, especially because there isn’t even a free “lite” version. On the other hand, I think a lot of people would have happily donated the $35, so this should be no different. For example, if work didn’t pay for ReSharper, I’d most likely buy my own license, so what’s $35? It’s all these mixed feelings from suddenly having to pay for something that you had for free. Of course all sins would be forgiven if only RedGate offered a free “lite” version.

It's illogical, Captain.

Software Engineers writing fiction?

2011-03-08T09:58:00.001-08:00

I’ve just noticed that Mark Russinovich (of Sysinternals fame) is publishing a fiction book. There was another CompSci author, Roger Pressman, who’s non-fiction work was part of the reading during my university course, who had also written a fiction book, and despite only having one review on Amazon sounds like an exciting read. Both are techno-thrillers. I’d not expect a historical romance from a software developer!

I guess in both cases the authors have enough “street cred” to take a bit of time off to experiment with this idea of being a published author, and return back to their ordinary careers in case of failure – was this something they always wanted to do, but were too afraid to pursue full time?

One interesting side note about the “Aymara Bridge” is that (from another review I cannot find), it touches on trivalent logic – i.e. instead of True/False, it has an additional third state. Apparently this logic type can be inferred from the language of the Aymara culture - http://aymara.org/biblio/html/igr/igr3.html. The third value indicates a kind of uncertainty that’s absent from “traditional” logic, but is linguistically somewhat natural. Fascinating concept. You can still construct some kind of logic system out of it, it’s just that it’s not terribly suited to computers. Pressman cites this idea as the inspiration for his book.

From the description at Amazon:

The horror of cyberterrorism explodes on the page in Russinovich’s first novel. A plane over the Atlantic suddenly needs to reboot its computer to stay in the air, and the pilots barely avert disaster. A hospital network mixes up patient information, resulting in the death of several people. A law firm, which has lost all of its clients' data and can’t get its system running again, turns to Jeff Aiken, a former government analyst and computer expert. He discovers that all of the crashes are insidiously connected, and an even greater disaster is coming. Computer technospeak is handled with ease by Russinovich, who makes the jargon understandable for nongeeks but does so without losing authenticity. His background at Microsoft ensures that he knows what he’s writing about, but, equally important, he constructs a gripping narrative. A terrifying tale made all the more frightening by our concern that it could offers a glimpse into the future, Russinovich’s thriller just could become one of those books that prompts a real-world response, in this case a wake-up call for greater cybersecurity methods.

From the description at Amazon:

Matt Sousa is young archaeologist with an unusual background. On a dig in the foothills of the Andes mountains, Matt uncovers a mysterious Inca object that appears to be almost 1,000 years old, but there's something odd about it, and it's unlike anything he's ever seen.
RJ Fanler is the deposed founder of a major computer company who believes that its time for the personal computer to morph into something much more advanced. He believes that machine intelligence is the next great leap, but is struggling with how to achieve it.
Marco Paena is a member of the Peruvian terrorist group, Shining Path. He wants to bring down the Peruvian government and needs money to finance a major terrorist strike.
An ancient Inca language—Aymara—puts these men on a collision path.
Set in Peru and the United States, The Aymara Bridge is a technological thriller that melds the mysteries of the Incas with one vision of a not too far distant future.

Humour link - What if Visual Studio had achievements?

2011-01-31T05:45:00.001-08:00

A proposal to have Steam/XBox style “achievements” in Visual Studio – and apparently it’s being made into an extension! I’ve come dangerously close to getting some of these I think. Some examples from the site:

The Enterprise – Build Solution took more than 10 minutes

Highway to Hell – Successfully created a WCF service

Job Security – Written a LINQ query with over 30 lines of code

I’d love one for architectural anti-patterns – a bunch of them were given unofficial names at work, but a bit too rude to put in print.

.NET Reflector 7 Beta is out!

2011-01-11T10:55:00.001-08:00

Haven’t actually played with it, but it has got to be good. Apparently a popular community plugin is now built in.

Get it here, use the link on the right side, the one that asks for you email is version 6.5.

A list of new features available here (which was where I heard the news). There’s apparently a “Write LINQ against loaded assemblies” feature that sounds… intriguing.

Launchy vs Windows 7 Start Menu Search

2011-01-09T09:20:00.001-08:00

For the last few years I’ve been using Launchy for rapid keyboard based application launching. It’s a distant, weaker cousin of QuickSilver on the Mac, with a tiny subset of its functionality, but still fantastic. Some people, when seeing Launchy in action comment that in Windows 7 this functionality is built in via the Start menu search box.

Not quite.

For example typing “vis 10” in Launchy will match “Visual Studio 2010”, however in the Windows Start search thingie, it will not. You would need “vis 2010”.

In the below screenshot, you can see that Launchy matched anything that has a “vis” followed by “08” anywhere in the application name.

In short, the Windows 7 search matches from the start of a word, whereas Launchy is happy for the match to be anywhere within the word. I think that’s a lot more useful, especially in situations where a program name is a concatenation of some words. E.g. I might have a bunch of XML related applications, all with “XML” somewhere in the name… is it “XMLSpy” or “SpyXML” ? The point is, I should be able to look for it by just typing “XML”.

Anyhow, even if you’re on Windows 7, get Launchy. Or if you’ve been spoiled on a Mac, get QuickSilver. You won’t regret it.

Tron Legacy

2010-05-15T23:09:00.001-07:00

Tron is back! Forget Avatar, Iron Man, whatever. This looks awesome:

Visual Studio 2010 – Smartphone abandoned

2010-05-02T05:19:00.001-07:00

Started using the new and shiny VS’10 at work. A glorious IDE. However, Smart-device (PDAs and phones) development is no longer supported, except for the upcoming Windows Mobile 7 – version 6.5 and prior are out. Talk about drawing a line in the sand – presumably they think that in a year Windows Mobile 6.5 will be forgotten. Presumably.

And to think I thought I’d uninstall VS’08. Silly me. Have to keep it around for some time yet I think.

Escalator of dooooom

2010-03-05T22:04:00.001-08:00

Saw this in Melbourne Central station. An unfortunate positioning of visual elements, lends an escalator sign an entirely more sinister meaning (it’s even better with its caption):

Is it only me that sees this as an escalator with a drop into a chasm on the end? Makes me think of “Lemmings”.

Behold, the Meta-stamp

2009-09-12T18:39:00.001-07:00

Checking my paper mail, I noticed that Australia Post is peddling a new stamp. Except it looked like two stamps. Now usually stamps are tiny postcard of cultural history – endangered species, famous people/places and the like, but one day you run out of ideas, and that’s how we get this– the Meta-stamp – a stamp depicting… errrm… a stamp!

I guess it’s kind of similar to when Hollywood remakes classic movies.

Can’t wait for the next one – the stamp of a stamp of a stamp. A bit like the infinite cat project.

Robot arm ride

2009-05-25T23:45:00.001-07:00

This was forwarded to me a little while ago, by now making the rounds through the “tubes”. Not sure if it’s a group of factory workers with too much time on their hands (a spectacular breach of Occupational Health & Safety, surely), or an equally unsafe attraction where you actually pay for the privilege of experiencing your face being millimetres away from being ground into the pavement by a giant robot arm. Step right up.

Printing MSDN Magazine articles from Firefox

2009-05-14T20:51:00.001-07:00

I like to read developer articles. Some of the best dev articles (in my opinion) for the .NET space are published in the MSDN magazine, available online for free. Sometimes it’s convenient to print them, to read on the train, or to make notes in the margins. But Microsoft wants to make that difficult for you.

They do this in two ways:

1. Refusal to remove a print redirection tag in the header of the articles. The print redirection tag exists to suggest to browsers where they should go in order to get a more printer friendly version of the page. For example, say you look at this article. If you are in IE7 or IE8, when you go to File->Print Preview, you’ll see a version of the page that’s entirely unlike what the original page looked like. That was achieved via this directive in the header: <link rel="alternate" media="print" href="/en-us/magazine/dd569757(printer).aspx" />. The browser sees this, takes you to the “print version”, and prints it. Note that Firefox seems to ignore this, and stubbornly refuses acknowledge that the creator of the page went to the effort of providing an optimised print page.

But ignoring that… so far so good – but then the web designers did something weird. The print version of the page (to which you can get by clicking the “Print View” button on the page) has a printing header link… back to itself. This has the effect of cancelling any changes you performed on the page, like expanding code regions etc – the page reloads just before printing.

Now recently, MS seems to have realized this, but their fix was nothing short of bizarre – add a script to automatically expand all the collapsed regions on load. Why does the print view redirect to the print view at all?? Anyhow, the bizarre fix kind of works, in the past you couldn’t actually expand the collapsed regions, as they’d collapse as soon as you tried to print (because the page reloads). Now they’re open by default. But this doesn’t work for Firefox - when you print in Firefox, the regions are collapsed, the script for some reason doesn’t execute, and you’re off trying to expand all the regions by hand.

2. Somebody forgot their stylesheets. The generated page is full of class attributes, but the stylesheet definitions for a lot of them are missing:

There is no differentiation between headings and text. The class attributes are there.

So what to do? Greasemonkey, that’s what. Greasemonkey is an add-on for Firefox, that allows you to execute custom scripts after the page loads. You can do funky things like tweaking the specific HTML elements, adjust styles etc. There is a handy online reference, that gets you started using it in no time. So now every time I visit an MSDN print page, the styles are magically fixed.

The resulting script I present here improves on the page in more ways than the above features – I’ve changed the code font to Courier, as that’s a far more common programmer font family, and removed items that “float”, upsetting printing, as well as removing the Copy Code link.

Here is are some snapshots of the results, side by side (“before” on the left, “after” on the right):

Header and “Sidebar” entries (I’ve moved the “Sidebar” element to appear inline, for better printing):

Section headings and code snippets (no need for the grey background in my opinion, or the “Copy Code “link):

The entire script is below, enjoy (feel free to comment on any styles that I might have missed):

The obvious barrier of entry is that you have to know how to actually add a script to Greasemonkey. An exercise left to the reader.

// ==UserScript==
// @name MSDN Magazine
// @namespace a
// @include http://msdn.microsoft.com/en-*/*(printer).aspx
// ==/UserScript==
//this function is straight from the DiveIntoGreasemonkey tutorial
function addGlobalStyle(css) {
var head, style;
head = document.getElementsByTagName('head')[0];
if (!head) { return; }
style = document.createElement('style');
style.type = 'text/css';
style.innerHTML = css;
head.appendChild(style);
}
//get rid of the copy code header
addGlobalStyle(".CodeSnippetTitleBar {display: none !important;}");
//code snippets
addGlobalStyle("pre.libCScode {background:#FFFFFF none repeat scroll 0 0 !important;border-top:1px solid #C8CDDE !important;border-bottom:1px solid #C8CDDE !important;border-left:1px solid #C8CDDE !important;border-right:1px solid #C8CDDE !important;display:block !important;font-family:courier,monospace !important;margin:0 0 10px !important;padding-left:5px !important;padding-right:5px !important;padding-top:5px !important;}");
addGlobalStyle(".ArticleTypeTitle {color:#008080 !important;font-family:'Segoe UI',Arial !important ;font-size:14px !important;font-style:normal !important ;font-variant:normal !important ;font-weight:bold !important;margin-top:3px !important;text-transform:none !important;}");
//article section title
addGlobalStyle(".ColumnTypeTitle, .FeatureSmallHead {border-bottom:2px solid #008080 !important;color:#003399 !important;font-family:'Segoe UI',Arial !important;font-size:36px !important;font-style:normal !important;font-variant:normal !important;font-weight:normal !important;line-height:36px !important;margin-bottom:4px !important;padding-bottom:2px !important;text-decoration:none !important;text-transform:uppercase !important;}");
addGlobalStyle(".ColumnTypeSubTitle, .FeatureHeadline {color:#000000 !important;font-family:'Segoe UI',Arial !important;font-size:20px !important;font-style:normal !important;font-variant:normal !important;font-weight:normal !important;line-height:20px !important;margin-bottom:8px !important;text-decoration:none !important;}");
//author name in header
addGlobalStyle(".ColumnByLine, .FeatureByLine {color:#000000 !important;font-family:'Segoe UI',Arial;font-size:16px !important;font-style:normal;font-variant:normal;font-weight:normal !important;text-decoration:none !important;}");
//sidebar header
addGlobalStyle(".SidebarHeadline {color:#008080;font-family:'Segoe UI',Arial;font-size:18px;font-weight:bold;line-height:20px;}");
//recently they've started using floating divs as well this gets rid of them
addGlobalStyle("div {float:none !important; height:auto !important; width:auto !important;}");
//show all hidden panels
var allLinks, thisLink;
allLinks = document.evaluate(
"//div[@style]", //any style, as they start off invisible
document,
null,
XPathResult.UNORDERED_NODE_SNAPSHOT_TYPE,
null);
for (var i = 0; i < allLinks.snapshotLength; i++) {
thisLink = allLinks.snapshotItem(i);
thisLink.style.display='block';
}

MIX 09 Videos

2009-04-05T01:46:00.001-07:00

The MIX 09 conference is well over in the US, and all the content is now available online, all the presentations were recorded. You can even get them all on the one page, instead of search the presentations for each day:

http://videos.visitmix.com/MIX09/All

iPod lap timer parser

2009-03-23T06:16:00.001-07:00

The iPod Nano is a beautiful thing. That is unless you want to do something custom. So it is with the lap timer in it. There is no obvious way of synchronising the timer logs with the computer. Even if there is a way in ITunes (I’m pretty sure there isn’t), the point is moot, as I don’t use ITunes (instead I’m using the wonderful ml_ipod plugin for Winamp). This is “suboptimal” if you want to, for example, use the lap timer to chart your jogging progress over the course of a year. You want to crunch those stats in Excel or some such, you certainly aren’t going to copy them from the screen.

The information on the web is a little sparse on this matter, but there are a number of applications that will let you do this. The one that caught my eye was http://wafflesoftware.net/ipodtimer/ which luckily provided source code (Mac OS X only – aren’t we feeling exclusive). A relatively trivial exercise in parsing. The timer format was loosely described in the TimerFormat.txt in the zipped source. At least detailed enough for me to whip up an implementation in C#. Hence this post.

The underlying format is a binary file. Time to dust off .NET’s BinaryReader. The below method ReadTimerEntries(string filepath) will read in an iPod timer file (on my Nano it’s /IPod_Control/device/timer) and produce a list containing the date of the recording, and the associated laps. From there you have all the info you need. Anyway, it works for me, on an iPod Nano (3rd gen). I don’t imagine that the file structure changes much though.

public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
private void button1_Click(object sender, EventArgs e)
{
OpenFileDialog ofd = new OpenFileDialog();
ofd.FileName = Application.StartupPath;
DialogResult dr =ofd.ShowDialog();
if (dr == DialogResult.OK)
{
List<TimeEntry> entries = ReadTimerEntries(ofd.FileName);
foreach (TimeEntry entry in entries)
{
System.Diagnostics.Debug.WriteLine(entry.EntryDate.ToString());
foreach (TimeSpan ts in entry.Laps)
{
System.Diagnostics.Debug.WriteLine(ts.ToString());
}
}
}
}
public List<TimeEntry> ReadTimerEntries(string filePath)
{
List<TimeEntry> results = new List<TimeEntry>();
//open the file
var stream = File.OpenRead(filePath);
BinaryReader br = new BinaryReader(stream);
br.ReadBytes(4); //start file
while (!br.ReadBytes(4).SequenceEqual(new byte[] { 0x13, 0, 0, 0xC0 }))
{
//first 4 bytes are already consumed
ReadRound(br, results);
}
return results;
}
private void ReadRound(BinaryReader br, List<TimeEntry> itemList)
{
br.ReadBytes((16 * 5));// guff
br.ReadBytes(8);// start date
byte seconds = br.ReadByte();
byte minutes = br.ReadByte();
byte hour = br.ReadByte();
byte day = br.ReadByte();
byte month = br.ReadByte();
int year = br.ReadInt16();
br.ReadByte(); //unknown use
TimeEntry entry = new TimeEntry();
entry.EntryDate = new DateTime(year, month, day, hour, minutes, seconds);
itemList.Add(entry);
br.ReadBytes(4); //end date
br.ReadBytes(4); //start laps
while (!br.ReadBytes(4).SequenceEqual(new byte[] { 0x10, 0, 0, 0xC0 }))
{
br.ReadBytes(4); //start lap
int lapMilliseconds = br.ReadInt32(); //4 bytes
TimeSpan lapTime = TimeSpan.FromMilliseconds(lapMilliseconds);
entry.Laps.Add(new TimeSpan(0, 0, 0, 0, lapMilliseconds));
br.ReadBytes(4); //end lap
}
br.ReadBytes(4); //end round
}
}
public class TimeEntry
{
public DateTime EntryDate;
public List<TimeSpan> Laps = new List<TimeSpan>();
}

Terminator Salvation – coming to cinemas May 21

2009-03-22T00:40:00.001-07:00

Ahh, the Terminator movie that we’ve always wanted – entirely set in the future, focusing on the war between man and machine. I’ve always wondered when they would show the future they hinted at, a void slightly filled by the Terminator TV series (Sarah Connor Chronicles). Another point of note is that this coming instalment is but the beginning of a Terminator revival – part of a new trilogy apparently. And you can’t go wrong with Christian Bale as John Connor.

Reading database tables in Powershell

2009-03-05T03:42:00.001-08:00

Firstly, a correction. Last time I posted about Powershell, I came up with something that did the job, but was, well, a tat long winded. Kind of like building a skyscraper so that you can store garden tools in the basement. Basically I didn’t realise that the Sort cmdlet already had a –Unique switch. Thanks to Stephen Mills for pointing it out:

import-csv c:\mydata.csv | select Category, Subcategory | sort category, subcategory -Unique | Group-Object -Property category

Now to the business at hand – reading database tables. Consider a situation where you’re asked to read in the data in a database table into some readable form, for reference, printing or whatnot. Rather than using SQL Management Studio to query the results, selecting the entire results grid, copying it, pasting into Excel (if available on the same machine), or into a text file, then getting it into Excel – you could just use Powershell to connect to the DB and dump out the results into a HTML table. Behold:

# Parse Database tables
# This will connect to a database, do a "select *" on a table or view, and produce a html file with the data in a table.
# Note: don't forget to loosen up your execution policy "Set-ExecutionPolicy Unrestricted"
# and the connection string is ADO style : "Data Source=database;Initial Catalog=oesc_offerman;User Id =username;Password=password;Trusted_Connection=False;"
param (
$connectionString = $(throw "Specify connection string" ),
$tableName = $(throw "Specify a table or view name"),
$outputPath = $(throw "Specify output path")
)
echo ("Processing " + $tableName)
$table = new-object System.Data.DataTable;
$sqlConn = new-object System.Data.SqlClient.SqlConnection($connectionString);
$sqlConn.Open();
$adapter = new-object System.Data.SqlClient.SqlDataAdapter(("select * from " +$tableName),$sqlConn);
$adapter.Fill($table);
$sqlConn.Close();
$table.Rows | ConvertTo-Html | out-file $outputPath

Fish blimp

2009-03-04T03:11:00.001-08:00

Had to share this one – from a homemade airship competition, footage of a small blip that uses a tail fin for propulsion – it looks very graceful, especially for a blimp – sure it’s not encumbered, with no wind, but it’s still mesmerising to watch:

Air Art from flip on Vimeo.

“SELECT DISTINCT” in Excel and Powershell

2009-02-21T18:33:00.001-08:00

Just recently at work we had to import a large dataset into a database, but two of the columns were “category” and “subcategory”. Naturally we wanted these in a separate table, so we needed a way of parsing through the existing data and get the unique categories, and the unique matching subcategories for each.

Excel:

Firstly, there’s the Excel ‘07 way (most likely also possible in other versions too):

Select the two columns that hold your categories and subcategories. Go to the “Advanced” menu item in Filtering:

The presented dialog has the option of “Unique Records Only”

Done, it’s the equivalent of doing a “distinct” in SQL. Now sort the data, and you’re ready to go.

Powershell:

But no discussion of parsing/manipulating data is complete without mentioning Powershell – surely we can do this in PS. Let’s assume that the data was saved in csv form (you could always save your Excel spreadsheet as .csv).

Luckily, there’s already a command that is able to parse csv’s for us: “Import-Csv”. This cmdlet will parse the csv and create objects that have properties named after the columns – in the below image you can see that the object has a “Category” and “SubCategory” property.

Now that we have this collection of objects, use your favourite way of flushing out duplicates. I’m keen on building up a new list, checking on each insert that the same entry doesn’t exist:

$temp = @{}
$data | foreach {
if ($temp.ContainsKey($_.Category))
{
#check if subcategory exists, insert subcategory if doesnt
if ($temp[$_.Category].ContainsValue($_.Subcategory) -eq $false)
{
$temp[$_.Category].Add($_.Subcategory, $_.Subcategory);
}
}
else
{
#put it in:
$temp.Add($_.Category, @{$_.Subcategory = $_.Subcategory});
}
}

This is all well and good, but there are issues. The "Category” property names are hardcoded, and it depends on an existing $data variable. Ideally we want a cmdlet that can support piping, and will let us specify the names of the columns.

Luckily for us, Powershell is able support the following constructs:

What happened there is by putting the variable in brackets, PS will evaluate the variable and put it in the script. I was able to create a variable $somestring, which had a string value of “Subcategory”. When I used it in brackets, it expanded out to “Subcategory” in the script, and functioned in the same way as $data[2].Subcategory. Neat. This is an invaluable feature in scripting.

So we replacing all places where we previously had “Category” and “Subcategory” with expansions, and add them as required parameters.

Last thing, we need to get rid of the assumption of pre-existing $data variable. Ideally the input will be piped in. No problem, the reserved $input variable is the piped in parameters.

So now we can call it like so:

import-csv c:\MyTest.csv | c:\categoryParser.ps1 "Category" "Subcategory"

The final script looks like this:

param (
$categoryName = $(throw "Specify category column name" ),
$subcategoryName = $(throw "specify subcategory name")
)
$temp = @{}
$input | foreach {
if ($temp.ContainsKey($_.($categoryName)))
{
#check if subcategory exists, insert subcategory if doesnt
if ($temp[$_.($categoryName)].ContainsValue($_.($subcategoryName)) -eq $false)
{
$temp[$_.($categoryName)].Add($_.($subcategoryName), $_.($subcategoryName));
}
}
else
{
#put it in:
$temp.Add($_.($categoryName), @{$_.($subcategoryName) = $_.($subcategoryName)});
}
}
#dump out the output
$temp

And produces something like this, a hashtable of objects each of which has the category name, and a hashtable of subcategories.

From here it would be easy to traverse them all with two nested foreach loops, and do whatever with them, like generating INSERT statements.

Minotaur in a China Shop

2009-02-06T18:22:00.001-08:00

I have a soft spot for independent games – possibly the last bastion of innovation in computer gaming. Came across a very amusing little independent game, about a Minotaur running a fine china shop. The game is 3D, isometric perspective. Runs in the browser. The minotaur isn’t exactly light on his feet, and handles like fridge - thus there are two ways to win – trash your own shop to get an insurance payoff, or service customers. Give it a go – most amusing.

http://blurst.com/minotaur-china-shop/play

Minotaur China Shop Trailer from Flashbang Studios on Vimeo.

On Precision

2009-01-31T21:55:00.001-08:00

Do YOU know your car’s height down to the millimetre? You’d better…

Saw this in Southbank:

Blade Runner 2, Brave New World, Forever War – coming to a cinema near you… umm.. soon…

2009-01-29T01:21:00.001-08:00

In case you missed it, Slashdot had coverage of the fact that a Blade Runner sequel may be in the works, as would a film based on Brave New World (by Aldous Huxley), and Forever War (by Joe Haldeman).

Brave New World didn’t really do much for me, but Forever War, if properly done, could be the serious sci-fi movie people have been waiting for – they seem a bit thin on the ground these days. More importantly, all this is supposedly to be directed by Ridley Scott (director of Blade Runner, Alien, Gladiator and Black Hawk Down among others). Apparently he’s secured the rights to Forever War, and is now looking for a script.

Read all about it: http://entertainment.slashdot.org/article.pl?sid=09%2F01%2F28%2F1355238&from=rss

Chernoff Faces and Data Visualization

2009-01-27T04:47:00.001-08:00

While we might have a good feel for what makes a good UI, few of us give much thought to the niche are of data visualization. Too often we’re content to present users with drab boring datagrids and lists. While on one side there is the issue of familiarity (everyone has seen tables), the usability of grids for slicing and dicing data is limited at best. Filtered columns, sorted columns can only get you so far, and certainly don’t provide you with a “feel” for the data.

Having just finished reading Blindsight by Peter Watts (freely available online – not too bad, with an interesting central idea looking at the potential for self-awareness to be unnecessary for intelligence), one of the characters makes use of Chernoff faces to visualise data.

Chernoff faces are an attempt to make use of the special brain circuitry that we have for recognizing faces in the context of data visualization. The idea is to present the user with a field of stylized faces, associate a specific data dimension to a facial feature, and let the brain do the rest. The below example maps data onto a real geographical space, but that is not strictly speaking necessary, the map could have been a set of two dimensional data of more abstract nature. Using the following key to map facial features:

You can for example sweep your eye across the map looking for all the “big eared” faces to give an idea of crime rate. Or you could look for “frowning faces with big ears” (which would map to high unemployment and high crime rate).

(image from http://gis.esri.com/library/userconf/educ04/papers/pap5000.pdf)

(for another interesting example, go to http://kspark.kaist.ac.kr/Human%20Engineering.files/Chernoff/Chernoff%20Faces.htm)

While not bad on paper, it appears the applications are quite limited, and the data still doesn’t “jump out” at you, as well as the difficult issue of non-intuitiveness when dealing with abstract data sets (what facial feature should be “average sales”?). Your mileage may vary – it still seems quite good at performing visual searches for specific patterns (for example consider the lot of faces with brown skin and short hair – higher percentage of collage graduates, but with lower incomes – what gives?) .

Regardless, this is but one attempt at visualizing data in a non-tabular way. There are other stabs at doing this, some which even add motion to the visualization to suggest another data dimension. Consider “Anymails” where email is visualized as various critters moving around in the visual space – older email moves more slowly, and various characteristics of the email are encoded into the shape/color of the insects:

(from http://carohorn.de/anymails/, there’s also a video of the application in action there, I believe)

Why am I bringing this up? Well it’s all to do with with the trend we’re seeing in UI technologies. Silverlight and WPF are both platforms in which animation and custom drawing are readily available, giving the freedom to easily create “compelling” user interfaces that may give us the ability present data in a more intuitive way, as well as interaction options that would have been too “expensive” to create in the past. This will require more focus on what the users are trying to achieve as opposed to regurgitating the same dropdowns and text-boxes. It will take years for this trend to develop, as the area appears to be largely unexplored in mainstream software design, and would be classified as risky.

Regardless, creative thought in software engineering is back, and we should use the flexibility of SL and WPF to bring users the UIs that will give them truly better productivity, not just proxies of paper-based processes – Minority Report UI here we come!

Functionality vs Prettiness

2009-01-25T17:35:00.001-08:00

Although I’ve traditionally been a proponent of Windows Mobile as a platform for pocket devices, the arrival of the iPhone certainly shakes the foundations of that conviction. Watching now the rapid spread of the iPhone I wonder at how something like that could become so popular. After all, PDA devices that mixed the PDA functionality with a phone are nothing new. Consider the O2 XDA range for example, which has been out for years. Nor is the tilt sensor anything unique – Nokia N95 had done it earlier. GPS – others had it. We can argue that it was all these features finally have come together in one reasonable package, and we’d be right. Mostly. There is however another feature that I think cuts to the bottom of the iPhone sales mystery – the UI. Apple recognises that great functionality only gets you so far – the average user is a sucker for a pretty UI. The iPhone is simply a pleasure to look at and operate. Everything from the sliding menus to the gradient background screams “luxury”.

For a lark, consider a unit/currency conversion application. Let’s ignore multi-touch, GPS and tilt sensors Here’s a screenshot of an existing iPhone app that does this:

And here’s the same functionality, mocked up in a Windows Mobile 5 emulator under Visual Studio 2008, using the Compact Framework:

Sure, I could have made custom controls to beautify the thing, but the point is how far one could get with mostly out of the box tools, and there is no contest. I’d buy the mobile that gives me the great looking UI, given the same functionality. In fact I suspect a lot of people will take a hit on functionality for the sake of snazzy UI. This is one of the reasons Silverlight on Windows Mobile is a big deal, and I look forward to it in “Q1 of 2009”. No doubt MS will be playing catch-up to Apple for a few years.

PS: I don’t actually own an iPhone, the above is strictly based on short impressions, however I’d argue that a lot of purchasing decisions are made that way, for the majority of the shopping populace.

Silverlight 2 for mobile devices

2009-01-06T04:09:00.001-08:00

Just finished watching a presentation from PDC2008, and apparently Silverlight 2 will be available as a tech preview on the mobile in the first quarter of 2009. This is the first time I’ve heard of SL2 being released for mobile. Previous announcements have centered around SL1, not version 2. Version 2 was previously discussed as a distant Alpha, with no indication of timelines. Silverlight 1 always seemed like a sad excuse for a framework, so I can’t say I was looking forward to a mobile release. But the release of SL2 will finally give us the ability to design Iphone looking applications for the mobile.

As expected, designing will be available from Blend as well as from Visual Studio. According to the demo, the development experience will be very similar.

See session PC10 for the announcement. (http://channel9.msdn.com/pdc2008/PC10/)

Performance will of course be an interesting issue, as will the experience on a non touch enabled phone, eg Smartphone.