IronPython Article from September '06

Came across a short piece I wrote about IronPython while looking for something else - it was in response to some questions from Kathleen Richards in September of last year for Redmond Developer News (which I think was for this Article, though none of the content made it in - cest la vie).

At any rate, I figure I might as well post it here, it may be of use to someone... somewhere... who could say?  Looking over it briefly I'm not even sure I agree with some of the points I made any more :P

IronPython Article

How are you using IronPython?

At Syzmk ( we are using IronPython as a rich embedded scripting language within our core product, we expose a large part of the .Net server engine (written in C#) to the IronPython runtime, and offer key extension points which can be hooked into with python code - we chose python because the language is rich, dynamic and doesn't suffer from the verbosity that most of the compiled languages do.   At the moment we are moving beyond using IronPython as just an embedded scripting language and are evaluating how best to implement a number of domain specific languages in it for our server, firstly around the business rules and behavior of our application's core logic, and secondly for performance and diagnostic evaluation and control of the underlying server platform.

Why did you choose this implementation over C python or Jython or
other dynamic languages?

IronPython was attractive because of its first class support for .Net types and especially its excellent support for delegates (and because there wasn't any marshalling) - we were attracted to the python syntax and language structure, more than support for the standard python libraries, so the compatibility sacrifices that were apparent with many C Python libraries, especially during the beta, were of little consequence to us, especially as IronPython has full access to the base class libraries (BCL).

What types of applications are you building?

The core application we have used IronPython for is a server technology, the "Rich Media Processor" (working title) - it's in production, but has only been going to market for the last 6 months.   Effectively you can place it into an organization as a front end processor for rich and largely unstructured messages (think Audio, Video, Email, SMS and MMS messages from phones, images dumped from digital cameras, security camera feeds etc) with the end result being a structured and conforming set of messages and rich content (appropriately formatted images, videos, audio etc) suitable for passing onto line of business systems, or a product like biz talk.

When dealing with transforming this kind of content, python (and IronPython) is a great language to augment the message transformation process - strong string support, access to all the functionality of our underlying application and it's easy to develop and deploy - IronPython never seems to be working against us, I could never claim the same for any scripting language I've implemented into a product before.

Does the .NET environment offer any advantages?

Yes, I think it offers serious advantages for Python (access to the BCL!), developers and customers.

First off the business case for building .Net applications is immediate within our home country of New Zealand, customers are generally expecting to host Microsoft tools and solutions, and our IT industry is geared towards this expectation, there is an implied trust which is worth taking advantage of.

Business cases aside however I think the .Net platform is very engaging for us, our company has a love for technology, and in the last few years the .Net platform has been satisfying that need very well - it offers stability, security and productivity increases that we can't find elsewhere (and certainly not in an unmanaged language), also the language-agnostic approach and ability to build dynamic languages (like IronPython) in the CLR make the choice all that simpler.


We haven't really discovered any disadvantages with .Net or our approach, IronPython does everything we need and then some, as does the .Net platform - however I think it might be some time yet before dynamic languages on the CLR (or any language other then VB.Net or C#) will be acceptable recommendations from the consulting masses - perhaps more a reflection on the lack of education and published case studies I think.

For instance we already have languages competing with C# on the .Net platform, such as Boo (, which offer some advantages over the current version of C# and are very suitable for building domain specific languages (DSL's) - but I believe in many bespoke development houses it would be very hard to get buy-in for using these languages in upcoming projects, even if they will save time and money or just be a better fit for their problems.

Do you think dynamic, object oriented (scripting) languages
(Ruby, Python) will start to play a larger role in enterprise

I think the short answer is yes, the languages are often just a catalyst for methodologies and practices (such as functional programming, or the ruby on rails web development approach) - the elegance, simplicity and speed at which dynamic language guru's can deliver results is enviable for a lot of us who have cut our teeth on C++ & Java and known the other end of the spectrum.

If I am in a corporate environment should I be looking into these

I think at this point it should not be on the top of your list - if you're a Microsoft shop, or in an environment dedicated to employing solutions on top of Microsoft technologies then .Net 3.0 should be your key focus - however I think it's certainly worth having a few dynamic language books at your disposal (perhaps on the bedside table), they make for compelling reading, and will get your initial buy-in, so you feel comfortable considering a dynamic language on the .Net platform as a good idea.

That said, if you're not a Microsoft shop, and particularly if you are committed to open-source technologies I would look seriously, probably first at Ruby, it's very engaging story - though of course you can run IronPython on Mono with a little bit of tweaking ? ... and IronRuby is getting better every day.

What are some of the key issues from an application development

I can only talk from the perspective of deploying IronPython as a scripting engine, hosted within another application, though some of these points will probably apply elsewhere, I think it all comes down to one thing -  being prepared.

First off, you must change your point of view and adapt some of your practices, as they will confound you with the introduction of a scripting language that has access to your own .Net classes.  As you develop classes, try them on for size and make sure the selected scripting language doesn't have trouble accessing them - for instance heavily overloaded .Net methods in IronPython can be a recipe for frustration later on, same goes with using multiple out or ref parameters - that's not to say IronPython can't access them, just that it's awkward and unpleasant (something your scripting language should not be).

Second to this is security - dynamic languages effectively introduce a surface area to your application that didn't exist before - our server allows configuration (including iron python scripts) to be applied via a web service - you want to make absolutely sure there is sufficient trust between endpoints, and don't repeat mistakes of the past with other scripting languages (think injection attacks).

Read More

Rhino Mocks 3.0 beta is out!

It seemed a long time coming, but finally
Rhino Mock's 3.0
is available, which of course also means

Proxy 2 has made it to that sweet generic method support
nirvana as well :)

I don't know where I'd be without these tools, I haven't done a
.Net project without them in the last couple of years... in fact
it's fueled my reluctance over the last couple of years for
returning to bespoke development at any NZ dev shop which might
dictate other (crappier) alternatives ;o)

I also noticed that mono is getting it's C# 3.0 support slowly
... Extension methods are in, as is some Lambda
support... I need to sit down and spend some time thinking about
how I should go about unit testing some of this stuff, it's on my
list of "testing" things to do right after getting to grips with
RhinoMocks 3 and perhaps writing another blog entry on mocking
Base4 with all the new "goodies" - I need to do a monorail/base4
website for another little venture, so it's probably as good a time
as any.

Went for a swim at Omaha today... just can't get enough of the
beach this year - John
turned up at the same time we did, there's only so
many "Not in front of John" jokes I can handle in one day... or a
life time ;o)

Read More

Simplcity vs Ease of Injection...

Found myself writing a simple class for implanting "trial" logic for a client (as in you have 30 days left of your trial etc.) ... it relies on the current date and time, and so you can never really be assured your tests are robust unless you can control the current date & time within the test fixture (in fact any time you find yourself writing DateTime.Now alarm bells should be going off... even worse is logic based on DateTime.Now where it's evaluated more the once for a single function).

In the past I've generally Implemented something like an "IClock" interface, which would have a method like "DateTime GetCurrentTime()" or whatever seemed appropriate to the situation... but I decided to go with the lightweight solution of delegates instead in the morning, if only because I was being lazy I didn't see the need for creating an entire interface for such a trivial requirement... so we have:

private DateTime _trialExpirey;
private int _maxExecutions;
private TimeSpan _trialDuration;
private Func _nowFunction;

public TrialUpdater(DateTime trialExpirey, int maxExecutions, TimeSpan trialDuration)
: this(delegate { return DateTime.Now; }, trialExpirey, maxExecutions, trialDuration)

public TrialUpdater(Func nowFunction, DateTime trialExpirey, int maxExecutions, TimeSpan trialDuration)
if (nowFunction == null) throw new ArgumentNullException("nowFunction");

_nowFunction = nowFunction;
_trialExpirey = trialExpirey;
_maxExecutions = maxExecutions;
_trialDuration = trialDuration;

But I'm starting to wonder if this code doesn't smell a bit if I start using it other places... for instance:

  • Could I supply an alternative delegate via the castle container...?
  • Could I mock out the nowFunction parameter easily using something like RhinoMocks and still have it participate in constraints...?

I'm thinking in the case of mocking it out you would probably need to implement some interfaces which lines up with the generic delegates (Func and Proc etc.) - so I would end mocking out an IFuncEvaluator interface (or in this case IFuncEvaluator) -which would implement a method "T Evaluate();"... and the pass an anonymous delegate which in turn invokes the mocked IFuncEvaluator.

Though maybe I've missed something and mocking out delegates is doable with the current RhinoMocks..?

As for being able to inject a value for the nowFunction parameter, I'm thinking a nice solution would be to actually wire it up to a method on another service declaratively i.e.


Which I believe should be doable using a facility and a type converter... but I'm sure the devil's in the detail :)

At any rate, just a minor distraction, back to work.

As an aside the clocks a pretty boring example, but there are times
when you want to manipulate the time for instance:

  • If you want to manipulate the time (return dates as UTC instead of Local)
  • Clamp the current Time so it's limited to a certain level of accuracy... perhaps you have an external expectation that all dates will be recorded at a 10ms level of accuracy (perhaps you want to align database timestamps with timestamps within your application).
  • Perhaps your application uses the clock to work with snapshots of your domain model in the past.

How many people are actually using anonymous delegates and generic Func / Proc's in their day to day coding for tasks like this?
Read More

LINQ fun - part two

Just a quick post, for part two let's look at grouping our results for display purposes... first off, I grab only the episodes which we think are "valid" (in this case ones which are assigned a series, episode and part number)...

EpisodeParser episodeParser = new EpisodeParser();

YouTubeSearcher searcher = new YouTubeSearcher(DeveloperId);

// get a list of all the parts we are interested in

IEnumerable parts = (from result in searcher.QueryByTags("QI", "Quite", "Interesting")
select episodeParser.Parse(result))
.Where(p => p.SeriesNumber > 0 && p.EpisodeNumber > 0 && p.PartNumber > 0)

Obviously what I've done here is probably rather poor style, surely the where clause could have been inside the select statement??... but I'm just trying to illustrate how you can mix the two notations... next we're going to group the results up using some nested "group by" and object projections... this is disgustingly easy, what more can you say about it.

// group those parts

var allSeries = from part in parts
group part by part.SeriesNumber into series
orderby series.Key
select new { SeriesNumber = series.Key,
Episodes = from episodePart in series
group episodePart by episodePart.EpisodeNumber into episodes
orderby episodes.Key
select new {EpisodeNumber = episodes.Key,
Parts = from chunk in episodes
orderby chunk.PartNumber
select chunk

Great, I think that saved us a lot of time, compared to doing it ourselves in C# 2.0...  let's now generate a little page to view the results - for now I'm just writing code in NUnit fixtures - so we'll use Console.WriteLine... though in a website I would probably just use some quite similar brail template code.

foreach (var series in allSeries)

Series {0}

rn", series.SeriesNumber);
foreach (var episode in series.Episodes)

Episode {0}

rn", episode.EpisodeNumber);

foreach (var author in episode.Parts.GroupBy(p => p.Result.Author))

Author: {0}rn", author.Key);

foreach(var part in author)
YouTubeResult result = part.Result;
Console.WriteLine("part {1}  ", result.Url, part.PartNumber, result.ThumbUrl);

I've also dumped the output from this to an html page, if you'd like to have a look at it.

Next time I might have a look at storing the results of the youtube query with base4 and writing some code for a little daemon which can identify newly posted episode parts on a periodic basis... which should all lead towards implementing the RSS feed capability I discussed in  part one.

But for now I'm off to enjoy the sun!

Read More

LINQ fun - part one

Well, the recent musings of Alex James have caught my interest around LINQ (part one and two) - and so I thought I would start having a play with LINQ and using it to query some well known web sites... and decided to go with dragging some info out of youtube...

Basically, I like QI (also known as Quite Interesting - british comedy show) - and there's a lot of episodes on youtube, querying on the tags "QI", "Quite" & "Interesting" returns about 1000 results... almost every result represents a part of an episode... and most people posting are kind enough to include something in the title or description of the video which lets you know which episode and series it belongs to...

However it's going to take some effort to actually start watching at series 1, episode 1 and work your way through the episodes in the right order by endlessly browsing youtube's search results... unless we start "value adding" - creating an "episodic" view over youtube's data.

At this point you could put your magic hat on and wish for features from you're youtube "episodes" site:

  • Perhaps an RSS feed notifying me of new episodes as they're listed on youtube daily.

  • A nice way to see a series, it's episodes, and each part that I need to play.
  • Maybe some aggregation of episode summaries from another web site for each episode (in this case I'm thinking maybe TVRage.Com...)

So you can quickly see that we can start making something virtually out of nothing, like MacGyver... but much lamer and without the hair.

First things first, I started having a look at YouTube's API - you can search by tags and get plenty of info back from their REST web service, but you can only get 20 results per page, 20 results isn't much cop so I built a simple class which gives access to the whole set of search results as an IEnumerableYouTubeResult> - here's the code for that:

public class YouTubeSearcher
private const string TagQuery = "{0}&tag={1}&page={2}";
private string _developerId;

public YouTubeSearcher(string developerId)
if (string.IsNullOrEmpty(developerId)) throw new ArgumentNullException("developerId");
_developerId = developerId;

public IEnumerable QueryByTags(params string[] tags)
if ((tags == null) || (tags.Length <= 0))="" throw="" new="" argumentnullexception("tags",="" "tags="" must="" contain="" one="" or="" more="">

for (int page=1; true; page++)
List results = QueryByTagAndPage(JoinTags(tags), page);
if (results.Count <= 0)="">
foreach (YouTubeResult result in results) yield return result;

private string JoinTags(string[] tags)
if (tags.Length == 1) return tags[0];

StringBuilder builder = new StringBuilder(tags[0]);

for (int i=1; i

return builder.ToString();

private List QueryByTagAndPage(string tag, int page)
Console.WriteLine("Querying by tag: {0}, page: {1}", tag, page);

Stopwatch watch = Stopwatch.StartNew();
List results = new List();

string uri = string.Format(TagQuery, _developerId, tag, page);
XPathDocument xpd = new XPathDocument(uri);

XPathNavigator xpn = xpd.CreateNavigator();

XPathNodeIterator xniError = xpn.Select(@"/ut_response");


if (xniError.Current.GetAttribute("status", String.Empty) == "fail")
string expression = "/ut_response/error/description";
string errorText = xpn.SelectSingleNode(expression).InnerXml;

throw new YouTubeException("Error occured while querying youtube: {0}", errorText);

XPathNodeIterator xni =

while (xni.MoveNext())
XPathNavigator navigator = xni.Current;

string title = navigator.SelectSingleNode("title").InnerXml;
string url = navigator.SelectSingleNode("url").InnerXml;
string thumbUrl = navigator.SelectSingleNode("thumbnail_url").InnerXml;
string id = navigator.SelectSingleNode("id").InnerXml;
string description = navigator.SelectSingleNode("description").InnerXml;
int lengthInSeconds = int.Parse(navigator.SelectSingleNode("length_seconds").InnerXml);
string author = navigator.SelectSingleNode("author").InnerXml;

results.Add(new YouTubeResult(id, url, title, thumbUrl, lengthInSeconds, description, author));
catch (XPathException xpe)
throw new YouTubeException("Xpath exception occured: {0}", xpe.Message);

return results;
Console.WriteLine("Query complete in {0}ms", watch.ElapsedMilliseconds);

Following on from that we need to parse each search result and attempt to pull out it's episode information:
  • Series Number

  • Episode Number
  • Part Number

At this point we might also make the assumption that parts should be grouped by the user who posted them - in case the same episode has been posted twice by two users (quite likely, people are silly).

Parsing part information could be done using successive LINQ queries, but It's actually not that pleasant considering we're generally interogating only two text fields - the title for the clip, and it's description - horses for courses - so instead I built a quick 'n dirty "EpisodeParser" class... here's the code for that:

public class EpisodeParser
private List _contributors = new List();

public EpisodeParser()
: this(
new SeriesContributor(),
new EpisodeContributor(),
new PartContributor(),
new XFormatContributor(),
new PilotContributor(),
new PartOfPartsContributor(),
new WordNumberPartsContributor())

public EpisodeParser(params AbstractContributor[] contributors)
if (contributors != null) _contributors.AddRange(contributors);

public EpisodePart Parse(YouTubeResult result)
EpisodePart ep = new EpisodePart(result);

foreach (AbstractContributor contributor in _contributors)

return ep;

* QI Series 4 EpisodePart 12 (part 3)
* s2e10 part 1/4
* Qi Series 1 Ep 5 Part 1/3
* QI 2x01
* Take Out 1
* QI Pilot EpisodePart part 6
* S2E09

private class SeriesContributor : AbstractContributor
public override void Contribute(EpisodePart episodePart)
int? seriesNumber = ParseNameNumber(episodePart, "series", "s");
AssignSeriesNumber(episodePart, seriesNumber);

private class EpisodeContributor : AbstractContributor
public override void Contribute(EpisodePart episodePart)
int? episodeNumber = ParseNameNumber(episodePart, "episode", "ep", "e");
AssignEpisodeNumber(episodePart, episodeNumber);

private class PartContributor : AbstractContributor
public override void Contribute(EpisodePart episodePart)
int? partNumber = ParseNameNumber(episodePart, "part", "p");
AssignPartNumber(episodePart, partNumber);

private class PilotContributor : AbstractContributor
public override void Contribute(EpisodePart episodePart)
if (episodePart.Result.Title.ToUpper().Contains("PILOT"))
AssignSeriesNumber(episodePart, 1);
AssignEpisodeNumber(episodePart, 1);

private class XFormatContributor : AbstractContributor
public override void Contribute(EpisodePart episodePart)
int seriesNumber = 0;
int episodeNumber = 0;

if (SplitNumber(episodePart, ref seriesNumber, ref episodeNumber, 'X'))
AssignSeriesNumber(episodePart, seriesNumber);
AssignEpisodeNumber(episodePart, episodeNumber);

private class PartOfPartsContributor : AbstractContributor
public override void Contribute(EpisodePart episodePart)
int partOf = 0;
int parts = 0;

if (SplitNumber(episodePart, ref partOf, ref parts, '/'))
AssignPartNumber(episodePart, partOf);

private class WordNumberPartsContributor : AbstractContributor
private static readonly Dictionary _phrases;

static WordNumberPartsContributor()
_phrases = new Dictionary();
_phrases.Add("part one", 1);
_phrases.Add("part two", 1);
_phrases.Add("part three", 1);
_phrases.Add("part four", 1);
_phrases.Add("part five", 1);
_phrases.Add("part six", 1);
_phrases.Add("part seven", 1);
_phrases.Add("part eight", 1);
_phrases.Add("part nine", 1);
_phrases.Add("part ten", 1);

public override void Contribute(EpisodePart episodePart)
AssignPartNumber(episodePart, FindPhrase(episodePart, _phrases));

At this point we have the building blocks for starting to write some LINQ queries... here's my first test - running a basic "select all"...

YouTubeSearcher searcher = new YouTubeSearcher(DeveloperId);

IEnumerable results = from result
in searcher.QueryByTags("QI", "Quite", "Interesting")
select result;

Console.WriteLine("total results: {0}", results.Count());

Trying a more explicit style of query, and parsing episodes:

EpisodeParser episodeParser = new EpisodeParser();

YouTubeSearcher searcher = new YouTubeSearcher(DeveloperId);

IEnumerable parts = searcher.QueryByTags("QI", "Quite", "Interesting")
.Select(result => episodeParser.Parse(result))
.Where(part => part.SeriesNumber == 1 && part.PartNumber == 1)
.OrderBy(part => part.EpisodeNumber);

foreach (EpisodePart part in parts.Distinct())

Which will let us know which episodes in series 1 exist...

Bit of a rush, but next time I'll start digging in a little deeper...

At this point though it's worth noting that we have some stuff for free because of IEnumerableT>...
  • Search results are being processed as their yielded, if we're just looking for the first matching item for a query we can stop without having to request additional result pages on a match is made.

  • Same goes for episodes, we only parse them as they are required - no unnecessary overhead.

So far nothing has required LINQ, but I think we'll start to see it being a great time saver come the next couple of parts... compared to writing the code ourselves.

We shall see!
Read More