Parallel Extensions in .NET Part 2

In the last post I covered the Co-ordination Data Structures in .NET 4.00, so now I will examine the Task Parallel Library and Parallel LINQ (PLINQ).

As I mentioned in the last post, Mike Taulty explained that the thought behind the Parallel Extensions is to abstract the multi-processor jobs away from the concept of threads.  The Task Parallel Library contains objects that allow you to achieve this.

At the heart of this is the Task object which represents a unit of work to be carried out.  There is a factory version (as shown below) and a standard object version of this class.

Task myTask = Task.Factory.StartNew(() =>

{

    //code to be carried out

});

myTask.Wait(); //wait for task to complete

Which ever method you use to construct the object it is passed a predicate which contains the work to be carried out.  Tasks are atomic operations that can be carried out on any of the processors in the PC.  The Wait() command is only applicable if you need to wait for the result of the Task

An interesting feature of Tasks is that a Task can start off other Tasks within them.

Task parentTask = Task.Factory.StartNew(() =>

{

    Task childTask = Task.Factory.StartNew(() =>

    {

       //code to be carried out

    });

    //other code to be carried out

});

These child classes are automatically linked to the Parent class, so that if you call Wait() on the parent class then this will wait for any child Tasks to finish.  This default behavior can be turned off (if required) in an extra parameter passed into the constructor.

Another intriguing feature about these Tasks is that when an exception occurs within the main Task or one of it's children, the exception thrown is a AggregateException.  This exception contains a new property called InnerExceptions that contains any exceptions that are thrown by the children.  So, if three child task are spawned and all of these throw an exception you can view all of these in this AggregateException.  Nice.

Mike also introduced the Parallel static class which provides an alternative way of spawning multiple Actions.  It has handy methods such as Parallel.ForEach that allows a number of actions to be thrown a number of times.  I won't go into too much detail about this one, but if you look on the recently published MSDN for .NET 4.00 there are plenty of examples of it's use.

Finally, Mike introduced use to Parallel LINQ, otherwise known as PLINQ!  This is a command that can be used on LINQ queries to make them execute across multiple processors.  It's purpose is to make LINQ queries on any large data sources more efficient.

PLINQ is very easy to use, all you need to do is utilise the AsParallel() at the appropriate place in the LINQ query.  For example:

var results = myData.AsParallel().Where(x => x.Name == "World").Count();

It's that easy!

That concludes my whirlwind tour of the Parallel Extensions in .NET 4.00.  There are plenty of other resources on the MSDN and various microsoft blogs on Parallel Extensions.  Here are a few:

Introduction to PLINQ
Microsoft Parallel Computing Center
Microsoft Parallel Programming Team Blog
Mike Taulty's Blog

Parallel Extensions in .NET 4.00 Part 1

Last night I attended an NxtGen User Group in Manchester at which Microsoft guru Mike Taulty was giving an introduction to the new parallel programming components of the .NET framework.

Whether we like it or not we, as developers, are going to have to adapt to using techniques that will run code on separate processors. The new chips from Intel and AMD are no longer continuing to increasing in Ghz, they are just containing more processors. So, we need to be able to utilise this power.

The good news is that the .NET CLR and certain components, such as WCF and WF, already spread their workload across multiple processors. But the bad news is that our own programs do not.

So, how would you currently handle this? The answer is by using a Threads or a ThreadPool. But as Mike demonstrated this is quite long winded and makes the code quite difficult to read.

The idea behind the Parallel Extensions in .NET 4.00 is to abstract the concept of the work that can be carried out on multiple processors away from the underlying concept of threads. They have split the components up into the following three categories:

1. Co-ordination Data Structures.
2. Task Parallel Library
3. Parallel LINQ (PLINQ)

Lets start with the Co-ordination Data Structures. This category contains all the replacement data structures that are required to support parallel programming. If you have ever performed any thread safe work that involves a collection, you will know that none of the collections in .NET are thread safe. The only solution is to lock the entire collection when carrying out any work. Which frankly is a bit rubbish.

The co-ordination data structures contain a selection of Concurrent Collections to get around this issue. These include ConcurrentLinkedList and ConcurrentQueue. It also includes some advanced Co-ordination helpers that include a Barrier (blocks work until a x number of threads reach) and a CountdownEvent (a concurrent countdown that can be signalled from multiple threads).

The last of the co-ordination data structures is the Data Synchronization classes. The main one of interest is the Lazy class. Great name, but what does it do? It allows you to create a thread safe Singleton class. For example, say we have a singleton class called MyClass which has a method called MyMethod(). Initialisation of the singleton can be complex as we may have several threads entering the initialisation code. The solution is to use the Lazy class to wrap the class like so:

Lazy myInstance = new Lazy<MyClass>();

myInstance.Value.MyMethod();

This ensures that ONLY one instance of this class will exist.

In the next part I will cover the Task Parallel Library.

Creating comma separated strings with LINQ

Here is a really quick LINQ based solution to the conversion of a collection of strings into a comma separated string:

//Prepare list

List myList = new List();

myList.Add("aString");

myList.Add("anotherstring");

//Get comma separated string

var stringBuilder = myList.Aggregate(new StringBuilder(), (builder, stringValue) =>

builder.AppendFormat((builder.Length == 0) ? "{0}" : ", {0}", stringValue));

string commaString = stringBuilder.ToString();

WCF Tracing with Activities

One of the nice features of Windows Communication Foundation (WCF) is that it can generate an excess of trace information that can help us identify issues with our services. All you need to do is turn the tracing on through the app.config like so....

name="System.ServiceModel"

switchValue="Information">

name="traceListener"

type="System.Diagnostics.XmlWriterTraceListener"

initializeData= "c:\log\Traces.svclog" />

Microsoft even provide a handy tool called Service Trace Viewer Tool (SvcTraceViewer.exe) that can be used to view the trace information that WCF produces.

However, if you have ever looked at one of these traces you will have noticed that it can be difficult to sieve through and find the service call(s) that you are interested in. It becomes especially difficult if the service calls other services. Wading through all this tracing information is time consuming and it can be very frustrating.

Fortunately WCF provides a mechanism called Activities that allow us to correlate related trace information.

Where these Activities start and stop are defined by the developer in code. They can also be propagated across service boundaries and can also be linked together with other related activities.

So, what do you need to do in order to use Activities in your tracing? They are quite simple to use, all you need to do is assign an activity id to the correlation manager. This is a GUID value and is assigned as follows:

Trace.CorrelationManager.ActivityId = Guid.NewGuid();

Now, all the trace messages that are fired after this will be included in this Activity. However, in order for this to appear in the trace log we need to add ActivityTracing to the switchValue of the diagnostics section of the app.config.

name="System.ServiceModel"

switchValue="Information ActivityTracing"

propagateActivity="true">

name="traceListener"

type="System.Diagnostics.XmlWriterTraceListener"

initializeData= "c:\log\Traces.svclog" />

Also, if we want the ActivityId to be passed from one service to another, we can set the propagateActivity attribute to True. This is useful if we are starting the activity in one service operations that is then calling other services operations to achieve it's goals. With propagateActivity set to true, all of the trace information from these services for this particular call would be grouped together.

UML Aggregation vs Composition

A colleague asked me a few weeks ago for some advice about their UML class diagram. One of the issues they were having was deciding on whether the link between two of the classes should have a "filled-in" or "non-filled-in" diamond.

You have probably already guessed that he was really asking about whether the two classes were linked by Aggregation ("non-filled-in") or Composition ("filled-in"). From my experience, this is a very confusing area of UML for some developers. I have reviewed may class diagrams where the developer seems to have just liked the look of one of the diamonds and just used it on all of their associations!

So, where should these notations be used? If you search on the internet you'll find a plethora of explanations, but I thought it might be useful to try and add a simplified explanation.

First off, this is the difficult part for developers, Aggregation and Composition are nothing to do with code. Both Aggregation and Composition are effectively talking about is one class contains another class. So, many explanations suggest it's Aggregation if it's public property and composition if it's a private field. But unfortunately it doesn't work like that.

The decision on whether a relationship is a Aggregate or Composite is based on how the classes are related in the real world. Take a classic UML example of a Car class. Your car class may have a engine class, a collection of wheel classes and a fluffy dice class. So, which of these are aggregates and which are composites?

Lets start with the engine class. This is quite a vital part of the car, you couldn't imagine a car without an engine. Likewise, the wheels are quite important too. In fact you could say that the car is composed of (among other things) an engine and wheels. So, these relationships are examples of Composition.

The remaining fluffy dice class is a bit of a looser relationship. A cars function does not depend on the tacky fluffy dice attached to the mirror. The car may not even have fluffy dice. So, this demonstrates quite nicely a Aggregation relationship.

So, our resulting class diagram looks something like this...

Oracle vs .NET int

One of the problems of using .NET with Oracle is the type conversion of integers through the ODAC component.  Basically, the problem comes about because the ODAC component has no idea which .NET type to use when returning a value in a NUMBER field.... so it panics and just tends to return everything as a double (remember that the NUMBER field can hold decimal values!).

Oracle have also provided the INTEGER data type, which is really just a short-cut to get a NUMBER(32) field.  The best solution I can find to this is to define any columns that are being mapped to a .NET int as NUMBER(9, 0).   This will always ensure that the field will hold data that is compatible with being held in a .NET int.  

Devweek Day Two

Session 1 - Building applications in the ‘cloud’ with Azure services.

Started the day back with Christian Weyer for another of his amusing style of talks. This time he was introducing how Azure can be used to build applications in the ‘cloud’. The overall impression that I am getting of Azure is that it is a work in-progress. Christian demonstrated this by showing a technology uptake diagram, which well and truly placed ‘cloud’ computing in the emerging technologies section.

Azure is organised into two types of service, web and worker services. The idea is that the services can be up scaled so that the capacity of the application can be increased. For example, if you have a particularly popular web service that starts getting hit by thousands or millions of users you can log into Azure and increase the number of service instances to cover this demand.

Underlying the two core service types is the service bus. This is basically an enterprise service bus in the ‘cloud’. It allows the management and routing of service interactions with services held within the cloud and locally.

Christian also briefly showed the Azure workflow services. These will allow processes or interactions between services to be mapped out visually. Azure workflow's are not based on standard Windows Workflow (WF). They only have a relatively small sub-set of activities that are very ‘cloud’ specific (e.g. connect to cloud service). Although WF seems to be a major part of Microsoft's ‘cloud’ master-plan, they are still far from stable.

Session 2 - The Patterns Dartboard.

For some light entertainment prior to lunch I decided to attend this talk on design patterns. The premise of this talk was that a dart was to be thrown at a ‘dartboard’ of design patterns. Whichever pattern the dart hit one of the presenters, Kevin Jones and Andy Clymery, would present to the group in no more than 15 minutes.

During the next hour and a half they covered the Decorator, Factory, Active Record, State and Mode View Controller (MVC) design patterns. Of these I had particular interest in the Active Record presentation. Actually a first I had no idea what this design pattern was! Then as the presentation moved onwards I realised this is another name for what used to be my favorite design pattern, the Persistence pattern.

I have to say, I very rarely use Active Record (AKA Persistence) anymore, Though it is quite well suited to a small reliable system. I don’t know a great deal about Ruby on Rails but apparently this is the design pattern that it utilises.

For those not familiar with Active Record is that the class should contain the business logic and the code necessary to persist the object in the data store (usually some kind of database). Their interpretation of Active Record suggested that all ‘read’ actions should be static methods and all ‘write’ actions should be instance methods. The example class below demonstrates this:

public class MyClass
{
private int Id;

public static MyClass GetMyClass(int classId)
{
//database select code and return new object with data...
 }

public void Save()
{
//if object does not have ID then INSERT to database else UPDATE...
 }
}

I must admit that in my my years of employing this design pattern it never occurred to me to implement the loading functions as static members.

Session 3 - WCF Security Architecture and Best Practice.

This was one of the ‘must-see’ sessions of the week for me, particularly due to recent issues that I have encountered with security within my WCF applications. Like many features within WCF this can be a confusing area as there is SO much that is configurable.

Dominick Baier ran this session which went into incredible detail about the security options that are available within WCF. I can’t say that I managed to find an exact solution to my particular problem, but I found it a great grounding in the elements of WCF security (it’s all about the Binding!). Plus, I now have his email address - beware Dominick!

As I’m still taking in this wealth of knowledge it is difficult for me to expand any further at the moment.

Session 4 - Visual Studio 2008 IDE Tricks and Tips.

Can I first say that I attended this session purely as my brain couldn't take any more heavy talks on the second day. The WCF security session well and truly finished me off!

Though I have to say that I will be using many of the short-cuts that were introduced in this session. Here is a website that summarises the standard shortcuts in Visual Studio.

WCF Exceptions and Faults

In Windows Communication Foundation (WCF) exceptions thrown from services have to be converted into Faults - which are basically cross domain friendly exceptions. Custom faults can be created and are just declared using the [DataContract] attribute. The example below shows the declaration of a simple Fault.

[DataContract]
    public class MyFault
    {
        [DataMember]
        public string Message { get; set; }      

        public PasswordFault()
        {
        }

        public PasswordFault(string message)
        {
            this.Message = message;
        }
    }

To generate a WCF fault you have to use the FaultException class. The example below shows how we could raise the fault declared in the previous example.

public class myService: IMyContract
{
 public void MyMethod()
 {
    try
    {
       ......
    {
    catch (myException exc)
    {
      MyFault = new MyFault(exc.Message);
      throw new FaultException(MyFault, "Failed to run MyMethod");
    }
 }
}

One other important part of setting up faults in WCF is that the OperationalContract of the method raising the fault needs to define the the FaultContracts that the method can raise. For example we might define the service contract of our service to be as follows:

[ServiceContract]
public interface IMyContract
{
 [OperationContract]
 [FaultContract(typeof(AuthenticationFault))]
  void MyMethod();
}

The FaultContract attribute is telling WCF which faults can be expected. If a fault is thrown that is not defined here, the client will just get an ambiguous FaultException raised and be unable to access the extra data defined in the Fault.

Alternative Fault Handling Techniques

Recently, during the development of quite a large SOA based system I noticed that Faults that are raised by sub-services were not being raised correctly through the next service. The only way to ensure that these faults were received by the client was to recapture the fault (as a FaultException() ) and re throw them!

After a bit of investigation and a very helpful reply to a post on the MSDN WCF forum I was pointed in the direction of this example of using the IErrorHandler interface to add custom error handlers to the individual channel dispatcher. I'm not going to go into any great amount of detail about this, but I strongly recommend reading the article. It gives you a much more elegant way of handling exception to fault conversions in WCF.

Safari 4

A beta of Safari 4 for Mac and PC was released last week, go here for download.
It features some rather nice eye candy - such as a "cover flow" browsing history and a rather nice favorite site window (see screen shots above).

I'm sure many web developers will be shuddering at the thought of yet another browser to support, but it does bring with it some new web technologies (such as CSS 3 and HTML 5). It also provides some rather nice looking developer tools, see here for full feature list.

I tried using Safari 4 for a couple of days on my work PC. It seemed to be noticeably faster than other browsers, though it did seem to hang a bit when adding new tabs - this might be because it was set to always start new tabs on the favorite site window. I think if the hanging issues could be resolved it could give Firefox some competition.

Oracle XmlType and ODAC .NET provider

I have been having some issues today with a method that inserted Xml into an XmlType field in an Oracle database. It had been working fine, but recently the Xml had grew in size considerably. I was now getting the following Oracle error returned from the ODAC component:

ORA-01461: can bind a LONG value only for insert into a LONG column

The Insert Query I was running looked a bit like this....

INSERT INTO myTable

(anId, myXmlField)

VALUES

(:anId, :myXmlField);

When I ran the query I had two parameters, the second (:myXmlField) was an Xml string. It was this that caused the issue. Thanks to this post by Stuart Campbell I found out that in order to pass a long string value to an XmlType field you must:

1) Set the parameter type of the xml parameter (:myXmlField) to Clob;

2) Use the XmlType() call in the query:

INSERT INTO myTable

(anId, myXmlField)

VALUES

(:anId, XmlType(:myXmlField));

This did resolve the issue, but in the end I ended up changing the XmlType field into a Clob! This is because 1)I didn't need to xPath the Xml in the database and 2) I couldn't use XmlType implementations in our SqlServer data access layer as we have to support earlier versions of this database.

For those in the North West.....

There is a free MSDN event on at the Lowry in Manchester on Thurday afternoon (26-Feb-2009).
The event is covering WPF and Silverlight 2.

See here for further details and registration.

Custom authentication

Occasionally you may have to develop a system that, rather then relying on an existing security system (e.g. active directory, novell), requires you to write a custom authentication system.

Usually, in such systems the user data and passwords are stored in a table in the database. These records can then be queried to determine whether the user requesting access has the correct password.

Okay, this is simple enough... but storing a users password in a table is not particularly great. A hacker or other unauthorised user might get access to the table and thus be able to log onto our system as anyone they choose!

It is better practice to encrypt the password before storing it in the database. The software will have to retrieve and decrypt this value before the users password entry can be validated.

Even if encryption is used, the account could still be open to attack as Users might choose an obvious password. Fortunately, extra precautions can be added to the security system to help further protect against unauthorised access. These are:

1) Adding rules for password format - Regular expressions can be used to ensure when a user sets their password it adheres to certain standards (e.g. contains 1+ numbers, is a certain length).

2) Password attempts - Implement code to lock the account if the password is entered incorrectly over a certain number of times. To save on hassle for the system administrators you might want to consider just locking the account for a set period (e.g. 2 hours).

3) Encrypt with Salt values - A salt value is a value that is added to password prior to encryption to make the encryption more difficult to crack for hackers using dictionary attacks. I would strongly recommend using salt values if you are using encryption.

In .NET salt values can be generated by using a class such as RNGCryptoServiceProvider. The code below shows a static method I use to generate salt values.

/// 

/// Generate a random salt value
/// 
/// 
public static string GenerateSalt()
{
  RNGCryptoServiceProvider rng = new RNGCryptoServiceProvider();
  byte[] randomNumber = new byte[16];

  rng.GetBytes(randomNumber);
  return Convert.ToBase64String(randomNumber);
}

The encryption algorithms in .NET (such as the preferred RijndaelManaged class) allow you to set the salt value when encrypting and decrypting values.

It is important to remember that when you use a salt value that this will also need to be stored somewhere, so that it can be retrieved by the code when it needs to decrypt. Salt values should NEVER be stored in code as a hacker could use a disassembler on the assembly and discover the value.

I always dynamically create a new salt value every time a users password is changed. This value
is then stored in a separate table in the database, linked to the record by the user table primary key. This can then be easily retrieved when the code needs to decrypt the data.


I hope I've managed to cover the main points of creating a custom authentication system. It'll will be interesting to see what the next generation of security systems in WCF 4.0 and Azure will bring. I'm hoping that there will be considerable improvements so that developers will not have to keep reinventing the wheel with their own authentication methods. I am attending DevWeek next month in London, so I'll blog/twitter my discoveries....

Stored procedure maintenance

One of the nice features of Oracle stored procedures, functions and packages is that you can create or recreate the database side code in the same statement. The advantage of this is that during upgrades/patches you can run ALL database code scripts again, which ensures nothing is missed - and also that the changes you have made haven't caused breaking changes in another procedure!

For example, the following will create or replace the following security package header..

CREATE OR REPLACE PACKAGE PKG_SECURITY
AUTHID DEFINER
AS
/*  ------------------------------------------------------
    Adds authentication for specified user to system    
    ------------------------------------------------------ */
PROCEDURE proc_AddUserAuthentication(
  userId                IN USERSECURITY.USER_FK%TYPE,
  encryptedPassword     IN USERSECURITY.PASSWORD%TYPE,
  saltValue             IN USERSALT.SALT%TYPE,
  auditUserId           IN USERSECURITY.AUDITUSERID%TYPE);
END;
/

My recent digging around has revealed that you can also mimic this CREATE or REPLACE functionality in MySQL and Sql Server.

In MySQL you can call the DROP PROCEDURE IF EXISTS method....

DROP PROCEDURE IF EXISTS premierenvoy.proc_AddUserAuthentication;
CREATE
DEFINER = CURRENT_USER
PROCEDURE proc_AddUserAuthentication(
............

...and in Sql Server you can use the OBJECT_ID function and check whether the returning value is not null to determine whether to drop the procedure.

IF OBJECT_ID('proc_AddUserAuthentication') IS NOT NULL
DROP PROCEDURE proc_AddUserAuthentication;
GO
CREATE PROCEDURE proc_AddUserAuthentication(
............

Using Data Grids in WPF

One of the biggest omissions from the .NET 3.5. WPF control library is a Data Grid. There are many 3rd party implementations for sale of free (just do a Google search on WPF and DataGrid). However, it is also possible to use the standard Windows forms DataGrid within your WPF form.

This excellent article by Rahul Saxen explains how this can be achieved.