Skip to main content

Parallel LINQ (PLINQ) - Intro

.Net 4.0 supports parallel LINQ or PLINQ, PLINQ is a parallel implementation of LINQ.
PLINQ has the same characteristics has LINQ, in that it executes queries in a differed manner.
However, the main difference is that with PLINQ, your data source gets partitioned and each chunk is processed by different worker threads (taking into account the number of processor cores that you have) , making your query execute much faster in certain occasions.

Running a query in parallel is just a matter of calling the AsParallel() method of the data source, this will return a ParallelQuery<T> and your query will execute parallel.

Let's look a code sample...

var query = from num in source.AsParallel()
where num % 3 == 0
select ProcessNumber(num);

Now, when this query is iterated our a foreach loop or when you call ToList() etc..the query will be run in different worker threads.

Although, you have parallelized your query execution, if you want to do something with that result within a loop, then that processing will happen serially although you query executed in a parallel way.

You can achieve this parallelism by running the loop using a Parallel.ForEach() or you can use the ForAll method like this....

var query = from num in source.AsParallel()
where num % 3 == 0
select ProcessNumber(num);

( x => { /*Do Something*/ } );

In the code above the query will run in parallel as well as the result will be processed parallely.

Running LINQ queries in parallel is does not always gives you best performance, this is basically due to the fact that the initialization and partitioning outwits the cost of actually running the query in parallel.
Hence, it's necessary for you to compare which option is best LINQ or PLINQ.
MSDN documents that PLINQ will first see if the query can be run in parallel, then sees the cost of running this query in parallel vs sequentially, if the cost of running this in parallel is more then running it sequentially, then the runtime will run this query in a sequential manner.
I tried it out, but could not actually see the difference :)

Another good option that you might want to run your query if you are thinking of running it in a ForAll method is running the parallel query with a ParallelMergeOptions.
By default, although the query executes parallely, the runtime would have to merge the results from different worker threads into one single result if your running the query over a foreach loop or doing a ToList(), this sometimes causes a partial buffering.

However if you are iterating the query over a ForAll() you can take the benefit of not buffering the record and processing it once the result return from the thread without is a code sample on how to do this...

var query = from num in source.AsParallel().WithMergeOptions(ParallelMergeOptions.NotBuffered)
where num % 3 == 0
select ProcessNumber(num);

( x => { /*Do Something*/ } );

Although using a ForAll() consumes the items when it returns from the thread, I saw some noticeable difference when running the query with a ParallelMergeOption.


Popular posts from this blog

Hosting WCF services on IIS or Windows Services?

There came one of those questions from the client whether to use II7 hosting or windows service hosting for WCF services. I tried recollecting a few points and thought of writing it down.
WCF applications can be hosted in 2 main ways- In a Windows service- On IIS 7 and aboveWhen WCF was first released, IIS 6 did not support hosting WCF applications that support Non-HTTP communication like Net.TCP or Net.MSMQ and developers had to rely on hosting these services on Windows Services.With the release of IIS 7, it was possible to deploy these Non-Http based applications also on IIS 7. Following are the benefits of using IIS 7 to host WCF applications
Less development effort
Hosting on Windows service, mandates the creating of a Windows service installer project on windows service and writing code to instantiate the service, whereas the service could just be hosted on IIS by creating an application on IIS, no further development is needed, just the service implementation is n…

Task based Asynchronous pattern, Async & Await and .NET 4.5

One of the key features in .Net 4.5 is to write asynchronous programs much easier. So if I was to write asynchronous programs in .Net 2.0/3.5, I would either follow the event based model or the callback based model. For an example, a synchronous method that does intensive work (say the DoWork()) can be made asynchronous by using the following patterns
1) Implementing the IAsyncResult pattern. in this implementation, 2 methods are exposed for the DoWork() synchronous method, the BeginDoWork() and the EndDoWork() method. The user will call the BeingDoWork() passing in the required parameters and a callback of the delegate type AsyncCallback(IAsyncResult). The BeginDoWork() will spawn a new thread a return control back to the user. Once work is completed in the spawned method, as a last step, it will call the inform the AsyncResult implementation, which in turns will call the EndDoWork() (which is the callback that was passed in to the BeginDoWork()).
2) Implementing the event pattern. Her…

MEF (Managed Extensibility Framework), .NET 4, Dependency Injection and Plug-in Development

Almost after .Net 4 was released I remember reading about MEF (Managed Extensibility Framework), this was a framework for developers to compose their application with required dependencies. At first this looks like the Unity Container used for dependency injection, but MEF is much more than a dependency container but there is nothing stopping you from using MEF as a dependency injector.I remember around 5 years back when I was in a project that created a framework that allows developers to plug-in there modules as WinForm screens. The developer would create a set of screens with the intended functionalities and the drop this component in the bin folder of the framework and then will move on to do some painful configurations for the framework to pick up the module. Dropping the component into the bin folder and doing a bit of configuration is all that s needed for the framework to pick up display the screens. Typically, the configurations would also contain metadata about the screen.Al…