Posts Tagged ‘queryable collections’

Magiq: New release and API Changes

April 28, 2011 Leave a comment

New release of Magiq, supporting Linq-to-sql. This should be the last one for Linq-to-sql, the next one I hope will have the minimal support for Entity Framework. You can download it from here.

For those reading this for the first time (?), Magiq is a framework that let you do mass operation in a Linq fashion.

So… as Magiq will be supporting Entity Framework soon, it made sense to change the API to make it support Entity Framework limitations. Also, I found a better Insert API (the one that always made me doubt!). The changes are:

Query extension method changes
The new method looks like this:

country.Query(x => x.Cities); 

It will return an IQueryable that will let you do all the Magiq stuff.

No more IEnumerable support
This is a hard one. You can only do Magiq stuff between IQueryables. If you have an IEnumerable collection, use the Query extension method of it parent entity.

New Insert API

destination.Insert( source.Where( condition ).Select( new Item{ ... } ));

Cool, isn’t it? 🙂

Entities extension methods
Because we no longer support extension methods over collections but over the entity, there are some extension methods for improving the code readavility.


entity.Delete(x => x.Collection.Where( condition ));
//Instead of 
entity.Query(x => x.Collection).Where( condition ).Delete();


entity.Update(x => x.Collection.Where( condition ).Set(x => x.Prop, value));
//Instead of 
entity.Query(x => x.Collection).Where( condition ).Set(x => x.Prop, value).Update();


entity.Query(x => x.Collection.Where( condition ));
//Instead of 
entity.Query(x => x.Collection).Where( condition );


entity.InsertInto( x => x.Collection, source.Where( condition ).Select( new Item{ ... } ));
//Instead of
entity.Query(x => x.Collection)
      .Insert( source.Where( condition ).Select( new Item{ ... } ));

Magiq: Next steps – different approach

April 25, 2011 Leave a comment

Back to the Magiq game!

I started to play with Entity Framework 4 and realized some stuff is not possible the way it was designed. It has to do with the way EF4 manages Lazy collections.
In Linq-to-sql and NHibernate, the collection itself has the property of being lazy, materializing all the objects when you iterate it. For example, in Linq-to-sql collections are EntitySets. Because of this, you can do this:

parent.Children.Delete(x => x.SomeCondition);

This is because the Children collection contains all the information needed:

  • The parent object instance, for getting the Id
  • The collection name

Sadly, EntityFramework handles this feature in a different way: Is the parent object the lazy one and the collection is populated when you access the collection propery and not when you iterate it. Because of that, in the moment you do parent.Children you have executed the query in the database.
This lead us to have a different API, that avoid accessing the collection property. What I thought is:

parent.Delete(p => p.Children.Where(x => x.SomeCondition));

It’s not so bad, I must Say, and it also let us do things like

parent.Delete(p => p.SomeObjectInTheMiddle.Children.Where(x => x.SomeCondition));

without executing the query for getting the SomeObjectInTheMiddle instance.

Also, I give up with Magiq-to-NHibernate, at least for a while. NHibernate has huge features, it let you map your classes in thousand of different ways, and that is awesome. But also it makes it really complicated to work with. And since NHibernate already has mass operation support, it makes sense to stop worring. At least, as I said before, for a while.

So, I will be working on changing the Query API and only for Linq-to-Sql and Entity Framework. Everyone is welcome to contribute implementations for other ORMs.

Next step: Changing the API

Magiq-to-Sql: Compiled queries support

November 25, 2010 Leave a comment

Back on the road with Magiq, at least for a while. I’m working on the development of a framework that could make full use of the Named Scopes feature of Magiq. The problem is that linq-to-sql needs compiled queries in order to speed up the performance and the current API is awful. So I tried to give Magiq the ability of handling compiled queries.

Since Linq is all about expression trees, I knew it will be possible to get from

var value = 3000000;
var query = country.Cities.Query( x => x.Population > value)
                          .OrderBy( x => x.Population);

to something like

var value = 3000000;
var expresson = (dc, id, value) => dc.GetTable<City>()
                                     .Where( x=>x.Id == id )
                                     .SelectMany( x=> x.Cities)
                                     .Where( x => x.Population > 100000 )
                                     .OrderBy( x => x.Population);

var query = CompiledQuery.Compile( expression );

Well, two hours of playing with the Expressions and all worked. The only/main problem is that Expressions instances are created every time so each expression object has a different HashCode and it’s impossible to cache the compiled queries by expression. Sadly, I had to add a parameter named “key” for identifying the compiled query.

This si how the final API looks like:

var value = 3000000;
var query = country.Cities.CompiledQuery( "large-cities-by-country", 
                                          (q,p) => q.Where( x => x.Population > p)
                                                    .OrderBy( x => x.Population),
                                          value );
  • The query will be identified as “large-cities-by-country”.
  • We need to pass the expression to be applied to the collection and the parameters, because we don’t want to hardcode the values in the query.

A small benchmark I’ve done with a simple model executing 5000 queries on a new DataContext each gives, this result:

  • magiq – not compiled: 36457 millisecond
  • magiq – compiled: 22295 millisecond
  • linq2sql – compiled: 22194 millisecond

Magiq-to-NHibernate: Collection filtering and moving to NHibernate 3

February 25, 2010 1 comment

Few days ago I started having troubles with Linq-to-NHibernate from the current release 2.1.2. The main problem cames when I design the collection filtering strategy. Basically, my idea was to go from:



session.Linq<Child>().Single( x=>x.Id == parent.Id ).SelectMany( x=>x.ChildCollection );

Having this, we could add many expressions to that queryable as we want.

The thing is, as Chadly posted, Linq-to-nhibernate 1.0 uses the query parameter (the x in the previous expression) as the criteria alias, so this works:

       .Single( x=>x.Id == parent.Id )
       .SelectMany( x=>x.ChildCollection, (p,x) => x )
       .Where( x => x.Something );

but this doen’t:

       .Single( x=>x.Id == parent.Id )
       .SelectMany( x=>x.ChildCollection, (p,x) => x )
       .Where( z => z.Something );

Since the chosen strategy for dynamically generating the query involved the definition of the “alias” parameter, and the user will need to know which is, I gave up. So I tried NHibernate 3. And it worked 🙂

It was really painful to change what I’ve done for the old linq provider, because it uses criteria and the new one uses hql. It’s not easy at all to get a IDbCommand from a HQL, but not impossible.

By now, deletes, updates, inserts and bulkinserts are working properly for linq-to-nhibernate 2.0 with the same limitation as before: no inheritance, no collections.
In the next days I will be working on collection support, composite ids and mappings.

Magiq: Named scopes in C#

February 18, 2010 Leave a comment

Some weeks ago, my friend Jonas told me about Ruby’s ActiveRecord Named scopes and I thought what a marvelous feature! Later, I realize that it can be implemented in c# using Magiq.

Suppose you have a store that sells products and these products has expiration date. You call “Available” the products that had not yet expired. In order to find the not expired products, we could do, using Linq:

var availableProducts = store.Products.Where( x => x.ExpirationDate > DateTime.Now );

The problem with this approach is that the more time passes, more products the store will have and more expired, so it could retrieve the entire products collection just to filter some. Magiq to the rescue!

var availableProducts = store.Products.Query( x => x.ExpirationDate > DateTime.Now );

As I told in a previous post, it will generate a new sql query with the new condition.

Ok. In a perfect world, you will have something like an AvailableProducts property in the Store class, because the availability definition could change in the future.

public IEnumerable<Product> AvailableProducts {
    get {
         return Products.Query( x => x.ExpirationDate > DateTime.Now );

Now, what happens if besides store, we have a warehouse full of products and we need the available products from there? We will have add the same method in the Warehouse class.

Wouldn’t be nice if we could do it this way:

var availableProductsFromStore = store.Products.Available();

var availableProductsFromWarehouse = warehous.Products.Available();

Ok, we can 🙂

public static class ProductNamedScopes
	public static IEnumerable<Product> Available( this IEnumerable<Product> self )
		return self.Query( x => x.ExpirationDate > DateTime.Now );

I know, this could be implemented using pure linq, a foreach,, NHibernate’s collection filtering, etc. But Magiq let you hide all that stuff and use pure Linq in a generic way.

Magiq 0.3.0 available!

February 17, 2010 Leave a comment

I’m glad to announce that the new version of Magiq is available.
Magiq now support:

  • Plugin model
  • BulkCopy as a Plugin for using SqlBulkCopy when inserting into sql server from any source.

And also there was a huge refactoring in order to have simplicity when implementing new providers.

You can download this version from here.

Magiq-0.2.0 is avaliable!

January 26, 2010 Leave a comment

I’m glad to annunce that the new version of Magiq is out. You can download it here.

Magiq 0.2.0 supports:

  • Mass operations to Linq-to-sql collections (EntitySet)
  • Collection filterig support for Linq-to-sql
  • Support to sorting and grouping
  • Supports when you want to insert into a repository from another one (for example, insert a lot of entities from a string collection). In this case, it uses the same approach as using a foreach, but it works 😉