Archive for January, 2010

Magiq: Plugin support & the insert API changes

January 30, 2010 Leave a comment

The the main feature of Magiq 0.3.0 is sql server bulk insert support, we needed to improve Magiq with plugins.
These plugins would act as interceptors, and they could be applied at configuration level (for example, logging) and query level (bulk insert).

What I wanted is to have a natural way to assign plugins that would work the same for the four operations: insert, update, delete and query.
Ok, what I wanted was actually complicated.
Since I thougth this for Bulk insert, I tried an API for Inserts:

source.Insert( x => new Item{ Name = x.Name })
      .Using( new BulkInsert() )
      .Into( destination );

Same could work for Updates, but no for Deletes and Queries, because those doesn’t have “intermediate” class. Let me explain: insert and update are performed using several calls. Insert have one for declaring the new expression ( x => new Item{ Name = x.Name }) and one for declaring the destination. The same happens with update, we have several for declaring the “sets” and one Update() to finish de operation. Delete and Query, however, works with one single call (They doesn’t need anything else in order to work), so I don’t have a place to put the “Using” method :(.

So, how could it be implemented for Delete and Query? I thought it should be before calling the operation, something like

source.Using( new BulkInsert() )
      .Insert( x => new Item{ Name = x.Name })
      .Into( destination );

The issue with this approach is that the plugin should be for all the operations and not the specific one. But I’m ok with that, I suppose It could be implemented like NHibernate listeners.
So… the idea is to wrap the IEnumerable or IQueryable with another one that is plugged to one or more plugins. Later, I could check for the type of the collection and make use of the plugins. The problem with this solution is that nothing ensures that collection to be the one used in the magiq operations. For example, something like this:

source.Using( new BulkInsert() )
      .Where( somecondition )
      .Insert( x => new Item{ Name = x.Name }).Into( destination );

Magiq will work with the last queryable and not the plugged one. I know, I could check all the chain but, again, nothing ensures that the plugged collection is there.

Talking to my friend Rezlaj, we got to a solution I don’t like a lot but… it’s a solution. The idea is to have all the stuff inside a interface IMagiq that could be getted using an extension method Magiq applied to a collection. It would look like:

      .Where( somecondition )
      .Insert( x => new Item{ Name = x.Name }).Into( destination );

This IMagiq interface have the methods to add plugins and could have another methods to interact with the operation in the future. Of course, there still are the extensions methods Delete, Set, ToQueryable (well, not this last one, because I changed it to Query because I think is better). But they delegates all to the IMagiq interface.
I think I should change the blog title to “The problem with that…”. You may ask way…

The problem with that (the previous that, actually. I mean, the new API) is that all the operations are applied to a T where T : class except Insert. Why? WHY?! because the source of an insert operation could be a colection of int’s, date’s, whatever. For example, this should be supported (and it is!):

sells.Where( x => x.Date == DateTime.Now)
     .Select( x => x.Total )
     .Insert( x => new SellRecord{ Total = x.Name, Date = DateTime.Now })
     .Into( records );

(I know, the example is not the happiest one, but it’s helpful for explaining my point)
So… the source collection is a collection of decimals. Ok, how could it be solved? because IMagiq needs the T : class constraint. Well, there was an idea in my head for the last couple of days about how linq and all the magiq operations except insert reverses the analogous sql structure. So… It makes more sense to have the insert reversed. Something like:

records.Insert( x => new SellRecord{ Total = x.Name, Date = DateTime.Now })
       .From( sells.Where( x => x.Date == DateTime.Now) )

Ok, it’s just an example, it won’t compile (but It will complain… cuak). I need the source type before the NewExpression, so it have no choice that be:

records.From( sells.Where( x => x.Date == DateTime.Now) )
       .Insert( x => new SellRecord{ Total = x.Name, Date = DateTime.Now })

Now, the collection wrapped by IMagiq is the destination and it could have the constraint.

So, summarizing:

  • Magiq() Extension method was added
  • IMagiq adds support to plugins
  • Insert API changed to destination.From( source ).Insert( new )

Magiq-0.2.0 is avaliable!

January 26, 2010 Leave a comment

I’m glad to annunce that the new version of Magiq is out. You can download it here.

Magiq 0.2.0 supports:

  • Mass operations to Linq-to-sql collections (EntitySet)
  • Collection filterig support for Linq-to-sql
  • Support to sorting and grouping
  • Supports when you want to insert into a repository from another one (for example, insert a lot of entities from a string collection). In this case, it uses the same approach as using a foreach, but it works 😉

Easy lazy loading

January 25, 2010 3 comments

A common use of lazy loading make use this snippet:

public Thing Something {
    get {
            if( something == null )
                    something = new Something();
            return something;

Sometimes, you need to lazy load something inside your class. I mean, a private lazy field. But as c# doesn’t allow us to make a field as “lazy”, so you need to use the previous snippet again, but for a private property, making your code more ugly each time. So I tried to improve the use of this “pattern”.

What I wanted in the first place is to change the previous code into:

private Lazy<Thing> thing;


thing = Lazy.Load<Thing>(); 
thing = Lazy.Load<Thing>( thing => thing.Id = 123); 
thing = Lazy.Load( () => new Thing() );

I know, it could implement an interface and I could make use of dynamic proxies, but the idea behind this is to get to a simple solution.

Anyway, it was pretty easy to have that working. I add a “Instance” property to Lazy class and there I put a little if to check if the instance was loaded. I know, that name reminds to the singleton (anti)pattern, but I couldn’t think any better.

public T Instance {
    get {
            if( !loaded ){
                 instance = load();
                 loaded = true;
            return something;

where load is the delegate used to create the actual instance of the class.

Then I decide that I don’t like that if to be called every time, so I changed it so Instance call a getInstance() delegate that is assigned in the Lazy class constructor:

private T instance;

private T getLoadedInstance() {
	return instance;

private T loadInstance() {
	instance = load();
	getInstance = getLoadedInstance;
	return instance;

public Lazy(Func<T> load) {
	this.load = load;
	getInstance = loadInstance;

private Func<T> getInstance;

public T Instance {
	get {
		return getInstance();

The first time that Instance is called, it calls the load method and the changes the getInstance delegate itself in order to return the created instance.

This is the final code (I wanted to upload it as an attachment but wordpress doesn’t allow zips nor source code files).
Edit: You could download the source code from

Categories: C# Tags: ,

Magiq: Querying a collection

January 11, 2010 1 comment

One of the future features of Magiq will be mass operations to collections. I mean that

var country = CountryRepository.GetById(1);
country.Cities.Where( x => x.Population > 100000 ).Set( x => x.IsBigCity, true ).Update();

should generate a t-sql like

UPDATE [t0] SET [t0].IsBigCity = @p2 FROM Cities [t0] where [t0].Population > @p1 AND [t0].CountryId = @p2

Of course, this should be possible only if the collection is a Lazy one, so all the information needed is inside of it (it’s the same information needed to retrieve the items). For Linq-to-sql is EntitySet. For NHibernate, AbstractPersistentCollection (PersistentGenericBag, PersistentGenericList, PersistentGenericSet and PersistentGenericMap).

The problem (because there’re always problems) is that those classes doesn’t implement IQueryable and the previous magiq statement will work always as magiq-to-objects, because the collection is enumerated once the Where extension method is called and the Set method is called to a simple IEnumerable.

Talking with Sebas and my friend Jonas, we conclude that the best way to solve this is adding an extension method ToQueryable that converts the lazy enumerable to a queryable one (I would like to call it “AsQueryable”, but it already exists :S). I know, it’s a workaroud, but… it seem like I have no choice.

So, the API will look like:

var country = CountryRepository.GetById(1);
country.Cities.ToQueryable().Where( x => x.Population > 100000 ).Set( x => x.IsBigCity, true ).Update();

The great news is that you can use it for sub selecting collections. It would be the same feature that nhibernate has, but for all the orm’s that have lazy collections (of course, nhibernate magiq provider will implement this feature using it own feature).

Something like:

var country = CountryRepository.GetById(1);
country.Cities.ToQueryable().Where( x => x.Population > 100000 );

will execute

SELECT [t0].Id, [t0].Population, [t0].Name FROM Cities [t0] where [t0].Population > @p1 AND [t0].CountryId = @p2

Isn’t it pretty?

First release of Magiq: 0.1.0

January 5, 2010 2 comments

I’m glad to annunce that the first version of Magiq is finished. It fully supports Magiq-to-objects and Magiq-to-sql.
The future plan includes Magiq-to-NHibernate and Magiq-to-Entities.
You can download it here. Also, you can read the brief documentation.

Magiq-to-sql: Creating annonymous types at runtime

January 5, 2010 Leave a comment

In a previous post about magiq, I faced a problem that would be solved generating a class at runtime. The idea was to create a linq query that creates a new instance of this class just to look the generated sql statement. Since this feature is fully supported by Dynamic Query (I mean, the generation of new classes in order to support the classic “select new { … }”), I looked the source code in order to see how it was implemented. Since I don’t want to recreate the wheel, I used the same code (I’ve done some refactoring due coding style and left the “Copyright (C) Microsoft Corporation. All rights reserved.” comment. I hope this is enough).
The great news about using this dynamic class is that I can retrieve all the values I need using one single select:
For the update

items.Where( condition ).Set( x => x.Count, x => x.Count + 2)
                        .Set( x => x.Name, "test" )

and the insert

items.Where( condition ).Insert( x => new Item{ 
                                            Count = x.Count + 2 
                                            Name= "Test" 
                                           }).Into( items );

I have this matching select

items.Where( condition ).Select( x => new{ Count = x.Count + 2, Name = "Test");

This way I can handle everything with one single command generated by linq-to-sql :D. Indeed, I change the command text property of that command (and add some new parameters) so I can execute the update/insert without creating a new command.
Finally, the insert behaviour is pretty much the same as the update, so it was really easy to implement it once the update was done.
The first release of Magiq is coming 🙂

Categories: C#, Magiq Tags: , , ,

Magiq-to-sql: Knowing which table should be updated

January 2, 2010 Leave a comment

Today I will follow with the Magiq saga. You can take a look to the previous posts:

This post is about a problem I had with Magiq-to-sql. For an update, what magiq will do is:

  1. Generate a linq-to-sql query dynamically in order to retrieve each value that should be assigned to each property and the where clause.
  2. Create a command for that query.
  3. Parse the selected columns.
  4. Create the update statement setting each defined column to each selected column and using the same where as for the linq-to-sql query command.

For example, suppose you have this update:

people.Where( x => x.Name == "Ivo" ).Set( x => x.NickName, x.Name + "wiblo" ).Update();

The linq-to-sql query would be:

people.Where( x => x.Name == "Ivo" ).Select( x => new { NickName = x.Name + "wiblo" }  );

The select sql statement:

SELECT [t0].[Name] + @p1 FROM [People] [t0] WHERE [t0].Name = @p2

The column:

[t0].[Name] + @p1

The generated update sql statement:

UPDATE [People] SET [NickName] = [t0].[Name] + @p1 FROM [People] [t0] WHERE [t0].Name = @p2

If you take a look closer, you will realize that it’s using the actual table name instead of the alias used in the FROM clause. This is actually wrong. It work with that simple query but it becomes more complicated as the query does. For example, when you start updating properties of an association:

people.Where( x => x.Name == "Ivo" ).Set( x => x.User.NickName, x.Name + "wiblo" ).Update();

It will generate this UPDATE sql clause:

UPDATE <<TABLE>> SET [NickName] = [t0].Name + @1 FROM [People] AS [t0] LEFT OUTER JOIN [Users] AS [t1] ON [t1].[Id] = [t0].[UserId] WHERE [t0].[Name] = @p0

It could use the actual table name (UPDATE [Users] SET…) but it would crash if you have tables with references to itself. Anyway, using the actual table generates an update that doesn’t work properly (it updates more rows, I don’t know why. It makes sense, but I didn’t research about it).
So… the big question: How do it know which table to use? Looking the where clause, it’s easy for a human to realize it, but how tell the machine to think that way? Furthermore, there’re times when linq-to-sql uses subselects that introduces new tables in the game. Beautiful.

Well… I must say it was pretty easy. The only thing it had to do is to add a property of that table to the dynamically generated select as the first column to retrieve, so Magiq will know that the table used in that column is the right one. For the last example, it will generate this linq-to-sql query:

people.Where( x => x.Name == "Ivo" ).Select( x => new {
                                               __TypeReference__ = x.User.Id,
                                               NickName = x.Name + "wiblo"

And this select sql statement:

SELECT [t1].Id AS [__TypeReference__], [t0].Name + @1 AS [NickName] FROM [People] AS [t0] LEFT OUTER JOIN [Users] AS [t1] ON [t1].[Id] = [t0].[UserId] WHERE [t0].[Name] = @p0

Parsing the first column ([t1].Id AS [__TypeReference__]), it generates the following update sql statement:

UPDATE [t1] SET [NickName] = [t0].Name + @1 FROM [People] AS [t0] LEFT OUTER JOIN [Users] AS [t1] ON [t1].[Id] = [t0].[UserId] WHERE [t0].[Name] = @p0

I’m pretty happy with this 🙂

Categories: C#, Magiq Tags: , ,