Archive for the ‘Magiq’ Category

Magiq-to-NHibernate: First commit!

February 22, 2010 1 comment

The past weekend I started to play with NHibernate for getting Magiq-to-NHibernate features up. It’s wonderful to work with an actually well done-designed framework 🙂
By now, only updates and deletes are working, and for simple classes that doesn’t have subclasses. Collection operations aren’t done either. Oh, and also it only works with SQL Server.
I know the actual problem will be when adding support to joined and union subclasses… well… one step at a time 🙂


Magiq: BulkCopy support

February 19, 2010 Leave a comment

With all the behavior stuff, I have normalized the design in favor of new Magiq providers. Now there’s a MagiqToDatabases abstract class that only needs a few methods to implement. Also, there is a normalization in the operations: Now Magiq have the classes DatabaseInsert, DatabaseUpdate, DatabaseQuery and DatabaseDelete, which contains properties like the columns to update, the expressions of the values, etc.

For implementing BuilkCopy, there was only two things needed to override:

  • The way Magiq-to-objects inserts into a SqlServer based destination: For example, for a Linq-to-Sql table, the way Magiq-to-objects inserts is using the Insert method of Table. The idea is to override that with a SqlBulkCopy. Note: this “way Magiq-to-objects inserts” is called IInsertDestination.
  • Use magiq-to-objects instead any SqlServer insert strategy defined: For using BulkCopy when the source belongs to the same provider, avoiding the regular insert…select… statement

So, following the behavior schema, Magiq provides two interfaces for all this: IObjectsInsertBehavior and IInsertBehavior.

  • IObjectsInsertBehavior provides the way for overriding the IInsertDestination.
  • IInsertBehavior provides the way for overriding the strategy to use, so we can avoid using the default strategy

Also, it needed a way to know if the destination is a SqlServer. Well… all the database operations and strategies implements an interface called IDatabaseBasedExecution that provides a IDatabaseAdapter. And this last interface give us a IDialect that could be a SqlServerDialect.

Ok, ok, but show me some code!!

public class BulkCopy : MagiqBehavior, IObjectsInsertBehavior, IInsertBehavior
	public int BatchSize { get; set; }
	public int Timeout { get; set; }

	public BulkCopy()
		BatchSize = 0;
		Timeout = 30;

	private static bool Applies(IDatabaseBasedExecution execution)
		return execution != null && execution.DatabaseAdapter.Dialect is SqlServerDialect;
	public IInsertDestination<TNew> GetInsertDestination<TSource, TDestination, TNew>(IInsert<TSource, TDestination, TNew> insert)
		where TDestination : class
		where TNew : class, TDestination, new()
		var insertDestination = Next<IObjectsInsertBehavior>().Call(x => x.GetInsertDestination(insert));
		var databaseOperation = insertDestination as IDatabaseBasedExecution;

		if (Applies(databaseOperation))
			return new BulkCopyDestination<TSource, TDestination, TNew>(insert, databaseOperation.DatabaseAdapter)
						   BatchSize = BatchSize,
						   Timeout = Timeout

		return insertDestination;

	public IInsertStrategy GetInsertStrategy<TSource, TDestination, TNew>(IInsert<TSource, TDestination, TNew> insert) where TDestination : class where TNew : class, TDestination, new()
		var strategy = Next<IInsertBehavior>().Call(x => x.GetInsertStrategy(insert));
		var databaseOperation = strategy as IDatabaseBasedExecution;

		if (Applies(databaseOperation))
			return new InsertStrategy<TSource, TDestination, TNew>(insert);

		//overrides the strategy with magiq-to-objects
		return strategy;

	public int Execute(IInsertStrategy strategy)
		//default behavior
		return Next<IInsertBehavior>().Call(x => x.Execute(strategy));

And the way to use it:

var sellsFromSomeRepository = ...;

       .Insert(x => new Record
                        Type = RecordType.Sell,
                        Description = "Sell from " + x.Date,
                        Amount = x.Total

And of course, if you use BulkCopy for inserting into a non SqlServer repository, the plugin continues without changes, so it could be used in a testing environment in which the repositories are setted to a List.

Magiq: Named scopes in C#

February 18, 2010 Leave a comment

Some weeks ago, my friend Jonas told me about Ruby’s ActiveRecord Named scopes and I thought what a marvelous feature! Later, I realize that it can be implemented in c# using Magiq.

Suppose you have a store that sells products and these products has expiration date. You call “Available” the products that had not yet expired. In order to find the not expired products, we could do, using Linq:

var availableProducts = store.Products.Where( x => x.ExpirationDate > DateTime.Now );

The problem with this approach is that the more time passes, more products the store will have and more expired, so it could retrieve the entire products collection just to filter some. Magiq to the rescue!

var availableProducts = store.Products.Query( x => x.ExpirationDate > DateTime.Now );

As I told in a previous post, it will generate a new sql query with the new condition.

Ok. In a perfect world, you will have something like an AvailableProducts property in the Store class, because the availability definition could change in the future.

public IEnumerable<Product> AvailableProducts {
    get {
         return Products.Query( x => x.ExpirationDate > DateTime.Now );

Now, what happens if besides store, we have a warehouse full of products and we need the available products from there? We will have add the same method in the Warehouse class.

Wouldn’t be nice if we could do it this way:

var availableProductsFromStore = store.Products.Available();

var availableProductsFromWarehouse = warehous.Products.Available();

Ok, we can 🙂

public static class ProductNamedScopes
	public static IEnumerable<Product> Available( this IEnumerable<Product> self )
		return self.Query( x => x.ExpirationDate > DateTime.Now );

I know, this could be implemented using pure linq, a foreach,, NHibernate’s collection filtering, etc. But Magiq let you hide all that stuff and use pure Linq in a generic way.

Magiq 0.3.0 available!

February 17, 2010 Leave a comment

I’m glad to announce that the new version of Magiq is available.
Magiq now support:

  • Plugin model
  • BulkCopy as a Plugin for using SqlBulkCopy when inserting into sql server from any source.

And also there was a huge refactoring in order to have simplicity when implementing new providers.

You can download this version from here.

Magiq: Plugin support & the insert API changes

January 30, 2010 Leave a comment

The the main feature of Magiq 0.3.0 is sql server bulk insert support, we needed to improve Magiq with plugins.
These plugins would act as interceptors, and they could be applied at configuration level (for example, logging) and query level (bulk insert).

What I wanted is to have a natural way to assign plugins that would work the same for the four operations: insert, update, delete and query.
Ok, what I wanted was actually complicated.
Since I thougth this for Bulk insert, I tried an API for Inserts:

source.Insert( x => new Item{ Name = x.Name })
      .Using( new BulkInsert() )
      .Into( destination );

Same could work for Updates, but no for Deletes and Queries, because those doesn’t have “intermediate” class. Let me explain: insert and update are performed using several calls. Insert have one for declaring the new expression ( x => new Item{ Name = x.Name }) and one for declaring the destination. The same happens with update, we have several for declaring the “sets” and one Update() to finish de operation. Delete and Query, however, works with one single call (They doesn’t need anything else in order to work), so I don’t have a place to put the “Using” method :(.

So, how could it be implemented for Delete and Query? I thought it should be before calling the operation, something like

source.Using( new BulkInsert() )
      .Insert( x => new Item{ Name = x.Name })
      .Into( destination );

The issue with this approach is that the plugin should be for all the operations and not the specific one. But I’m ok with that, I suppose It could be implemented like NHibernate listeners.
So… the idea is to wrap the IEnumerable or IQueryable with another one that is plugged to one or more plugins. Later, I could check for the type of the collection and make use of the plugins. The problem with this solution is that nothing ensures that collection to be the one used in the magiq operations. For example, something like this:

source.Using( new BulkInsert() )
      .Where( somecondition )
      .Insert( x => new Item{ Name = x.Name }).Into( destination );

Magiq will work with the last queryable and not the plugged one. I know, I could check all the chain but, again, nothing ensures that the plugged collection is there.

Talking to my friend Rezlaj, we got to a solution I don’t like a lot but… it’s a solution. The idea is to have all the stuff inside a interface IMagiq that could be getted using an extension method Magiq applied to a collection. It would look like:

      .Where( somecondition )
      .Insert( x => new Item{ Name = x.Name }).Into( destination );

This IMagiq interface have the methods to add plugins and could have another methods to interact with the operation in the future. Of course, there still are the extensions methods Delete, Set, ToQueryable (well, not this last one, because I changed it to Query because I think is better). But they delegates all to the IMagiq interface.
I think I should change the blog title to “The problem with that…”. You may ask way…

The problem with that (the previous that, actually. I mean, the new API) is that all the operations are applied to a T where T : class except Insert. Why? WHY?! because the source of an insert operation could be a colection of int’s, date’s, whatever. For example, this should be supported (and it is!):

sells.Where( x => x.Date == DateTime.Now)
     .Select( x => x.Total )
     .Insert( x => new SellRecord{ Total = x.Name, Date = DateTime.Now })
     .Into( records );

(I know, the example is not the happiest one, but it’s helpful for explaining my point)
So… the source collection is a collection of decimals. Ok, how could it be solved? because IMagiq needs the T : class constraint. Well, there was an idea in my head for the last couple of days about how linq and all the magiq operations except insert reverses the analogous sql structure. So… It makes more sense to have the insert reversed. Something like:

records.Insert( x => new SellRecord{ Total = x.Name, Date = DateTime.Now })
       .From( sells.Where( x => x.Date == DateTime.Now) )

Ok, it’s just an example, it won’t compile (but It will complain… cuak). I need the source type before the NewExpression, so it have no choice that be:

records.From( sells.Where( x => x.Date == DateTime.Now) )
       .Insert( x => new SellRecord{ Total = x.Name, Date = DateTime.Now })

Now, the collection wrapped by IMagiq is the destination and it could have the constraint.

So, summarizing:

  • Magiq() Extension method was added
  • IMagiq adds support to plugins
  • Insert API changed to destination.From( source ).Insert( new )

Magiq-0.2.0 is avaliable!

January 26, 2010 Leave a comment

I’m glad to annunce that the new version of Magiq is out. You can download it here.

Magiq 0.2.0 supports:

  • Mass operations to Linq-to-sql collections (EntitySet)
  • Collection filterig support for Linq-to-sql
  • Support to sorting and grouping
  • Supports when you want to insert into a repository from another one (for example, insert a lot of entities from a string collection). In this case, it uses the same approach as using a foreach, but it works 😉

Magiq: Querying a collection

January 11, 2010 1 comment

One of the future features of Magiq will be mass operations to collections. I mean that

var country = CountryRepository.GetById(1);
country.Cities.Where( x => x.Population > 100000 ).Set( x => x.IsBigCity, true ).Update();

should generate a t-sql like

UPDATE [t0] SET [t0].IsBigCity = @p2 FROM Cities [t0] where [t0].Population > @p1 AND [t0].CountryId = @p2

Of course, this should be possible only if the collection is a Lazy one, so all the information needed is inside of it (it’s the same information needed to retrieve the items). For Linq-to-sql is EntitySet. For NHibernate, AbstractPersistentCollection (PersistentGenericBag, PersistentGenericList, PersistentGenericSet and PersistentGenericMap).

The problem (because there’re always problems) is that those classes doesn’t implement IQueryable and the previous magiq statement will work always as magiq-to-objects, because the collection is enumerated once the Where extension method is called and the Set method is called to a simple IEnumerable.

Talking with Sebas and my friend Jonas, we conclude that the best way to solve this is adding an extension method ToQueryable that converts the lazy enumerable to a queryable one (I would like to call it “AsQueryable”, but it already exists :S). I know, it’s a workaroud, but… it seem like I have no choice.

So, the API will look like:

var country = CountryRepository.GetById(1);
country.Cities.ToQueryable().Where( x => x.Population > 100000 ).Set( x => x.IsBigCity, true ).Update();

The great news is that you can use it for sub selecting collections. It would be the same feature that nhibernate has, but for all the orm’s that have lazy collections (of course, nhibernate magiq provider will implement this feature using it own feature).

Something like:

var country = CountryRepository.GetById(1);
country.Cities.ToQueryable().Where( x => x.Population > 100000 );

will execute

SELECT [t0].Id, [t0].Population, [t0].Name FROM Cities [t0] where [t0].Population > @p1 AND [t0].CountryId = @p2

Isn’t it pretty?