Language Integrated Query

From Wikipedia, the free encyclopedia

Language Integrated Query (LINQ, pronounced "link") is a Microsoft .NET Framework component that adds native data querying capabilities to .NET languages.

Microsoft LINQ defines a set of proprietary query operators that can be used to query, project and filter data in arrays, enumerable classes, XML (XLINQ), relational database, and third party data sources. While it allows any data source to be queried, it requires that the data be encapsulated as objects. So, if the data source does not natively store data as objects, the data must be mapped to the object domain. Queries written using the query operators are executed either by the LINQ query processing engine or, via an extension mechanism, handed over to LINQ providers which either implement a separate query processing engine or translate to a different format to be executed on a separate data store (such as on a database server as SQL queries (DLINQ)). The results of a query are returned as a collection of in-memory objects that can be enumerated using a standard iterator function such as C#'s foreach.

Many of the concepts that LINQ has introduced were originally tested in Microsoft's Cω research project. LINQ was released as a part of .NET Framework 3.5 on November 19, 2007.

[edit] Architecture of LINQ in .Net Framework 3.5

[edit] Standard Query Operators

The set of query operators defined by LINQ are exposed to the user as the Standard Query Operator API. The query operators supported by the API are:^[1]

Select / SelectMany

The Select statement performs a projection on the collection to select either all the data members that make up the object or a subset of it. The SelectMany operator performs a one-to-many projection("nested"), i.e., if the objects in the collection contain another collection as a data member, SelectMany can be used to select the entire sub-collection. The user supplies a function, as a delegate, which projects the data members. Selection creates an object of a different type, which has either some or as many data members as the original class. The class must be already defined for the code to be compilable.

Where

The Where operator allows the definition of a set of predicate rules which are evaluated for each object in the collection, while objects which do not match the rule are filtered away. The predicate is supplied to the operator as a delegate.

Sum / Min / Max / Average / Aggregate

These operators take a predicate that retrieves a certain numeric value from each element in the collection and uses it to find the sum, minimum, maximum, average or aggregate values of all the elements in the collection, respectively.

Join / GroupJoin

The Join operator performs an inner join on two collections, based on matching keys for objects in each collection. It takes two functions as delegates, one for each collection, that it executes on each object in the collection to extract the key from the object. It also takes another delegate via which the user specifies which data elements, from the two matched elements, should be used to create the resultant object. The GroupJoin operator performs a group join. Like the Select operator, the results of a join are instantiations of a different class, with all the data members of both the types of the source objects, or a subset of them.

Take / TakeWhile

The Take operator selects the first n objects from a collection, while the TakeWhile operator, which takes a predicate, selects those objects which match the predicate.

Skip / SkipWhile

The Skip and SkipWhile operators are complements of Take and TakeWhile - they skip the first n objects from a collection, or those objects which match a predicate (for the case of SkipWhile).

OfType

The OfType operator is used to select the elements of a certain type.

Concat

The Concat operator concatenates two collections.

OrderBy / ThenBy

The OrderBy operator is used to specify the primary sort ordering of the elements in a collection according to some key. The default ordering is in ascending order, to reverse the order the OrderByDescending operator is to be used. ThenBy and ThenByDescending specifies subsequent ordering of the elements. The function to extract the key value from the object is specified by the user as a delegate.

Reverse

The Reverse operator reverses a collection.

GroupBy

The GroupBy operator takes a delegate that extracts a key value and returns a collection of IGrouping<Key, Values> objects, for each distinct key value. The IGrouping objects can then be used to enumerate all the objects for a particular key value.

Distinct

The Distinct operator removes duplicate instances of a key value from a collection. The function to retrieve the key value is to be supplied as a delegate.

Union / Intersect / Except

These operators are used to perform a union, intersection and difference operation on two sequences, respectively.

EqualAll

The EqualAll operator checks if all elements in two collections are equal.

First / FirstOrDefault / Last / LastOrDefault

These operators take a predicate. The First operator returns the first element for which the predicate yields true or throws an exception if nothing matches. The FirstOrDefault operator is like the First operator except that it returns the default value for the element type (usually a null reference) in case nothing matches the predicate. The last operator retrieves the last element to match the predicate, or throws an exception in case nothing matches. The LastOrDefault returns the default element value if nothing matches.

Single

The Single operator takes a predicate and returns the element which matches the predicate. An exception is thrown if none or more than one element match the predicate.

ElementAt

The ElementAt operator retrieves the element at a given index in the collection.

Any / All / Contains

The Any operator checks if there are any element in the collection matching the predicate. It does not select the element, but returns true for a match. The All operator checks if all elements match the predicate. The Contains operator checks if the collection contains a given value.

Count

The Count operator counts the number of elements in the given collection.

The Standard Query Operator also specifies certain operators which converts a collection into other types:^[1]

• AsEnumerable: converts the collection to IEnumerable<T> type.
• ToQueryable: converts the collection to IQueryable<T> type.
• ToArray: converts the collection to an array.
• ToList: converts the collection to IList<T> type.
• ToDictionary: converts the collection to IDictionary<K, T> type, indexed by the key K.
• ToLookup: converts the collection to ILookup<K, T> type, indexed by the key K.
• Cast: converts a non-generic IEnumerable collection to one of IEnumerable<T> by casting each element to type T. Throws an exception for incompatible types.
• OfType: converts a non-generic IEnumerable collection to one of IEnumerable<T>. Only elements of type T are included.

The query operators are defined in the IEnumerable<T> interface as generic extension methods, and a concrete implementation is provided in the Sequence class. As a result, any class which implements the IEnumerable<T> interface has access to these methods and are queryable. LINQ also defines a set of generic Func delegates, which define the type of delegates handled by the LINQ query methods. Any function wrapped in a Func delegate can be used by LINQ. Each of these methods return an IEnumerable<T>, so the output of one can be used as input to another, resulting in query composability. The functions, however, are lazily evaluated, i.e., the collections are enumerated only when the result is retrieved. The enumeration is halted as soon as a match is found, and the delegates evaluated on it. When a subsequent object in the resultant collection is retrieved, the enumeration of the source collection is continued beyond the element already evaluated. However, grouping operations, like GroupBy and OrderBy, as well as Sum, Min, Max, Average and Aggregate, require data from all elements in collection, and force an eager evaluation. LINQ does not feature a query optimizer and the query operators are evaluated in the order they are invoked. The LINQ methods are compilable in .NET Framework 2.0, as well.^[1]

[edit] Language Extensions

While LINQ is primarily implemented as a library for .NET Framework 3.5, it also defines a set of language extensions that can be optionally implemented by languages to make queries a first class language construct and provide syntactic sugar for writing queries. These language extensions have initially been implemented in C# 3.0, VB 9.0 and Oxygene, with other languages like F# and Nemerle having announced preliminary support. The language extensions include:^[2]

• Query syntax: Languages are free to choose a query syntax, which it will recognize natively. These language keywords must be translated by the compiler to appropriate LINQ method calls. The languages can implement operator reordering and other optimizations at the keyword level.
• Implicitly typed variables: This enhancement allows variables to be declared without specifying their types. The languages C# 3.0 and Oxygene declare them with the var keyword. In VB9.0, the use of the Dim keyword without type declaration accomplishes the same declaration. Such objects are still strongly typed; for these objects the compiler uses type inference to infer the type of the variables. This allows the result of the queries to be specified and their result defined without declaring the type of the intermediate variables.
• Anonymous types: Anonymous types allow classes, which contain only data member declarations, to be inferred by the compiler. This is useful for the Select and Join operators, whose result types may differ from the types of the original objects. The compiler uses type inference to determine the fields contained in the classes and generates accessors and mutators for these fields.
• Object Initializer: Object initializers allow an object to be created and initialized in a single scope, this allows creation of delegates that extract fields from an object, create a new object and assign the extracted data to the fields of the new object in a single statement, as is required for Select and Join operators.
• Lambda expressions: Lambda expressions are used to create delegates inline with other code. This allows the predicates and extraction functions to be written inline with the queries.

For example, in the query to select all the objects in a collection with SomeProperty less than 10,

int someValue = 5; 
 
var results =  from c in someCollection
               let x = someValue * 2
               where c.SomeProperty < x
               select new {c.SomeProperty, c.OtherProperty};
 
foreach (var result in results)
{
        Console.WriteLine(result);
}

the types of variables result, c and results all are inferred by the compiler - assuming SomeCollection is IEnumerable<SomeClass>, c will be SomeClass, results will be IEnumerable<SomeOtherClass> and result will be SomeOtherClass, where SomeOtherClass will be a compiler generated class with only the SomeProperty and OtherProperty properties and their values set from the corresponding clauses of the source objects. The operators are then translated into method calls as:

IEnumerable<SomeOtherClass> results =
     SomeCollection.Where
     (
          c => c.SomeProperty < (SomeValue * 2)
     )
     .Select
     (
          c => new {c.SomeProperty, c.OtherProperty}
     )
foreach (SomeOtherClass result in results)
{
     Console.WriteLine(result.ToString());
}

[edit] LINQ Providers

LINQ also defines another interface, IQueryable<T>, which defines the same interfaces to the Standard Query Operators as IEnumerable<T>. However, the concrete implementation of the interface, instead of evaluating the query, converts the query expression, with all the operators and predicates, into an expression tree.^[3] The Expression tree preserves the high level structure of the query and can be examined at runtime. The type of the source collection defines which implementation will run - if the collection type implements IEnumerable<T>, it executes the local LINQ query execution engine and if it implements the IQueryable<T> implementation, it invokes the expression tree-based implementation. An extension method is also defined for IEnumerable<T> collections to be wrapped inside an IQueryable<T> collection, to force the latter implementation.

The expression trees are at the core of LINQ extensibility mechanism, by which LINQ can be adapted for any data source. The expression trees are handed over to LINQ Providers, which are data source-specific implementations that adapt the LINQ queries to be used with the data source. The LINQ Providers analyze the expression trees representing the query ("query trees") and generate a DynamicMethod (which are methods generated at runtime) by using the reflection APIs to emit CIL code. These methods are executed when the query is run.^[3] LINQ comes with LINQ Providers for in-memory object collections, SQL Server databases, ADO.NET datasets and XML documents. These different providers define the different flavors of LINQ:

LINQ to Objects

The LINQ to Objects provider is used for querying in-memory collections, using the local query execution engine of LINQ. The code generated by this provider refer the implementations of the standard query operators as defined in the Sequence class and allows IEnumerable<T> collections to be queried locally. Current implementation of LINQ to Objects uses e.g. O(n) linear search for simple lookups, and is not optimised for complex queries^[4].

LINQ to XML

The LINQ to XML provider converts an XML document to a collection of XElement objects, which are then queried against using the local execution engine that is provided as a part of the implementation of the standard query operator.^[5]

LINQ to SQL

The LINQ to SQL provider allows LINQ to be used to query SQL Server databases as well as SQL Server Compact databases. Since SQL Server data resides on a remote server, and because it already includes a querying engine, LINQ to SQL does not use the query engine of LINQ. Instead, it converts a LINQ query to a SQL query which is then sent to SQL Server for processing.^[6] However, since SQL Server stores the data as relational data and LINQ works with data encapsulated in objects, the two representations must be mapped to one another. For this reason, LINQ to SQL also defines the mapping framework. The mapping is done by defining classes which corresponds to the tables in database, and containing all or a subset of the columns in the table as data members.^[7] The correspondence, along with other relational model attributes such as primary keys are specified using LINQ to SQL-defined attributes. For example,

[Table(Name="Customers")]
public class Customer
{
     [Column(IsPrimaryKey = true)]
     public int CustID;
 
     [Column]
     public string CustName;
}

this class definition maps to a table named Customers and the two data members correspond to two columns. The classes must be defined before LINQ to SQL can be used. Visual Studio 2008 includes a mapping designer which can be used to create the mapping between the data schemas in the object as well as relational domain. It can automatically create the corresponding classes from a database schema, as well as allow manual editing to create a different view by using only a subset of the tables or columns in a table.^[7]

The mapping is implemented by the DataContext which takes a connection string to the server, and can be used to generate a Table<T> where T is the type that the database table will be mapped to. The Table<T> encapsulates the data in the table, and implements the IQueryable<T> interface, so that the expression tree is created, which the LINQ to SQL provider handles. It converts the query into T-SQL and retrieves the result set from the database server. Since the processing happens at the database server, local methods, which are not defined as a part of the lambda expressions representing the predicates, cannot be used. However, it can use the stored procedures on the server. Any changes to the result set are tracked and can be submitted back to the database server.^[7]

LINQ to DataSets

The LINQ to SQL provider works only with Microsoft SQL Server databases; to support any generic database, LINQ also includes the LINQ to DataSets, which uses ADO.NET to handle the communication with the database. Once the data is in ADO.NET Datasets, LINQ to Datasets execute queries against these datasets.^[8]

Other providers

The LINQ providers can be implemented by third parties for various data sources as well. Several database server specific providers are available from the database vendors. Some of the popular providers include:

• Data Services: LINQ to ADO.NET Data Services^[9]
• Entity Framework: LINQ to Entities^[10]
• DbLinq: LINQ to MySQL, PostgreSQL, Oracle, Ingres, SQLite and Microsoft SQL Server^[11]
• Google search: LINQ to Google^[12]
• Windows Search: LINQ to System Search^[13]
• NHibernate: LINQ to NHibernate^[14]
• OpenMapi: LINQ to MAPI^[15]
• LLBLGen: LINQ to LLBLGen^[16]

[edit] Performance

Some benchmark on simple use cases tend to show that LINQ to Objects performance have a large overhead compared to normal operation^[17][18].

LINQ to XML and LINQ to SQL performance compared to ADO.NET depend on the use case^[19].

[edit] PLINQ

This article or section contains information about scheduled or expected future software. The content may change as the software release approaches and more information becomes available.

Microsoft, as a part of the Parallel FX Library, is developing PLINQ, or Parallel LINQ, a parallel execution engine for LINQ queries. It defines the IParallelEnumerable<T> interface. If the source collection implements this interface, the parallel execution engine is invoked. The PLINQ engine executes a query in a distributed manner on a multi-core or multi-processor system.^[20]

[edit] Other language implementations

• JSINQ is Kai Jäger's JavaScript implementation of LINQ to Objects. Also provides a compiler that translates LINQ-style query expressions into JavaScript code.
• Chris Pietschmann's LINQ to JavaScript is a LINQ implementation which extends JavaScript's Array object with LINQ capabilities.
• PHPLinq is Maarten Balliauw's PHP implementation of LINQ.
• Quaere is a Java implementation of LINQ.
• JaQue is a typesafe Java implementation of LINQ.
• JaQu a Java implementation of LINQ.
• Querydsl is a typesafe Java implementation of LINQ.

[edit] References

[edit] See also

• Object-relational mapping
• Object-relational impedance mismatch
• http://weblogs.asp.net/rajbk/archive/2007/09/26/building-an-rss-feed-using-linq-to-xml.aspx

[edit] External links

• Official Microsoft LINQ Project
• 101 C# LINQ Samples
• 101 Visual Basic LINQ Samples
• LINQ to XML Documentation
• Microsoft LINQ forum
• Linq to Objects for the .NET developer
• LINQ page on NetFXGuide.com
• LINQ wiki
• LINQ books
• Continuous LINQ
• LINQ To Sharepoint
• LINQ To Active Directory
• LINQ for Novell.Directory.Ldap
• Linq Tutorials and Active Articles
• Looking to LINQ - Will Microsoft's Language Integrated Query transform programmatic data access?
• Obtics (Observable Object LINQ)
• LINQ to SNMP
• Different Ways Of Retrieving Data From Collections
• LINQ Exchange - Learn LINQ and Lambda Expressions