Dynamically generate LINQ select with nested properties

Question

Currently we have a package that generates linq select dynamically from fields from string. It works well with flat properties but it is not designed to work with nested fields like someObj.NestedObj.SomeField.

Our current code works as below in the service method:

_context.Shipments
    .Where(s => s.Id == request.Id) // it does not matter just an example
    .Select(request.Fields)
    .ToPage(request); // ToPage extension comes from a nuget package

The parameter "fields" of request object is just a string which seperated with commas including Shipment object's properties.

I made some refactoring to Shipment, I grouped some fields into a new class named as Address and add it to Shipment as below:

// before refactoring
class Shipment {
    // other fields...
    public string SenderAddress;
    public string SenderCityName;
    public string SenderCityId;

    public string RecipientAddress;
    public string CityName;
    public string CityId;
}

// after refactoring
class Shipment {
   // other fields...
   public Address Sender;
   public Address Recipient;
}

class Address {
    public string AddressText;
    public string CityName;
    public string CityId;
}

For the sake of current database mapping I added the corresponding mappings as :

public class ShipmentMap : DataEntityTypeConfiguration<Shipment>
    {
        public ShipmentMap()
        {
            ToTable("Shipments");
            // other property mappings
            Property(s => s.Recipient.AddressText).HasMaxLength(1100).HasColumnName("RecipientAddress");
            Property(s => s.Recipient.CityName).HasMaxLength(100).HasColumnName("CityName");
            Property(s => s.Recipient.CityId).IsOptional().HasColumnName("CityId");

            Property(s => s.Sender.AddressText).HasMaxLength(1100).HasColumnName("SenderAddress");
            Property(s => s.Sender.CityName).HasMaxLength(100).HasColumnName("SenderCityName");
            Property(s => s.Sender.CityId).IsOptional().HasColumnName("SenderCityId");
        }
    }

DataEntityTypeConfiguration comes from nuget packages as :

  public abstract class DataEntityTypeConfiguration<T> : EntityTypeConfiguration<T> where T : class
  {
    protected virtual void PostInitialize();
  }

So, my problem is with the select(fields) not works for when fields = "Recipient.CityId".

How can I dynamically generate linq for selecting with nested fields?

I tried below using LINQ : Dynamic select but it does not work.

// assume that request.Fields= "Recipient.CityId"

// in the service method
List<Shipment> x = _context.Shipments
    .Where(s => s.Id == request.Id)
    .Select(CreateNewStatement(request.Fields))
    .ToList();


 // I tried to generate select for linq here    
 Func<Shipment, Shipment> CreateNewStatement(string fields)
        {
            // input parameter "o"
            var xParameter = Expression.Parameter( typeof( Shipment ), "o" );

            // new statement "new Data()"
            var xNew = Expression.New( typeof( Shipment ) );

            // create initializers
            var bindings = fields.Split( ',' ).Select( o => o.Trim() )
                .Select(o =>
                {
                    string[] nestedProps = o.Split('.');
                    Expression mbr = xParameter;

                    foreach (var prop in nestedProps)
                        mbr = Expression.PropertyOrField(mbr, prop);

                    // property "Field1"
                    PropertyInfo mi = typeof( Shipment ).GetProperty( ((MemberExpression)mbr).Member.Name );
                    //
                    // original value "o.Field1"
                    var xOriginal = Expression.Property( xParameter, mi );

                    MemberBinding bnd = Expression.Bind( mi, xOriginal );
                    return bnd;
                });

            // initialization "new Data { Field1 = o.Field1, Field2 = o.Field2 }"
            var xInit = Expression.MemberInit( xNew, bindings );

            // expression "o => new Data { Field1 = o.Field1, Field2 = o.Field2 }"
            var lambda = Expression.Lambda<Func<Shipment,Shipment>>( xInit, xParameter );

            // compile to Func<Data, Data>
            return lambda.Compile();
        }

It throws exception because mbr becomes CityId after the loop and "mi" is null because there is no field CityId on shipment. What am I missing here? How can I create dynamic select for given string with nested properties?

UPDATE:

I found the solution and added it as answer, also I created a github gist for solution:

https://gist.github.com/mstrYoda/663789375b0df23e2662a53bebaf2c7c

Answer 1

It's good that you've found a solution of your specific problem.

Here is a more general solution which handles different source and target types as soon as the primitive property names and types match (e.g. Entity -> Dto etc.), as well as multiple levels of nesting:

public static Expression<Func<TSource, TTarget>> BuildSelector<TSource, TTarget>(string members) =>
    BuildSelector<TSource, TTarget>(members.Split(',').Select(m => m.Trim()));

public static Expression<Func<TSource, TTarget>> BuildSelector<TSource, TTarget>(IEnumerable<string> members)
{
    var parameter = Expression.Parameter(typeof(TSource), "e");
    var body = NewObject(typeof(TTarget), parameter, members.Select(m => m.Split('.')));
    return Expression.Lambda<Func<TSource, TTarget>>(body, parameter);
}

static Expression NewObject(Type targetType, Expression source, IEnumerable<string[]> memberPaths, int depth = 0)
{
    var bindings = new List<MemberBinding>();
    var target = Expression.Constant(null, targetType);
    foreach (var memberGroup in memberPaths.GroupBy(path => path[depth]))
    {
        var memberName = memberGroup.Key;
        var targetMember = Expression.PropertyOrField(target, memberName);
        var sourceMember = Expression.PropertyOrField(source, memberName);
        var childMembers = memberGroup.Where(path => depth + 1 < path.Length);
        var targetValue = !childMembers.Any() ? sourceMember :
            NewObject(targetMember.Type, sourceMember, childMembers, depth + 1);
        bindings.Add(Expression.Bind(targetMember.Member, targetValue));
    }
    return Expression.MemberInit(Expression.New(targetType), bindings);
}

The first two methods are just the publicly exposed high level helpers. The actual work is done by the private recursive NewObject method. It groups the current level properties and for each grouping, either creates simple assignment like PropertyN = source.Property1.Property2...PropertyN if it is the last level, or recursively PropertyN = new TypeN { … } otherwise.

Sample usage which matches the expression from your example:

var test = BuildSelector<Shipment, Shipment>(
    "Recipient.CityName, Sender.CityId, Sender.CityName, ParcelUniqueId");

Simply call Compile when you need Func.

Answer 2

Finally I found the solution. It generates correct lambda for two level nested properties like Shipment.Sender.CityName. So anyone who needs the same thing can use it.

I hope it helps.

/* this comes from request
*  request.Fields = "Sender.CityId,Sender.CityName,Recipient.CityName,parcelUniqueId"
*/

// in the service method

var shipmentList = _context.Shipments.
                .OrderByDescending(s => s.Id)
                .Skip((request.Page -1) * request.PageSize)
                .Take(request.PageSize)
                .Select(new SelectLambdaBuilder<Shipment>().CreateNewStatement(request.Fields))
                .ToList();

public class SelectLambdaBuilder<T>
{
    // as a performence consideration I cached already computed type-properties
    private static Dictionary<Type, PropertyInfo[]> _typePropertyInfoMappings = new Dictionary<Type, PropertyInfo[]>();
    private readonly Type _typeOfBaseClass = typeof(T);

    private Dictionary<string, List<string>> GetFieldMapping(string fields)
    {
        var selectedFieldsMap = new Dictionary<string, List<string>>();

        foreach (var s in fields.Split(','))
        {
            var nestedFields = s.Split('.').Select(f => f.Trim()).ToArray();
            var nestedValue = nestedFields.Length > 1 ? nestedFields[1] : null;

            if (selectedFieldsMap.Keys.Any(key => key == nestedFields[0]))
            {
                selectedFieldsMap[nestedFields[0]].Add(nestedValue);
            }
            else
            {
                selectedFieldsMap.Add(nestedFields[0], new List<string> { nestedValue });
            }
        }

        return selectedFieldsMap;
    }

    public Func<T, T> CreateNewStatement(string fields)
    {
        ParameterExpression xParameter = Expression.Parameter(_typeOfBaseClass, "s");
        NewExpression xNew = Expression.New(_typeOfBaseClass);

        var selectFields = GetFieldMapping(fields);

        var shpNestedPropertyBindings = new List<MemberAssignment>();
        foreach (var keyValuePair in selectFields)
        {
            PropertyInfo[] propertyInfos;
            if (!_typePropertyInfoMappings.TryGetValue(_typeOfBaseClass, out propertyInfos))
            {
                var properties = _typeOfBaseClass.GetProperties();
                propertyInfos = properties;
                _typePropertyInfoMappings.Add(_typeOfBaseClass, properties);
            }

            var propertyType = propertyInfos
                .FirstOrDefault(p => p.Name.ToLowerInvariant().Equals(keyValuePair.Key.ToLowerInvariant()))
                .PropertyType;

            if (propertyType.IsClass)
            {
                PropertyInfo objClassPropInfo = _typeOfBaseClass.GetProperty(keyValuePair.Key);
                MemberExpression objNestedMemberExpression = Expression.Property(xParameter, objClassPropInfo);

                NewExpression innerObjNew = Expression.New(propertyType);

                var nestedBindings = keyValuePair.Value.Select(v =>
                {
                    PropertyInfo nestedObjPropInfo = propertyType.GetProperty(v);

                    MemberExpression nestedOrigin2 = Expression.Property(objNestedMemberExpression, nestedObjPropInfo);
                    var binding2 = Expression.Bind(nestedObjPropInfo, nestedOrigin2);

                    return binding2;
                });

                MemberInitExpression nestedInit = Expression.MemberInit(innerObjNew, nestedBindings);
                shpNestedPropertyBindings.Add(Expression.Bind(objClassPropInfo, nestedInit));
            }
            else
            {
                Expression mbr = xParameter;
                mbr = Expression.PropertyOrField(mbr, keyValuePair.Key);

                PropertyInfo mi = _typeOfBaseClass.GetProperty( ((MemberExpression)mbr).Member.Name );

                var xOriginal = Expression.Property(xParameter, mi);

                shpNestedPropertyBindings.Add(Expression.Bind(mi, xOriginal));
            }
        }

        var xInit = Expression.MemberInit(xNew, shpNestedPropertyBindings);
        var lambda = Expression.Lambda<Func<T,T>>( xInit, xParameter );

        return lambda.Compile();
    }

It compiles the lambda as below:

s => new Shipment {
    Recipient = new Address {
        CityName = s.Recipient.CityName
    },
    Sender = new Address {
        CityId = s.Sender.CityId,
        CityName = s.Sender.CityName
    },
    ParcelUniqueId = s.ParcelUniqueId
}

I share the some screenshots from debug :