How to return transformed data from an ML.Net pipeline before a predictor is applied

Question

Here is the creation of the ML.Net pipeline object copied from the TaxiFarePrediction example.

        LearningPipeline pipeline = new LearningPipeline
        {
            new TextLoader(TrainDataPath).CreateFrom<TaxiTrip>(separator:','),
            new ColumnCopier(("FareAmount", "Label")),
            new CategoricalOneHotVectorizer("VendorId","RateCode","PaymentType"),
            new ColumnConcatenator("Features","VendorId","RateCode","PassengerCount","TripDistance","PaymentType"),
            new FastTreeRegressor()
        };

Essentially, I'd like to return the data after the ColumnCopier, the CategoricalOneHotVectorizer and the ColumnConcatenator have been applied.

Answer 1

For visualization in debugger Microsoft programmed class

LearningPipelineDebugProxy

That class has two fields which are quite informative: Rows and Columns. And of course class that is intended for debugging is not very easy to create because it is internal sealed:

namespace Microsoft.ML
{
  /// <summary>
  /// The debug proxy class for a LearningPipeline.
  /// Displays the current columns and values in the debugger Watch window.
  /// </summary>
  internal sealed class LearningPipelineDebugProxy
  {

according to source code. In such cases if debugger visualization is not something that is enough I use reflection. In order to create instance of LearningPipelineDebugProxy in TaxiTrip instance I've used following tricks:

1. Created instance of LearningPipelineDebugProxy via CreateInstance method
2. Added method that via reflection gets needed field.

In code that fragment looked like this:

    public static PredictionModel<TaxiTrip, TaxiTripFarePrediction> Train()
    {
        var pipeline = new LearningPipeline();
        pipeline.Add(new TextLoader(_datapath).CreateFrom<TaxiTrip>(useHeader: true, separator: ','));


        Type obj = AppDomain.CurrentDomain.GetAssemblies().SelectMany(t => t.GetTypes()).
            Where(t => String.Equals(t.Name, "LearningPipelineDebugProxy", StringComparison.Ordinal)).First();


        var instObject = Activator.CreateInstance(obj, new []{pipeline});

        pipeline.Add(new ColumnCopier(("FareAmount", "Label")));
        pipeline.Add(new CategoricalOneHotVectorizer("VendorId", "RateCode", "PaymentType"));
        pipeline.Add(new ColumnConcatenator("Features", "VendorId", "RateCode", "PassengerCount", "TripDistance", "PaymentType"));

        var rws = GetPropValue(instObject, "Rows");
        var clms = GetPropValue(instObject, "Columns");

        pipeline.Add(new FastTreeRegressor());

        PredictionModel<TaxiTrip, TaxiTripFarePrediction> model = pipeline.Train<TaxiTrip, TaxiTripFarePrediction>();
        return model;
    }

    public static object GetPropValue(object src, string propName)
    {
        return src.GetType().GetProperty(propName).GetValue(src, null);
    }

In debugger window and not only in debugger Rows become available: