getting transformer results from sklearn.pipeline.Pipeline

Question

I am using a sklearn.pipeline.Pipeline object for my clustering.

pipe = sklearn.pipeline.Pipeline([('transformer1': transformer1),
                                  ('transformer2': transformer2),
                                  ('clusterer': clusterer)])

Then I am evaluating the result by using the silhouette score.

sil = preprocessing.silhouette_score(X, y)

I'm wondering how I can get the X or the transformed data from the pipeline as it only returns the clusterer.fit_predict(X).

I understand that I can do this by just splitting the pipeline as

pipe = sklearn.pipeline.Pipeline([('transformer1': transformer1),
                                  ('transformer2': transformer2)])

X = pipe.fit_transform(data)
res = clusterer.fit_predict(X)
sil = preprocessing.silhouette_score(X, res)

but I would like to just do it all in one pipeline.

Answer 1

If you want to both fit and transform the data on intermediate steps of the pipeline then it makes no sense to reuse the same pipeline and better to use a new one as you specified, because calling fit() will forget all about previously learnt data.

However if you only want to transform() and see the intermediate data on an already fitted pipeline, then its possible by accessing the named_steps parameter.

new_pipe = sklearn.pipeline.Pipeline([('transformer1': 
                                           old_pipe.named_steps['transformer1']),
                                      ('transformer2': 
                                          old_pipe.named_steps['transformer2'])])

Or directly using the inner varible steps like:

transformer_steps = old_pipe.steps
new_pipe = sklearn.pipeline.Pipeline([('transformer1': transformer_steps[0]),
                                  ('transformer2': transformer_steps[1])])

And then calling the new_pipe.transform().

Update: If you have version 0.18 or above, then you can set the non-required estimator inside the pipeline to None to get the result in same pipeline. Its discussed in this issue at scikit-learn github Usage for above in your case:

pipe.set_params(clusterer=None)
pipe.transform(df)

But be aware to maybe store the fitted clusterer somewhere else to do so, else you need to fit the whole pipeline again when wanting to use that functionality.