Here's a dataframe:
A B C
0 6 2 -5
1 2 5 2
2 10 3 1
3 -5 2 8
4 3 6 2
I could retrieve a column which is basically a tuple of columns from the original df
using df.apply
:
out = df.apply(tuple, 1)
print(out)
0 (6, 2, -5)
1 (2, 5, 2)
2 (10, 3, 1)
3 (-5, 2, 8)
4 (3, 6, 2)
dtype: object
But if I want a list of values instead of a tuple of them, I can't do it, because it doesn't give me what I expect:
out = df.apply(list, 1)
print(out)
A B C
0 6 2 -5
1 2 5 2
2 10 3 1
3 -5 2 8
4 3 6 2
Instead, I need to do:
out = pd.Series(df.values.tolist())
print(out)
0 [6, 2, -5]
1 [2, 5, 2]
2 [10, 3, 1]
3 [-5, 2, 8]
4 [3, 6, 2]
dtype: object
Why can't I use df.apply(list, 1)
to get what I want?
Appendix
Timings of some possible workarounds:
df_test = pd.concat([df] * 10000, 0)
%timeit pd.Series(df.values.tolist()) # original workaround
10000 loops, best of 3: 161 µs per loop
%timeit df.apply(tuple, 1).apply(list, 1) # proposed by Alexander
1000 loops, best of 3: 615 µs per loop
The culprit is here. With func=tuple
it works, but using func=list
raises an exception from within the compiled module lib.reduce
:
ValueError: ('function does not reduce', 0)
As you can see, they catch the exception but don't bother to handle it.
Even without the too-broad except clause, that's a bug in pandas. You might try to raise it on their tracker, but similar issues have been closed with some flavour of wont-fix or dupe.
16321: weird behavior using apply() creating list based on current columns
15628: Dataframe.apply does not always return a Series when reduce=True
That latter issue got closed, then reopened, and converted into a docs enhancement request some months ago, and now seems to be being used as a dumping ground for any related issues.
Presumably it's not a high priority because, as piRSquared commented (and one of the pandas maintainers commented the same), you're better off with a list comprehension:
pd.Series([list(x) for x in df.itertuples(index=False)])
Typically apply
would be using a numpy ufunc or similar.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With