Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How can I count the number of consecutive TRUEs in a DataFrame?

I have a dataset made of True and False.

Sample Table:
       A      B      C
0  False   True  False
1  False  False  False
2   True   True  False
3   True   True   True
4  False   True  False
5   True   True   True
6   True  False  False
7   True  False   True
8  False   True   True
9   True  False  False

I want to count the number of consecutive True values for every column, and if there's more than one consecutive True series, I want to get the max of it.

For the table above, I would get:

length = [3, 4, 2]

I found similar threads but none resolved my problem.

Since I do and will have many more columns(products), I need to do this regardless of the column name, for the whole table and get an array as the result.

And if possible, I'd like to learn the index of the first true of the longest sequence aka where this longest true series starts, so the result would be for this one:

index = [5, 2, 7]
like image 766
meliksahturker Avatar asked Oct 09 '18 09:10

meliksahturker


People also ask

How do you count trues in a data frame?

Select the Dataframe column using the column name and subscript operator i.e. df['C']. It returns the column 'C' as a Series object of only bool values. After that, call the sum() function on this boolean Series object, and it will return the count of only True values in the Series/column.

How do I count the number of values in a row in Pandas?

In Pandas, You can get the count of each row of DataFrame using DataFrame. count() method. In order to get the row count you should use axis='columns' as an argument to the count() method. Note that the count() method ignores all None & nan values from the count.

How do you count occurrences of a string in Pandas?

The str. count() function is used to count occurrences of pattern in each string of the Series/Index. This function is used to count the number of times a particular regex pattern is repeated in each of the string elements of the Series.

How do I count the number of values in a column in a DataFrame?

Use Sum Function to Count Specific Values in a Column in a Dataframe. We can use the sum() function on a specified column to count values equal to a set condition, in this case we use == to get just rows equal to our specific data point.


1 Answers

Solution should be simplify, if always at least one True per column:

b = df.cumsum()
c = b.sub(b.mask(df).ffill().fillna(0)).astype(int)

print (c)
   A  B  C
0  0  1  0
1  0  0  0
2  1  1  0
3  2  2  1
4  0  3  0
5  1  4  1
6  2  0  0
7  3  0  1
8  0  1  2
9  1  0  0

#get maximal value of all columns
length = c.max().tolist()
print (length)
[3, 4, 2]

#get indexes by maximal value, subtract length and add 1 
index = c.idxmax().sub(length).add(1).tolist()
print (index)
[5, 2, 7]

Detail:

print (pd.concat([b,
                  b.mask(df), 
                  b.mask(df).ffill(), 
                  b.mask(df).ffill().fillna(0),
                  b.sub(b.mask(df).ffill().fillna(0)).astype(int)
                  ], axis=1, 
                  keys=('cumsum', 'mask', 'ffill', 'fillna','sub')))

  cumsum       mask           ffill           fillna           sub      
       A  B  C    A    B    C     A    B    C      A    B    C   A  B  C
0      0  1  0  0.0  NaN  0.0   0.0  NaN  0.0    0.0  0.0  0.0   0  1  0
1      0  1  0  0.0  1.0  0.0   0.0  1.0  0.0    0.0  1.0  0.0   0  0  0
2      1  2  0  NaN  NaN  0.0   0.0  1.0  0.0    0.0  1.0  0.0   1  1  0
3      2  3  1  NaN  NaN  NaN   0.0  1.0  0.0    0.0  1.0  0.0   2  2  1
4      2  4  1  2.0  NaN  1.0   2.0  1.0  1.0    2.0  1.0  1.0   0  3  0
5      3  5  2  NaN  NaN  NaN   2.0  1.0  1.0    2.0  1.0  1.0   1  4  1
6      4  5  2  NaN  5.0  2.0   2.0  5.0  2.0    2.0  5.0  2.0   2  0  0
7      5  5  3  NaN  5.0  NaN   2.0  5.0  2.0    2.0  5.0  2.0   3  0  1
8      5  6  4  5.0  NaN  NaN   5.0  5.0  2.0    5.0  5.0  2.0   0  1  2
9      6  6  4  NaN  6.0  4.0   5.0  6.0  4.0    5.0  6.0  4.0   1  0  0

EDIT:

General solution working with only False columns - add numpy.where with boolean mask created by DataFrame.any:

print (df)
       A      B      C
0  False   True  False
1  False  False  False
2   True   True  False
3   True   True  False
4  False   True  False
5   True   True  False
6   True  False  False
7   True  False  False
8  False   True  False
9   True  False  False

b = df.cumsum()
c = b.sub(b.mask(df).ffill().fillna(0)).astype(int)

mask = df.any()
length = np.where(mask, c.max(), -1).tolist()
print (length)
[3, 4, -1]

index =  np.where(mask, c.idxmax().sub(c.max()).add(1), 0).tolist()
print (index)
[5, 2, 0]
like image 58
jezrael Avatar answered Oct 04 '22 23:10

jezrael