dask df.col.unique() vs df.col.drop_duplicates()

Question

In dask what is the difference between

df.col.unique()

and

df.col.drop_duplicates()

Both return a series containing the unique elements of df.col. There is a difference in the index, unique result is indexed by 1..N while drop_duplicates indexed by an arbitrary looking sequence of numbers.

What is the significance of the index returned by drop_duplicates?

Is there any reason to use one over the other if the index is not important?

Answer 1

Dask.dataframe has both because Pandas has both, and dask.dataframe mostly copies the Pandas API. Unique is a holdover from Pandas' history with Numpy.

In [1]: import pandas as pd

In [2]: df = pd.DataFrame({'x': [1, 2, 1], 'y': [1., 2., 3.]}, index=pd.Index(['a', 'b', 'A'], name='I'))

In [3]: df.x.drop_duplicates()
Out[3]: 
I
a    1
b    2
Name: x, dtype: int64

In [4]: df.x.unique()
Out[4]: array([1, 2])

In dask.dataframe we deviate slightly and choose to use a dask.dataframe.Series rather than a dask.array.Array because one can't precompute the length of the array and so can't act lazily.

In practice there is little reason to use unique over drop_duplicates