retrieving more than 1000 rows from table storage using python

Question

I am using the following pattern from this forum to pull data from table storage:

next_pk=0
next_rk=0
while True:
    xa=table_service.query_entities(table_name=tableName, filter=dataFilter, num_results=1000, next_partition_key = next_pk, next_row_key = next_rk )
    i+=1
    if hasattr(accelxa, 'x_ms_continuation'):
        x_ms_continuation = getattr(accelxa, 'x_ms_continuation')
        next_pk = x_ms_continuation['nextpartitionkey']
        next_rk = x_ms_continuation['nextrowkey']
    else:
        break;

But when I run it I get the following exception:

TypeError was unhandled by user code
Message: query_entities() got an unexpected keyword argument 'next_partition_key'

So it would appear the api has changed from when the example was written and I am unable to determine what the new api is.

I have looked at github and there is reference to a marker object in the method call but I am unclear how to use it. Any example would be helpful. https://github.com/Azure/azure-storage-python/blob/master/azure/storage/table/tableservice.py

 def _query_entities(self, table_name, filter=None, select=None, max_results=None, marker=None, accept=TablePayloadFormat.JSON_MINIMAL_METADATA, property_resolver=None, timeout=None, _context=None): 

Answer 1

Look at Github code:

def _query_entities(self, table_name, filter=None, select=None, max_results=None,
                   marker=None, accept=TablePayloadFormat.JSON_MINIMAL_METADATA,
                   property_resolver=None, timeout=None, _context=None):
        ...

        next_partition_key = None if marker is None else marker.get('nextpartitionkey')

From this syntax, you have to initialize marker keyword argument as a dictionaty (dict) and set marker['nextpartitionkey'] (not 'next_partition_key') equal to value you want.

xa = table_service.query_entities(..., marker={'nextpartitionkey'= ...}, ...)

Also you should remove next_row_key keyword argument from your code and initialize marker['nextrowkey'] the same way.

Answer 2

I tried to reproduce the issue successfully, then I discovered the issue was caused by the query_entities method in your code which was incompatible with the azure-storage package version you installed.

According to the source code from GitHub, you can find the version difference as below.

In the version v0.20.x, for example, the latest version v0.20.3, the method query_entities definition as below, it has the arguments next_partition_key & next_row_key.

def query_entities(self, table_name, filter=None, select=None, top=None, 
                   next_partition_key=None, next_row_key=None):

But after v0.20.3, the method definition (for example v0.33.0) changed as @DmitryFrolov said, as below.

def query_entities(self, table_name, filter=None, select=None, num_results=None,
                   marker=None, accept=TablePayloadFormat.JSON_MINIMAL_METADATA,
                   property_resolver=None, timeout=None):

So there are two solutions for resolving the issue.

1. Rollback the package version via commands pip uninstall azure-storage & pip install azure-storage==0.20.3, then your code will works fine.
2. Under your current version, as @DmitryFrolov said, construct a marker object to include the parameters next_partition_key & next_row_key for following the new usage.