MongoEngine throws exception TypeError: cannot deepcopy this pattern object

Question

I am getting the exception on querying mongodb model (Python/MongoEngine) with list and regex query. The code is

from mongoengine import *
import re 

db = connect('testdb')

class Team(Document):
    name = StringField()
    groups = ListField(ReferenceField('Group'))

class Group(Document):
    name = StringField()

Team.drop_collection()
Group.drop_collection()

g1 = Group('G1')
g1.save()

g2 = Group('G2')
g2.save()

g3 = Group('G3')
g3.save()

g4 = Group('G4')
g4.save()

t = Team('Team1',[g1,g2,g3])
t.save()

t = Team('Team2',[g1,g2,g4])
t.save()

t = Team('Team3',[])
t.save()

t = Team('Team3',[g3,g2])
t.save()

t = Team('Team3',[g4,g1])
t.save()


# TypeError: cannot deepcopy this pattern object
teams = Team.objects( Q(groups__in=[g3,g2]) & Q(name=re.compile('eam3')))
for team in teams:
   print team.name  


#-------------------------

The exception traceback is

Traceback (most recent call last):
  File "so_mongoengine_query.py", line 46, in <module>
    for team in teams:
  File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/queryset.py", line 81, in _iter_results
    self._populate_cache()
  File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/queryset.py", line 93, in _populate_cache
    self._result_cache.append(self.next())
  File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/base.py", line 1137, in next
    raw_doc = self._cursor.next()
  File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/base.py", line 1182, in _cursor
    self._cursor_obj = self._collection.find(self._query,
  File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/base.py", line 1215, in _query
    self._mongo_query = self._query_obj.to_query(self._document)
  File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/visitor.py", line 91, in to_query
    query = self.accept(SimplificationVisitor())
  File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/visitor.py", line 141, in accept
    return visitor.visit_combination(self)
  File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/visitor.py", line 41, in visit_combination
    return Q(**self._query_conjunction(queries))
  File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/visitor.py", line 61, in _query_conjunction
    combined_query.update(copy.deepcopy(query))
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
TypeError: cannot deepcopy this pattern object

Update-1: The following code generates same exception

query = ( Q(groups__in=[g3,g2]) & Q(name=re.compile('eam3')))
copy.deepcopy(query)

Any suggestions ? Thanks

Answer 1

Because the deepcopy of compiled regex pattern is not supported in Python(since Python 2.5), i.e.,

copy.deepcopy(re.compile('eam3'))

is not supported. mongoengine will use copy.deepcopy when combine multiple query objects. So if you use a single regex filter, the query is OK with mongoengine, but not multiples.

mongoengine support a set of string queries, you may find them in the document.

Answer 2

Using mongoengine==0.22.1 I had a query like this:

regex = re.compile(".*v3\.[0-9]{1,4}/transaction/[0-9a-fA-F]*$")
from_date = kwargs.get("from_date", "2000-01-01")
to_date = kwargs.get("to_date", (datetime.utcnow() + timedelta(days=1)).strftime("%Y-%m-%d"))

qs = (
    ApiLog.objects.filter(
        time_stamp__gte=parse(from_date),
        time_stamp__lt=parse(to_date) + timedelta(days=1),
    )
    .filter(url=regex)
    .only("id", "url") 
    .order_by("time_stamp")
)

This query yielded the dreaded

  "TypeError: cannot deepcopy this pattern object" 

when I tried to paginate the results.

Upgrading from python 3.6.12 to 3.7.9 fixed the problem for me.

I am not sure why this error even occurred nor if it's origin is the same as what the OP shared but since this page came up on a google search for this error, I thought I'd post this here.