ValueError: malformed string when using ast.literal_eval

Question

It is widely known that using eval() is a potential security risk so the use of ast.literal_eval(node_or_string) is promoted

However In python 2.7 it returns ValueError: malformed string when running this example:

>>> ast.literal_eval("4 + 9") 

Whereas in python 3.3 this example works as expected:

>>> ast.literal_eval('4+9') 13 

Why does it run on python 3 and not python 2? How can I fix it in python 2.7 without using the risky eval() function?

Answer 1

The reason this doesn’t work on Python 2 lies in its implementation of literal_eval. The original implementation only performed number evaluation for additions and subtractions when the righth operand was a complex number. This is syntactically necessary for complex numbers to be expressed as a literal.

This was changed in Python 3 so that it supports any kind of valid number expression to be on either side of the addition and subtraction. However, the use of literal_eval is still restricted to additions and subtractions.

This is mostly because literal_eval is supposed to be a function that turns a single constant literal (expressed as a string) into a Python object. Kind of like a backwards repr for simple built-in types. Actual expression evaluation is not included, and the fact that this works with Python 3 is just a nice-to-have side effect from its implementation.

In order to evaluate actual expressions, without having to use eval (which we don’t want to), we can write our own expression evaluation algorithm that operates on the AST. This is pretty simple, especially for simple arithmetic operations on numbers (for example to build your own calculator etc.). We simply parse the string into an AST and then evaluate the resulting tree by looking at the different node types and applying the correct operation.

Something like this:

import ast, operator  binOps = {     ast.Add: operator.add,     ast.Sub: operator.sub,     ast.Mult: operator.mul,     ast.Div: operator.div,     ast.Mod: operator.mod }  def arithmeticEval (s):     node = ast.parse(s, mode='eval')      def _eval(node):         if isinstance(node, ast.Expression):             return _eval(node.body)         elif isinstance(node, ast.Str):             return node.s         elif isinstance(node, ast.Num):             return node.n         elif isinstance(node, ast.BinOp):             return binOps[type(node.op)](_eval(node.left), _eval(node.right))         else:             raise Exception('Unsupported type {}'.format(node))      return _eval(node.body) 

As you can see, this implementation is pretty straightforward. Of course it does not support more complex stuff like exponentiation and some unary nodes yet, but it’s not too difficult to add that. And it works just fine:

>>> arithmeticEval('4+2') 6 >>> arithmeticEval('4*1+2*6/3') 8 

You could even introduce more complex things later (for example function calls for things like sin()).

Answer 2

It's in order to support complex numbers (since issue 4907). For example, 1 + 2j is parsed by the parser as an expression consisting of an integer literal, an addition operation and an imaginary literal; but since complex numbers are a built-in type, it is desirable for ast.literal_eval to support complex number syntax.

The change in behaviour between 2.x and 3.x is to support writing the complex number the "wrong way round" e.g. 1j + 2; the fact that it allows arbitrary addition or subtraction expressions is a (mostly unintended) side effect.

If you want to parse arbitrary arithmetic expressions, you should parse to a syntax tree (using ast.parse), verify it with a whitelist, and then evaluate.