Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Speeding Up Python

This is really two questions, but they are so similar, and to keep it simple, I figured I'd just roll them together:

  • Firstly: Given an established python project, what are some decent ways to speed it up beyond just plain in-code optimization?

  • Secondly: When writing a program from scratch in python, what are some good ways to greatly improve performance?

For the first question, imagine you are handed a decently written project and you need to improve performance, but you can't seem to get much of a gain through refactoring/optimization. What would you do to speed it up in this case short of rewriting it in something like C?

like image 338
akdom Avatar asked Oct 05 '08 21:10

akdom


People also ask

Is there a way to speed up Python?

Use the Built-In Functions Many of Python's built-in functions are written in C, which makes them much faster than a pure python solution. Take a very simple task of summing a lot of numbers. We could loop through each number, summing as we go.

How do you speed up a loop in Python?

A faster way to loop using built-in functions A faster way to loop in Python is using built-in functions. In our example, we could replace the for loop with the sum function. This function will sum the values inside the range of numbers. The code above takes 0.84 seconds.

Why is my Python code so slow?

In summary: code is slowed down by the compilation and interpretation that occurs during runtime. Compare this to a statically typed, compiled language which runs just the CPU instructions once compilated. It's actually possible to extend Python with compiled modules that are written in C.


2 Answers

Regarding "Secondly: When writing a program from scratch in python, what are some good ways to greatly improve performance?"

Remember the Jackson rules of optimization:

  • Rule 1: Don't do it.
  • Rule 2 (for experts only): Don't do it yet.

And the Knuth rule:

  • "Premature optimization is the root of all evil."

The more useful rules are in the General Rules for Optimization.

  1. Don't optimize as you go. First get it right. Then get it fast. Optimizing a wrong program is still wrong.

  2. Remember the 80/20 rule.

  3. Always run "before" and "after" benchmarks. Otherwise, you won't know if you've found the 80%.

  4. Use the right algorithms and data structures. This rule should be first. Nothing matters as much as algorithm and data structure.

Bottom Line

You can't prevent or avoid the "optimize this program" effort. It's part of the job. You have to plan for it and do it carefully, just like the design, code and test activities.

like image 66
S.Lott Avatar answered Sep 23 '22 08:09

S.Lott


Rather than just punting to C, I'd suggest:

Make your code count. Do more with fewer executions of lines:

  • Change the algorithm to a faster one. It doesn't need to be fancy to be faster in many cases.
  • Use python primitives that happens to be written in C. Some things will force an interpreter dispatch where some wont. The latter is preferable
  • Beware of code that first constructs a big data structure followed by its consumation. Think the difference between range and xrange. In general it is often worth thinking about memory usage of the program. Using generators can sometimes bring O(n) memory use down to O(1).
  • Python is generally non-optimizing. Hoist invariant code out of loops, eliminate common subexpressions where possible in tight loops.
  • If something is expensive, then precompute or memoize it. Regular expressions can be compiled for instance.
  • Need to crunch numbers? You might want to check numpy out.
  • Many python programs are slow because they are bound by disk I/O or database access. Make sure you have something worthwhile to do while you wait on the data to arrive rather than just blocking. A weapon could be something like the Twisted framework.
  • Note that many crucial data-processing libraries have C-versions, be it XML, JSON or whatnot. They are often considerably faster than the Python interpreter.

If all of the above fails for profiled and measured code, then begin thinking about the C-rewrite path.

like image 24
I GIVE CRAP ANSWERS Avatar answered Sep 24 '22 08:09

I GIVE CRAP ANSWERS