how to perform profiling for a website?

Question

I currently have a django site, and it's kind of slow, so I want to understand what's going on. How can I profile it so to differentiate between:

• effect of the network
• effect of the hosting I'm using
• effect of the javascript
• effect of the server side execution (python code) and sql access.
• any other effect I am not considering due to the massive headache I happen to have tonight.

Of course, for some of them I can use firebug, but some effects are correlated (e.g. javascript could appear slow because it's doing slow network access)

Thanks

Answer 1

client side:

• check with firebug if/which page components take long to load, and how long the browser needs to render the page after loading is completed. If everything is fast but rendering takes its time, then probably your html/css/js is the problem, otherwise it's server side.

server side (i assume you sit on some unix-alike server):

• check the web server with a small static content (a small gif or a little html page), using apache bench (ab, part of the apache webserver package) or httperf, the server should be able to answerat least 100 requests per second (of course this depends heavily on the size of your test content, webserver type, hardware and other stuff, so dont take that 100 to seriously). if that looks good,

• test django with ab or httperf on a "static view" (one that doesnt use a database object), if thats slow it's a hint that you need more cpu power. check cpu utilization on the server with top. if thats ok, the problem might be in the way the web server executes the python code

• if serving semi-static content is ok, your problem might be the database or IO-bound. Database problems are a wide field, here is some general advice:

  • check i/o throughput with iostat. if you see lot's of writes then you have get a better disc subsystem, faster raid, SSD hard drives .. or optimize your application to write less.
  • if its lots of reads, the host might not have enough ram dedicated as file system buffer, or your database queries might not be optimized
  • if i/o looks ok, then the database might be not be suited for your workload or not correctly configured. logging slow queries and monitoring database activity, locks etc might give you some idea

if you let us know what hardware/software you use i might be able to give more detailed advice

edit/PS: forgot one thing: of course your app might have a bad design and does lots of unnecessary/inefficient things ...

Answer 2

Take a look at the Django debug toolbar - that'll help you with the server side code (e.g. what database queries ran and how long they took); and is generally a great resource for Django development.

The other non-Django specific bits you could profile with yslow.