Is there a good tutorial for figuring out what a website is doing so your program can do the same thing? [closed]

Question

Is there a good guide or tutorial for people who need to programmatically interact with dynamic websites? There's been a rash of Perl questions about that lately, and I haven't found a good resource to point people toward. I'm asking not because I need one but because I don't want to waste my time writing it if it already exists. Although I'm most interested in Perl, the extra tools and techniques are mostly the same.

Typically, I see see these problems in people's questions:

• Handling, setting, and saving cookies
• Finding and interacting with forms
• Handling JavaScript inside your user-agent
  • especially things like onLoad, onSumbit, and Ajax
• Using HTTP sniffer tools
• Using Web developer plugins in interactive browsers
• Interacting with DOM, screen scraping, etc.

If there's no good tutorial, I'll add it to my list of things to do (unless someone else wants to do it). Along the way, if you don't have a suggestion for an existing tutorial, please suggest the things that you think should be in a new one, including links, your favorite tools, and your own user-agent development experiences. I don't care about the particular language you use.

Answer 1

The best I've seen is a Defcon presentation video.