How would you attack a domain to look for "unknown" resources? [closed]

Question

Given a domain, is it possible for an attacker to discover one or many of the pages/resources that exist under that domain? And what could an attacker do/use to discover resources in a domain?

I have never seen the issue addressed in any security material (because it's a solved problem?) so I'm interested in ideas, theories, best-guesses, in addition to practices; anything an attacker could use in a "black box" manor to discover resources.

---

Some of the things that I've come up with are:

• Google -- if google can find it, an attacker can.
• A brute force dictionary attack -- Iterate common words and word combinations (Login, Error, Index, Default, etc.) As well, the dictionary could be narrowed if the resource extension was known (xml, asp, html, php.) which is fairly discoverable.
• Monitor traffic via a Sniffer -- Watch for a listing of pages that users go to. This assumes some type of network access, in which case URL discovery is likely small peanuts given the fact the attacker has network access.

Edit: Obviously directory listings permissions are turned off.

Answer 1

The list on this is pretty long; there are a lot of techniques that can be used to do this; note that some of these are highly illegal:

• See what Google, archive.org, and other web crawlers have indexed for the site.
• Crawl through public documents on the site (including PDF, JavaScript, and Word documents) looking for private links.
• Scan the site from different IP addresses to see if any location-based filtering is being done.
• Compromise a computer on the site owner's network and scan from there.
• Attack an exploit in the site's web server software and look at the data directly.
• Go dumpster diving for auth credentials and log into the website using a password on a post-it (this happens way more often than you might think).
• Look at common files (like robots.txt) to see if they 'protect' sensitive information.
• Try common URLs (/secret, /corp, etc.) to see if they give a 302 (unauthorized) or 404 (page not found).
• Get a low-level job at the company in question and attack from the inside; or, use that as an opportunity to steal credentials from legitimate users via keyboard sniffers, etc.
• Steal a salesperson's or executive's laptop -- many don't use filesystem encryption.
• Set up a coffee/hot dog stand offering a free WiFi hotspot near the company, proxy the traffic, and use that to get credentials.
• Look at the company's public wiki for passwords.

And so on... you're much better off attacking the human side of the security problem than trying to come in over the network, unless you find some obvious exploits right off the bat. Office workers are much less likely to report a vulnerability, and are often incredibly sloppy in their security habits -- passwords get put into wikis and written down on post-it notes stuck to the monitor, road warriors don't encrypt their laptop hard drives, and so on.

Answer 2

Most typical attack vector would be trying to find well known application, like for example /webstats/ or /phpMyAdmin/, look for some typical files that unexperienced user might left in production env (eg. phpinfo.php). And most dangerous: text editor backup files. Many text editors leave copy of original file with '~' appended or perpended. So imagine you have whatever.php~ or whatever.apsx~. As these are not executed, attacker might get access to source code.