Problems using Powershell/curl to check http headers

Question

This is turning out to be way more difficult than I thought (and than it probably should be).

I've tried Cygwin+curl, but that fails to run (Cygwin can never find curl.exe, but I did do a package check and it's there).

I've tried curl for Windows, using the command prompt, and it works. But I have quite a few urls to check and doing them individually just isn't time efficient. I can't figure out how to tell curl, through cmd prompt, to use a file (since there's no "xargs") like mentioned here.

I've also tried using PowerShell, but that's also being problematic. When I try to follow option 1 here

I get an error when I try running $xHTTP.open("GET",$url,$false):

Exception calling "open" with "3" argument(s): "Unspecified error (Exception from HRESULT: 0x80004005 (E_FAIL))"

Also with PowerShell, I have absolutely no idea how to get it to use a file containing urls. My knowledge of PS is very very limited (as in non-existent).

What would be nice here would be to figure out how to get the command prompt/curl to use a file, but I can't figure that out.

Answer 1

There is a more direct way to do this with PowerShell V3:

PS> Get-Content .\urls.txt
http://www.cnn.com
http://www.msn.com

PS> Get-Content urls.txt | Foreach { Invoke-WebRequest -Uri $_ -Method HEAD }

StatusCode        : 200
StatusDescription : OK
Content           :
RawContent        : HTTP/1.1 200 OK
                    Vary: Accept-Encoding
                    Connection: Keep-Alive
                    Cache-Control: max-age=60, private
                    Content-Type: text/html
                    Date: Tue, 08 Jan 2013 20:21:46 GMT
                    Expires: Tue, 08 Jan 2013 20:22:46 GMT...
Forms             : {}
Headers           : {[Vary, Accept-Encoding], [Connection, Keep-Alive], [Cache-Control, max-age=60, private],
                    [Content-Type, text/html]...}
...

To handle 404's use a try/catch e.g.:

PS> Get-Content urls.txt | 
        Foreach {try {Invoke-WebRequest -Uri $_ -Method HEAD} catch { "Oops - $_"}}

To redirect to a file, this works for me:

PS> Get-Content urls.txt | 
        Foreach {try {Invoke-WebRequest -Uri $_ -Method HEAD} catch { "Oops - $_"}} > 
        $home\Desktop\foo.txt