Downloading website files in powershell

Question

I'm trying to get a script to query files on an IIS website, then download those files automatically. So far, I have this:

$webclient = New-Object System.Net.webclient
$source = "http://testsite:8005/"
$destination = "C:\users\administrator\desktop\testfolder\"
#The following line returns the links in the webpage
$testcode1 = $webclient.downloadstring($source) -split "<a\s+" | %{ [void]($_ -match "^href=['"]([^'">\s]*)"); $matches[1] }
foreach ($line in $test2) {
    $webclient.downloadfile($source + $line, $destination + $line)
}

I'm not that good at PowerShell yet, and I get some errors, but I manage to get a couple test files I threw into my wwwroot folder (the web.config file seems undownloadable, so I'd imagine thats one of my errors). When I tried to change my $source value to a subfolder on my site that had some test text files(example = http://testsite:8005/subfolder/, I get errors and no downloads at all. Running my $testcode1 will give me the following links in my subfolder:
/subfolder/test2/txt
/
/subfolder/test1.txt
/subfolder/test2.txt
I don't know why it lists the test2 file twice. I figured my problem was that since it was returning the subfolder/file format, that I was getting errors because I was trying to download $source + $line, which would essentially be http://testsite:8005/subfolder/subfolder/test1.txt, but when I tried to remedy that by adding in a $root value that was the root directory of my site and do a foreach($line in $testcode1) { $webclient.downloadfile($root + $line, $destination + $line) }, I still get errors.
If some of you high speed gurus can help show me the error of my ways, I'd be grateful. I am looking to download all the files in each subfolder on my site, which I know would involve use of some recursive action, but again, I currently do not have the skill level myself to do that. Thank you in advance on helping me out!

Answer 1

I made a simple Powershell script to clone an openbsd package repo. It probably would work / could be implemented in other ways/use cases for similar things.

GitHub link

# Quick and dirty script to clone a package repo. Only tested against OpenBSD.
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$share = "\\172.16.10.99\wmfbshare\obsd_repo\"
$url = "https://ftp3.usa.openbsd.org/pub/OpenBSD/snapshots/packages/amd64/"
cd $share
$packages = Invoke-WebRequest -Uri $url -UseBasicParsing $url
$dlfolder = "\\172.16.10.99\wmfbshare\obsd_repo\"
foreach ($package in $packages.links.href){
    if ((get-item $package -ErrorAction SilentlyContinue)){
        write-host "$package already downloaded"
    } else {
        write-host "Downlading $package"
        wget "$url/$package" -outfile "$dlfolder\$package"
    }
}

Answer 2

This is to augment A_N's answer with two examples.

Download this Stackoverflow question to C:/temp/question.htm.

Invoke-RestMethod -Uri stackoverflow.com/q/19572091/1108891 -OutFile C:/temp/question.htm

Download a simple text document to C:/temp/rfc2616.txt.

Invoke-RestMethod -Uri tools.ietf.org/html/rfc2616 -OutFile C:/temp/rfc2616.txt

Answer 3

Best way to download files from a website is to use

Invoke-WebRequest –Uri $url

Once you are able to get hold of the html you can parse the content for the links.

$result = (((Invoke-WebRequest –Uri $url).Links | Where-Object {$_.href -like “http*”} ) | select href).href

Give it a try. Its simpler than $webclient = New-Object System.Net.webclient

Answer 4

I would try this:

$webclient = New-Object System.Net.webclient
$source = "http://testsite:8005/"
$destination = "C:\users\administrator\desktop\testfolder\"
#The following line returns the links in the webpage
$testcode1 = $webclient.downloadstring($source) -split "<a\s+" | %{ [void]($_ -match  "^href=['"]([^'">\s]*)"); $matches[1] }
foreach ($line in $testcode1) {
    $Destination = "$destination\$line"
    #Create a new directory if it doesn't exist
    if (!(Test-Path $Destination)){
        New-Item $Destination -type directory -Force
    }
    $webclient.downloadfile($source + $line, $destination + $line)
}

I think your only issue here is that you were grabbing a new file from a new directory, and putting it into a folder that didn't exist yet (I could be mistaken).

You can do some additional troubleshooting if that doesn't fix your problem:

Copy each line individually into your powershell window and run them up to the foreach loop. Then type out your variable holding all the gold:

    $testcode1

When you enter that into the console, it should spit out exactly what's in there. Then you can do additional troubleshooting like this:

    "Attempting to copy $Source$line to $Destination$line"

And see if it looks the way it should all the way on down. You might have to adjust my code a bit.

-Dale Harris