Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to download an entire directory and subdirectories using wget?

People also ask

How do I download a directory in Linux?

Download a directoryWget can download an entire directory, recursively, from either an FTP or web (HTTP/HTTPS) server. For FTP, just use the -r (recursive) option in your command and specify the directory you want to get.

What is wget recursive download?

This option tells Wget to delete every single file it downloads, after having done so. It is useful for pre-fetching popular pages through a proxy, e.g.: wget -r -nd --delete-after http://whatever.com/~popular/page/ The ' -r ' option is to retrieve recursively, and ' -nd ' to not create directories.

How do you wget multiple files?

If you want to download multiple files at once, use the -i option followed by the path to a local or external file containing a list of the URLs to be downloaded. Each URL needs to be on a separate line. If you specify - as a filename, URLs will be read from the standard input.


You may use this in shell:

wget -r --no-parent http://abc.tamu.edu/projects/tzivi/repository/revisions/2/raw/tzivi/

The Parameters are:

-r     //recursive Download

and

--no-parent // Don´t download something from the parent directory

If you don't want to download the entire content, you may use:

-l1 just download the directory (tzivi in your case)

-l2 download the directory and all level 1 subfolders ('tzivi/something' but not 'tivizi/somthing/foo')  

And so on. If you insert no -l option, wget will use -l 5 automatically.

If you insert a -l 0 you´ll download the whole Internet, because wget will follow every link it finds.


You can use this in a shell:

wget -r -nH --cut-dirs=7 --reject="index.html*" \
      http://abc.tamu.edu/projects/tzivi/repository/revisions/2/raw/tzivi/

The Parameters are:

-r recursively download

-nH (--no-host-directories) cuts out hostname 

--cut-dirs=X (cuts out X directories)

This link just gave me the best answer:

$ wget --no-clobber --convert-links --random-wait -r -p --level 1 -E -e robots=off -U mozilla http://base.site/dir/

Worked like a charm.


wget -r --no-parent URL --user=username --password=password

the last two options are optional if you have the username and password for downloading, otherwise no need to use them.

You can also see more options in the link https://www.howtogeek.com/281663/how-to-use-wget-the-ultimate-command-line-downloading-tool/


use the command

wget -m www.ilanni.com/nexus/content/

you can also use this command :

wget --mirror -pc --convert-links -P ./your-local-dir/ http://www.your-website.com

so that you get the exact mirror of the website you want to download


try this working code (30-08-2021):

!wget --no-clobber --convert-links --random-wait -r -p --level 1 -E -e robots=off --adjust-extension -U mozilla "yourweb directory with in quotations"