Linux wget command download file example

GNU Wget is a command-line program for downloading files from Web sites. Wget allows you to download files using HTTP, HTTPS and FTP protocols.

The wget command offers many options that allow you to download multiple files, resume downloads, limit speed, download recursively, download in the background, mirror websites, and more.

In this tutorial, we will show you how to use wget command to download files in Linux, and introduce the usage of wget with common examples of wget command.

Install wget

The wget command is now pre-installed on most Linux distributions, to check if the system has wget installed. Please press the shortcut key CTRL+ALT+T to open the terminal and run the command wget.

If wget is installed, the terminal prints the error message wget: missing URL, otherwise prints the message wget command not found.

If your computer is running a Debian-based Linux distribution. Such as Linux mint, Ubuntu. Please run the command to sudo apt install wget install wget.

If your computer is running a RedHat-based Linux distribution. For example CentOS, Fedora. Please run the command to sudo yum install wget install wget.

sudo apt install wget
sudo yum install wget

wget command

Before introducing how to use the wget command, let's review the basic syntax. The wget command syntax is wget [options] [url].

options Options for the wget command, optional arguments. url Remote server URL, also an optional parameter.

When running the wget command without specifying any parameters and options, the wget command downloads the file resource pointed to by the URL to the current directory.

During the download, wget displays a progress bar along with the filename, file size, download speed and estimated time to complete the download.

If you do not need the information output of these wget commands, you can use the -q option to turn it off. This option is often used in shell script programming.

After the download is complete, the downloaded file can be found in the current working directory.

If a file with the same name exists in the current directory, the wget command will append a number to the end of the file name and will not overwrite the existing file.

wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.17.2.tar.xz

Wget command specifies the name of the saved file

To save the file downloaded by wget under a different name, there are two ways to do this in wget, one is to use the -O option.

The other is to write the file downloaded by wget to the standard output , and then use the standard output redirection symbol > to write to the file.

Usually when using wget to download files, Linux system administrators will also specify the -c option to download files, which allows you to resume the download when the download is disconnected.

wget -cO myfreax.com https://www.myfreax.com/
wget -cO - https://www.myfreax.com/ > myfreax.com

Wget command specifies the save directory

To specify a different directory to save downloaded files, use the -P option of the wget command followed by the directory where you want to save the files.

The directory path can be relative or absolute. If the specified directory does not exist, the wget command will create it.

It is also worth mentioning that the -P option of the wget command can only specify the saved directory.

To specify the directory and name of the saved file at the same time, please use the -O option.

Usually when using wget to download files, Linux system administrators will also specify the -c option to download files, which allows you to resume the download when the download is disconnected.

wget -cP /home/myfreax/test https://www.myfreax.com/

Wget command to limit download speed

To limit the download speed, use the --limit-rate option of the wget command. The default unit of the option value is bytes, and you can also specify the units kilobytes k and megabytes m.

The --limit-rate option accepts not only integers, but also a floating point number, for example --limit-rate=2.5k is a legal value.

Note that wget limits network speed by sleeping for an appropriate amount of time, so that TCP transfers are slowed down to approximately the specified rate.

However when downloading files it doesn't really download at a very accurate speed as it takes some time to reach that balance, so don't be surprised if rate limiting doesn't work very well.

wget --limit-rate=1m url

Wget command resume download

If you lose your connection while downloading a large file, instead of restarting the download, you can use the wget command's -c option to resume the previous download.

You can also directly specify the -c option , and when the download of a large file is disconnected, you can resume the download by using the previous command directly.

It is also possible to create an alias to the wget command so that you do not have to repeatedly specify the -c option each time you download a file, which is a good practice.

But one thing worth noting is that the resuming download of the wget command requires the server to support resuming uploads, otherwise wget will start downloading files again.

wget -c url

Wget command background download

wget command -b options allow you to download files in the background. This is very useful when downloading large files. While downloading in the background.

The download status of the wget command, that is, the standard output of the wget command will be redirected to the wget-log file in the current directory.

To view the download progress of the wget command and other information, you can run the tail command to tail -f wget-log view the download status of the wget command.

wget -cb url
tail -f wget-log

Wget command to download FTP server files

In addition to supporting the HTTP protocol, the wget command also supports FTP protocol file downloads and lists files from FTP servers.

To download files from a password-protected FTP server, you need to specify an FTP username and password. Use the --ftp-password and --ftp-user options of the wget command .

wget --ftp-user=FTP_USERNAME --ftp-password=FTP_PASSWORD ftp://ftp.example.com/filename.tar.gz

Wget command to download multiple files

If you want to download multiple files at a time, there are two ways, one is to directly specify multiple URLs in a single command. The other is to use the -i option of the wget command to specify a file containing URLS.

To use the -i option of the wget command, specify after that the file containing the URL to download. Each URL in this file needs to be on its own line.

For example the command wget -i linux-distros.txt download  linux-distros.txt URL resource contained in the file.

wget URL1 URL2 
wget -i linux-distros.txt

http://mirrors.edge.kernel.org/archlinux/iso/2018.06.01/archlinux-2018.06.01-x86_64.iso
https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-9.4.0-amd64-netinst.iso
https://download.fedoraproject.org/pub/fedora/linux/releases/28/Server/x86_64/iso/Fedora-Server-dvd-x86_64-28-1.1.iso

Wget command downloads and decompresses with standard output

This is an advanced trick that saves time and allows you to unzip while downloading. Do whatever you want besides decompressing, as long as the command on the right side of the pipe allows reading from standard input .

To decompress while downloading, you need to use the wget command -q option to close the standard output of the download status, and then use the wget command -O option specify the standard output as a file.

When wget uses a hyphen - as the file, standard output is used as the file written, which is finally | piped to the tar command.

tar extracts the file downloaded by the wget command to the specified directory. The -C option specifies the target directory for writing, and the parameter - means to read data from standard input.

wget -q -O - "http://wordpress.org/latest.tar.gz" | tar -xzf - -C /var/www

Wget command specifies the user agent User-Agent

You should encounter the situation that when using different browsers, some browsers can download, or some browsers cannot download. Or provide different page content according to the visitor's device and browser.

This is because the server program distinguishes the user's device type according to the User-Agent, so as to determine whether to return different content or prevent your download and access.

To prevent the server from serving different content based on the visitor's User-Agent. Use the -U option of the wget command to specify the User-Agent user agent.

wget --user-agent="Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" http://wget-forbidden.com/

Conclusion

At this point, you should have a good understanding of the most common wget options, and be able to use wget to load multiple files, resume downloads, and combine wget options according to your needs.