Learn to download multiple files with wget by storing URLs in a text file and using the -i flag. Control bandwidth with --limit-rate and add delays between downloads with --wait to avoid overloading servers. Wget excels at bulk downloads while curl handles more protocols.
Chapters
So far, we've learned how to install and do basic file downloads using either Curl or wget. In this lesson, we'll focus on getting the most out of WGET by going over more advanced techniques for data downloading. A common way for data people to handle multiple file downloads is by storing the file locations in a file and passing that metafile to the downloading program like WGET. In this case, all the URLs for the files we want to download are stored in the file URL list txt. Let's use the cat command to print and preview the URLs really quickly. After confirming that the URLs are indeed stored in this file, we can now pass this file to wget. Note that we need to preface this with the I option flag so WGET knows that we are reading URLs from a local or external file. The command reads wget lowercase I URL list txt finally, it's worth noting not to insert any option flags in between the I and the URL file. If other option flags are needed, put it before I. Sometimes it's useful to make sure WGET does not consume your entire bandwidth with the file download. You can set an upper download bandwidth limit using the limit rate option. Set the limit rate equal to a whole number, which will automatically convert to kilobytes per second. For example, WGET limit rate equal to 200K will make sure your download rate will not exceed 200 kilobytes per second as you download the files saved in the URL list. For downloading smaller files, enforcing a download bandwidth won't work as well. To avoid overtaxing the file hosting server, it is more useful to enforce a mandatory wait time between file downloads. Using wait, the default time interval is set to seconds. For example, wget wait equals 2.5 creates a 2.5 second pause between downloading each file stored in a URL list file. As we round out this chapter, it is helpful to do a quick comparison between the two command line tools Curl and wget. Although both Curl and WGET can Download files from HTTP, HTTPs, and FTP, Curl alone can download and upload from 20 other protocols. It is also easier to install across all operating Systems compared to WGET. WGET's advantage is its ability to handle multiple file downloads gracefully. It can also be used to download just about anything from a full file directory to a HTML page.
