Skip to content

Package: wget

Overview

wget is GNU package

Documentation: Wget 1.24.5

Installment

On ubuntu::latest
apt-get update -y && apt install wget

You can use with docker for the development/trial with the package

docker run -it ubuntu

Usage

wget --version
# GNU Wget 1.21.2 built on linux-gnu.

# -cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls
# +ntlm +opie +psl +ssl/openssl

# Wgetrc:
#     /etc/wgetrc (system)
# Locale:
#     /usr/share/locale
# Compile:
#     gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc"
#     -DLOCALEDIR="/usr/share/locale" -I. -I../../src -I../lib
#     -I../../lib -Wdate-time -D_FORTIFY_SOURCE=2 -DHAVE_LIBSSL -DNDEBUG
#     -g -O2 -ffile-prefix-map=/build/wget-8g5eYO/wget-1.21.2=.
#     -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects
#     -fstack-protector-strong -Wformat -Werror=format-security
#     -DNO_SSLv2 -D_FILE_OFFSET_BITS=64 -g -Wall
# Link:
#     gcc -DHAVE_LIBSSL -DNDEBUG -g -O2
#     -ffile-prefix-map=/build/wget-8g5eYO/wget-1.21.2=. -flto=auto
#     -ffat-lto-objects -flto=auto -ffat-lto-objects
#     -fstack-protector-strong -Wformat -Werror=format-security
#     -DNO_SSLv2 -D_FILE_OFFSET_BITS=64 -g -Wall -Wl,-Bsymbolic-functions
#     -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now
#     -lpcre2-8 -luuid -lidn2 -lssl -lcrypto -lz -lpsl ftp-opie.o
#     openssl.o http-ntlm.o ../lib/libgnu.a

# Copyright (C) 2015 Free Software Foundation, Inc.
# License GPLv3+: GNU GPL version 3 or later
# <http://www.gnu.org/licenses/gpl.html>.
# This is free software: you are free to change and redistribute it.
# There is NO WARRANTY, to the extent permitted by law.

# Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
# Please send bug reports and questions to <bug-wget@gnu.org>.

wget Command-Line Utility

Overview

wget is a non-interactive command-line utility used to download files from the web. It supports HTTP, HTTPS, and FTP protocols and is designed for reliability in unstable network conditions. wget is commonly used in automation, scripting, and server environments where downloads must continue without user interaction.

The tool emphasizes robustness, resumable downloads, and recursive retrieval of web resources.

Scope and Applicability

Aspect Description
Scope File and resource retrieval over networks
Applicability General-purpose, multi-platform
Operating Systems Linux, macOS, Windows
Interaction Model Non-interactive (CLI-based)
Common Usage Automation, CI/CD, servers, data ingestion

wget is not intended for interactive browsing or rendering web content.

Configuration and Definition

wget behavior is defined through command-line options and optional configuration files.

Command Structure

wget [OPTIONS] <URL>

Global Configuration File

File Purpose
/etc/wgetrc System-wide configuration
~/.wgetrc User-specific configuration

Settings in configuration files define defaults such as retry limits, timeout values, and proxy behavior.

Detail | SAD | Component Breakdown

Core Components

Component Description
URL parser Resolves and validates target URLs
Network engine Manages HTTP/HTTPS/FTP connections
Download manager Handles retries, resuming, and progress
File writer Persists downloaded content to disk
Recursive crawler Traverses links when recursive mode is enabled

Common Options

Option Description
-O <file> Write output to a specific file
-c Resume partially downloaded files
-r Enable recursive download
-l <depth> Limit recursion depth
--mirror Enable mirroring mode
--limit-rate Throttle download speed

Concise Examples

Download a single file:

wget https://example.com/file.zip

Resume a failed download:

wget -c https://example.com/file.zip

Mirror a website:

wget --mirror https://example.com

Application and Execution

wget executes downloads following a deterministic procedural flow.

Execution Algorithm

Step 1: Parse command-line arguments and configuration files Step 2: Resolve target URL and protocol Step 3: Establish network connection Step 4: Request resource from remote server Step 5:

  • If partial file exists and -c is set, resume download
  • Otherwise, start a new download Step 6: Write data to disk incrementally Step 7:
  • On network failure, retry according to retry policy
  • On success, finalize file and exit

Termination occurs when the download completes or retry limits are exceeded.

Edge Cases and Limitations

Note

Some modern websites block automated download tools, which may cause wget requests to fail.

Limitation Description
JavaScript execution Not supported
Dynamic content Cannot render client-side apps
Authentication Limited support for complex auth flows
Rate limiting May be blocked by servers

Recursive downloads may unintentionally retrieve large amounts of data if depth limits are not set.

Reference