Proxy Setup for wget

To configure wget to download through a proxy, set the http_proxy, https_proxy, or ftp_proxy environment variables, specify proxy details via command-line options, or define them in a .wgetrc configuration file.

Proxy configuration for wget is necessary when direct internet access is restricted by firewalls, for routing traffic through a specific network, or for anonymization purposes. wget supports HTTP, HTTPS, and FTP protocols for both target resources and proxy connections.

Proxy Configuration Methods

There are three primary methods for configuring wget to use a proxy: environment variables, command-line options, and the .wgetrc configuration file. Each method offers different levels of scope and persistence.

Environment Variables

Environment variables provide a system-wide or session-specific proxy configuration that wget automatically detects. This is often the most common method for temporary or user-specific settings.

• http_proxy: Used for HTTP targets.
• https_proxy: Used for HTTPS targets.
• ftp_proxy: Used for FTP targets.
• no_proxy: Specifies a comma-separated list of domains or IP addresses that should bypass the proxy.

The general format for proxy URLs is http://[user:password@]host:port/. wget primarily uses HTTP proxies to fetch content, regardless of whether the target URL is HTTP or HTTPS. Therefore, http_proxy is often sufficient for both HTTP and HTTPS targets, unless a specific HTTPS proxy is required.

Temporary Configuration (Current Session)

Set these variables in your shell before invoking wget.

Linux/macOS (Bash/Zsh):

export http_proxy="http://proxy.example.com:8080"
export https_proxy="http://proxy.example.com:8080"
export ftp_proxy="http://proxy.example.com:8080"
export no_proxy="localhost,127.0.0.1,.internal.domain.com"

wget http://example.com/file.zip
wget https://secure.example.com/data.tar.gz

For proxies requiring authentication:

export http_proxy="http://user:password@proxy.example.com:8080"
export https_proxy="http://user:password@proxy.example.com:8080"

wget http://example.com/file.zip

Note: Embedding credentials directly in environment variables or URLs is generally not recommended for security-sensitive environments, as they may be visible to other processes or stored in shell history.

To disable proxy for a specific session:

unset http_proxy
unset https_proxy
unset ftp_proxy

Windows (CMD):

set http_proxy=http://proxy.example.com:8080
set https_proxy=http://proxy.example.com:8080
set ftp_proxy=http://proxy.example.com:8080
set no_proxy=localhost,127.0.0.1,.internal.domain.com

wget http://example.com/file.zip

Windows (PowerShell):

$env:http_proxy="http://proxy.example.com:8080"
$env:https_proxy="http://proxy.example.com:8080"
$env:ftp_proxy="http://proxy.example.com:8080"
$env:no_proxy="localhost,127.0.0.1,.internal.domain.com"

wget http://example.com/file.zip

Permanent Configuration (User-specific)

To make proxy settings persistent across sessions for a specific user, add the export commands to your shell's profile file (e.g., ~/.bashrc, ~/.zshrc, ~/.profile on Linux/macOS) or system-wide configuration files.

Example for ~/.bashrc:

# Proxy Settings
export http_proxy="http://proxy.example.com:8080"
export https_proxy="http://proxy.example.com:8080"
export ftp_proxy="http://proxy.example.com:8080"
export no_proxy="localhost,127.0.0.1,.internal.domain.com"

After editing, source the file or restart your shell: source ~/.bashrc.

Command-line Options

wget provides specific command-line options for proxy configuration, overriding environment variables and .wgetrc settings for a single invocation. This is suitable for one-off downloads or scripting where proxy settings need to be dynamically controlled.

• --proxy-on: Explicitly enable proxy usage.
• --proxy-off: Explicitly disable proxy usage.
• --no-proxy: Comma-separated list of domains that should bypass the proxy (similar to no_proxy environment variable).
• --proxy-user=<user>: Specify proxy username.
• --proxy-password=<password>: Specify proxy password.

Example Usage:

# Download a file through a proxy, overriding any default settings
wget --proxy-on --proxy-user=myuser --proxy-password=mypassword http://proxy.example.com:8080 http://example.com/file.zip

# Download a file directly, bypassing any configured proxy
wget --proxy-off http://example.com/file.zip

# Use proxy but bypass for a specific domain
wget --no-proxy=internal.example.com --proxy-on http://proxy.example.com:8080 http://example.com/file.zip

When using --proxy-user and --proxy-password, wget will attempt to authenticate with the proxy using the provided credentials. The proxy address itself is typically still defined via environment variables or .wgetrc.

.wgetrc Configuration File

The .wgetrc file allows for persistent, user-specific, or system-wide proxy settings. wget reads this file upon execution.

• User-specific: ~/.wgetrc (Linux/macOS), or a path specified by the WGETRC environment variable.
• System-wide: /etc/wgetrc (Linux/macOS).

Settings in ~/.wgetrc take precedence over /etc/wgetrc. Command-line options override both.

Configuration Directives:

# Enable or disable proxy usage by default
use_proxy = on

# Define HTTP proxy
http_proxy = http://proxy.example.com:8080/

# Define HTTPS proxy
https_proxy = http://proxy.example.com:8080/

# Define FTP proxy
ftp_proxy = http://proxy.example.com:8080/

# Proxy authentication credentials
proxy_user = myuser
proxy_password = mypassword

# Domains to bypass proxy
no_proxy = localhost,127.0.0.1,.internal.domain.com

Example ~/.wgetrc content:

# Enable proxy usage
use_proxy = on

# Proxy server details
http_proxy = http://user:password@proxy.example.com:8080/
https_proxy = http://user:password@proxy.example.com:8080/
ftp_proxy = http://user:password@proxy.example.com:8080/

# Domains to exclude from proxy
no_proxy = localhost,127.0.0.1,dev.local

Proxy Authentication

wget supports basic and digest proxy authentication.

• Via URL: As shown in environment variable examples (http://user:password@host:port). This is less secure as credentials can appear in logs or shell history.
• Via proxy_user and proxy_password: In .wgetrc or through command-line options (--proxy-user, --proxy-password). This is generally preferred for security, as wget handles the authentication handshake without exposing credentials in the clear within the URL.

Comparison of Configuration Methods

Feature	Environment Variables	Command-line Options	.wgetrc File
Scope	Session or User-specific (via profile files)	Single wget command invocation	User-specific (~/.wgetrc) or System-wide (/etc/wgetrc)
Persistence	Temporary (session) or Permanent (profile file)	None (ephemeral)	Permanent
Precedence	Overridden by command-line options	Highest (overrides all other methods)	Overridden by command-line options and environment variables
Use Case	General-purpose, frequent use, scripting	One-off downloads, dynamic proxy changes, testing	Default settings for a user or system
Authentication	Via URL (less secure)	Dedicated options (--proxy-user, --proxy-password)	Dedicated directives (proxy_user, proxy_password)

Troubleshooting Common Issues

• "Proxy authentication required": Ensure proxy_user and proxy_password are correctly set, or credentials are provided in the proxy URL. Verify the username and password are correct.
• "Connection refused" / "Unable to connect to proxy":
  • Verify the proxy host and port are correct and the proxy service is running.
  • Check network connectivity from the client to the proxy server.
  • Firewalls (client or proxy side) might be blocking the connection.
• "Bad Gateway" / "HTTP Error 502": The proxy server could not connect to the upstream server or encountered an error processing the request. This typically indicates an issue with the proxy server itself or its connection to the internet.
• wget not using proxy:
  • Verify use_proxy = on in .wgetrc or that environment variables are correctly set and exported.
  • Ensure no conflicting --proxy-off or no_proxy settings are active for the target URL.
  • Check for typos in proxy addresses.
• no_proxy not working: Ensure the no_proxy list is correctly formatted (comma-separated, no spaces around commas) and includes the exact domains or IP addresses. wget performs substring matching, so .example.com will match host.example.com.
• HTTPS targets failing through HTTP proxy: While wget can often tunnel HTTPS through an HTTP proxy, some proxies or network configurations may require a dedicated HTTPS proxy setting (using https_proxy) or direct connection for HTTPS. If https_proxy is not set, wget will default to http_proxy. Ensure your proxy server supports SSL/TLS tunneling.