Using Proxy with cURL

Using a proxy with cURL involves specifying the proxy server's address and port using the -x or --proxy command-line option, enabling routing of cURL requests through an intermediary server.

This article details the various methods and command-line options available in cURL for interacting with proxy services, covering basic HTTP/HTTPS proxies, authenticated proxies, SOCKS proxies, and configuration via environment variables.

Basic Proxy Configuration

The most common method to instruct cURL to use a proxy is with the -x or --proxy option, followed by the proxy server's address and port.

HTTP/HTTPS Proxy

To route an HTTP or HTTPS request through an HTTP/HTTPS proxy, specify the proxy address and port.

Syntax:

curl -x [protocol://][user:password@]proxyhost[:port] [URL]

Examples:

1. Using an HTTP proxy for an HTTP destination:
   bash curl -x http://your.proxy.server:8080 http://example.com
   This command directs cURL to send the request for http://example.com through your.proxy.server on port 8080 using the HTTP proxy protocol.

2. Using an HTTP proxy for an HTTPS destination:
   bash curl -x http://your.proxy.server:8080 https://secure.example.com
   When proxying an HTTPS destination through an HTTP proxy, cURL typically uses the CONNECT method to establish a tunnel through the proxy to the destination server. The SSL/TLS handshake then occurs directly between cURL and secure.example.com, not involving the HTTP proxy in the encryption.

3. Using an HTTPS proxy for an HTTP destination:
   bash curl -x https://your.secure.proxy.server:8443 http://example.com
   This routes the request through a proxy that itself uses HTTPS. The connection between cURL and your.secure.proxy.server is encrypted.

Authenticated Proxies

Many proxy services require authentication. cURL supports various authentication methods for proxies, primarily Basic, NTLM, and Digest.

Basic Authentication

For proxies requiring a username and password, use the --proxy-user option or embed credentials directly in the proxy URL.

Syntax:

curl -x [protocol://]proxyhost[:port] --proxy-user "username:password" [URL]
curl -x [protocol://]username:password@proxyhost[:port] [URL]

Examples:

1. Using --proxy-user:
   bash curl -x http://your.proxy.server:8080 --proxy-user "proxyuser:proxypass" http://example.com

2. Embedding credentials in the URL:
   bash curl -x http://proxyuser:proxypass@your.proxy.server:8080 http://example.com
   URL-encoding special characters in the username or password is required if they are present. For example, a password p@ssword should be p%40ssword.

Other Authentication Methods

While Basic authentication is common, cURL supports other methods for specific environments.

• --proxy-ntlm: Enables NTLM authentication for the proxy.
• --proxy-digest: Enables Digest authentication for the proxy.

Example (NTLM):

curl -x http://your.proxy.server:8080 --proxy-ntlm --proxy-user "DOMAIN\proxyuser:proxypass" http://example.com

SOCKS Proxies

SOCKS (Socket Secure) proxies operate at a lower level than HTTP proxies, handling various network protocols. cURL supports SOCKS4, SOCKS4a, and SOCKS5.

SOCKS5 Proxy

SOCKS5 is the most versatile SOCKS protocol, supporting IPv4, IPv6, UDP, and authentication.

Syntax:

curl --socks5 [user:password@]proxyhost[:port] [URL]

Examples:

1. Basic SOCKS5 proxy:
   bash curl --socks5 your.socks5.server:1080 http://example.com
   By default, cURL resolves the hostname example.com locally before sending the IP address to the SOCKS5 proxy.

2. SOCKS5 proxy with remote DNS resolution:
   bash curl --socks5-hostname your.socks5.server:1080 http://example.com
   The --socks5-hostname option instructs cURL to send the hostname example.com to the SOCKS5 proxy, allowing the proxy to perform the DNS resolution. This is crucial for anonymity and bypassing geo-restrictions where local DNS resolution might reveal your location.

3. SOCKS5 proxy with authentication:
   bash curl --socks5-hostname socksuser:sockspass@your.socks5.server:1080 http://example.com

SOCKS4/SOCKS4a Proxy

SOCKS4 is an older protocol, primarily for IPv4, and does not support authentication. SOCKS4a extends SOCKS4 to allow the proxy to resolve hostnames.

Syntax:

curl --socks4 proxyhost[:port] [URL]
curl --socks4a proxyhost[:port] [URL]

Example (SOCKS4a):

curl --socks4a your.socks4.server:1080 http://example.com

Similar to --socks5-hostname, --socks4a ensures the proxy resolves the destination hostname.

Proxying HTTPS Destinations with SSL/TLS

When cURL proxies an HTTPS request through an HTTP proxy, it initiates a CONNECT request to the proxy. The proxy then establishes a TCP tunnel to the destination HTTPS server. cURL performs the SSL/TLS handshake directly with the destination server through this tunnel.

SSL Certificate Verification

By default, cURL verifies the SSL certificate of the destination server. If the certificate is invalid, expired, or self-signed, cURL will abort the connection.

• --insecure or -k: Disables SSL certificate verification. Use with caution, primarily for testing or known trusted environments with self-signed certificates.
  bash curl -x http://your.proxy.server:8080 https://self-signed.example.com -k

• --cacert [file]: Specifies a custom CA certificate bundle for verification.
  bash curl -x http://your.proxy.server:8080 https://secure.example.com --cacert /path/to/custom_ca.pem

Environment Variables for Proxy Configuration

cURL can also pick up proxy settings from environment variables. This is useful for system-wide or session-wide proxy configurations without specifying the -x option for every command.

Common Environment Variables

• http_proxy: Proxy for HTTP requests.
• https_proxy: Proxy for HTTPS requests.
• all_proxy: A fallback proxy for all protocols if http_proxy or https_proxy are not set.
• no_proxy: A comma-separated list of hostnames, domains, or IP addresses that should bypass the proxy.

Syntax:

export http_proxy="http://your.proxy.server:8080"
export https_proxy="http://your.proxy.server:8080" # Note: often an HTTP proxy is used for HTTPS destinations
export no_proxy="localhost,127.0.0.1,.localdomain"

Examples:

1. Setting HTTP proxy:
   bash export http_proxy="http://proxyuser:proxypass@your.proxy.server:8080" curl http://example.com

2. Setting HTTPS proxy (via HTTP proxy):
   bash export https_proxy="http://your.proxy.server:8080" curl https://secure.example.com

3. Bypassing proxy for specific hosts:
   bash export http_proxy="http://your.proxy.server:8080" export no_proxy="internal.example.com" curl http://internal.example.com # This request will bypass the proxy curl http://external.example.com # This request will use the proxy

Precedence

Command-line options take precedence over environment variables. If -x is specified, it overrides any http_proxy, https_proxy, or all_proxy environment variables.

Troubleshooting Proxy Connections

When encountering issues with cURL and proxies, several options can assist in diagnosis.

• --verbose or -v: Provides detailed information about the request and response, including proxy connection attempts, headers, and SSL/TLS negotiation. This is often the first step in debugging.
  bash curl -v -x http://your.proxy.server:8080 http://example.com

• --trace [file]: Dumps a full trace of all incoming and outgoing data, including headers and body, to a specified file.
  bash curl --trace debug.log -x http://your.proxy.server:8080 http://example.com

• Proxy Server Logs: Check the logs of your proxy server for connection attempts, authentication failures, or error messages related to your cURL requests.

• Network Connectivity: Verify that the machine running cURL can reach the proxy server on the specified port using tools like ping, telnet, or netcat.
  bash telnet your.proxy.server 8080
  A successful telnet connection indicates basic network reachability.

Comparison of Key Proxy Options

cURL Option	Description	Protocols	Authentication	DNS Resolution
-x, --proxy	Generic proxy specification (HTTP/HTTPS proxy)	HTTP, HTTPS	Yes	Local
--proxy-user	Specifies username and password for proxy authentication	HTTP, HTTPS	Yes	N/A
--socks5	SOCKS5 proxy, local DNS resolution by default	All	Yes	Local
--socks5-hostname	SOCKS5 proxy, remote DNS resolution	All	Yes	Remote (proxy)
--socks4	SOCKS4 proxy, local DNS resolution	All	No	Local
--socks4a	SOCKS4a proxy, remote DNS resolution	All	No	Remote (proxy)
http_proxy (env)	Environment variable for HTTP proxy	HTTP	Yes	Local
https_proxy (env)	Environment variable for HTTPS proxy (often an HTTP proxy for tunneling)	HTTPS	Yes	Local
all_proxy (env)	Environment variable for all protocols if others not set	All	Yes	Local
no_proxy (env)	Environment variable to specify hosts that should bypass the proxy	N/A	N/A	N/A