Setting up proxies for Java applications using HttpClient, Jsoup, and Selenium involves configuring proxy host and port either globally via Java system properties or specifically for each library's connection objects, often including provisions for authentication.
Proxy Fundamentals in Java
Proxy servers act as intermediaries for network requests, offering benefits such as enhanced security, anonymity, and access to geo-restricted content. In Java, proxy configurations can be applied broadly across the Java Virtual Machine (JVM) or specifically to individual HTTP clients.
Proxy Types
- HTTP Proxy: Handles HTTP requests (
http://). - HTTPS Proxy: Handles HTTPS requests (
https://). Often, an HTTP proxy can also handle HTTPS requests via theCONNECTmethod. - SOCKS Proxy: Supports various network protocols, not just HTTP/HTTPS. Offers more flexibility and can handle TCP and UDP traffic.
Proxy Authentication
Many proxies require authentication (username/password). This can be handled differently depending on whether the proxy is configured globally or per-client.
Global Proxy Setup (Java System Properties)
Java provides system properties to configure proxies that apply to all java.net connections, including those made by HttpURLConnection (which Jsoup uses internally) and other libraries that default to these system settings.
// Set HTTP proxy
System.setProperty("http.proxyHost", "proxy.example.com");
System.setProperty("http.proxyPort", "8080");
// Set HTTPS proxy (can be the same as HTTP or different)
System.setProperty("https.proxyHost", "proxy.example.com");
System.setProperty("https.proxyPort", "8080");
// Set SOCKS proxy
System.setProperty("socksProxyHost", "socks.example.com");
System.setProperty("socksProxyPort", "1080");
// Exclude hosts from proxying (comma-separated list, wildcards allowed)
System.setProperty("http.nonProxyHosts", "localhost|127.0.0.1|*.local");
These properties can also be passed as JVM arguments:
java -Dhttp.proxyHost=proxy.example.com -Dhttp.proxyPort=8080 YourApplication
Global Proxy Authentication
For proxies requiring authentication, Java's Authenticator class can be used. This sets up a default authenticator for all java.net connections.
import java.net.Authenticator;
import java.net.PasswordAuthentication;
public class GlobalProxyAuthenticator {
public static void setup(String username, String password) {
Authenticator.setDefault(new Authenticator() {
@Override
protected PasswordAuthentication getPasswordAuthentication() {
// Check if the request is for a proxy
if (getRequestorType() == RequestorType.PROXY) {
return new PasswordAuthentication(username, password.toCharArray());
}
return null; // No authentication for other requests
}
});
}
public static void main(String[] args) {
// Set global proxy properties
System.setProperty("http.proxyHost", "authenticated-proxy.example.com");
System.setProperty("http.proxyPort", "8080");
System.setProperty("https.proxyHost", "authenticated-proxy.example.com");
System.setProperty("https.proxyPort", "8080");
// Set up authenticator
GlobalProxyAuthenticator.setup("proxyuser", "proxypass");
// Now, any HttpURLConnection-based request will attempt to use this proxy and credentials
// Example: Jsoup will implicitly use this.
try {
org.jsoup.Connection.Response response = org.jsoup.Jsoup.connect("http://httpbin.org/get")
.ignoreContentType(true)
.execute();
System.out.println("Status: " + response.statusCode());
System.out.println("Body: " + response.body());
} catch (java.io.IOException e) {
e.printStackTrace();
}
}
}
Apache HttpClient
Apache HttpClient provides robust, flexible methods for making HTTP requests and is widely used for complex integrations. It supports explicit proxy configuration per client instance.
HttpClient 4.x
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
public class HttpClient4ProxySetup {
public static void main(String[] args) throws Exception {
// 1. Define Proxy Host
HttpHost proxy = new HttpHost("proxy.example.com", 8080, "http"); // or "https" or "socks"
// 2. Configure Request with Proxy
RequestConfig config = RequestConfig.custom()
.setProxy(proxy)
.build();
// 3. (Optional) For Authenticated Proxy: Credentials Provider
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(
new AuthScope(proxy.getHostName(), proxy.getPort()),
new UsernamePasswordCredentials("proxyuser", "proxypass"));
// 4. Build HttpClient
CloseableHttpClient httpClient = HttpClients.custom()
.setDefaultRequestConfig(config)
.setDefaultCredentialsProvider(credentialsProvider) // For authenticated proxies
.build();
// 5. Execute Request
HttpGet request = new HttpGet("http://httpbin.org/get");
request.setConfig(config); // Ensure request uses the config
try (CloseableHttpResponse response = httpClient.execute(request)) {
System.out.println("Status: " + response.getStatusLine());
System.out.println("Body: " + EntityUtils.toString(response.getEntity()));
} finally {
httpClient.close();
}
}
}
HttpClient 5.x
import org.apache.hc.client5.http.auth.AuthScope;
import org.apache.hc.client5.http.auth.UsernamePasswordCredentials;
import org.apache.hc.client5.http.impl.auth.BasicCredentialsProvider;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.client5.http.classic.methods.HttpGet;
import org.apache.hc.client5.http.config.RequestConfig;
import org.apache.hc.core5.http.HttpHost;
import org.apache.hc.core5.http.io.entity.EntityUtils;
import org.apache.hc.core5.http.ClassicHttpResponse;
public class HttpClient5ProxySetup {
public static void main(String[] args) throws Exception {
// 1. Define Proxy Host
HttpHost proxy = new HttpHost("http", "proxy.example.com", 8080); // Scheme, Host, Port
// 2. (Optional) For Authenticated Proxy: Credentials Provider
BasicCredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(
new AuthScope(proxy.getHostName(), proxy.getPort()),
new UsernamePasswordCredentials("proxyuser", "proxypass".toCharArray()));
// 3. Build HttpClient
CloseableHttpClient httpClient = HttpClients.custom()
.setProxy(proxy) // Set proxy directly on the client builder
.setDefaultCredentialsProvider(credentialsProvider) // For authenticated proxies
.build();
// 4. Execute Request
HttpGet request = new HttpGet("http://httpbin.org/get");
// RequestConfig can still be used for per-request overrides if needed
RequestConfig requestConfig = RequestConfig.custom()
// .setProxy(proxy) // Can also be set here for per-request proxy
.build();
request.setConfig(requestConfig);
try (CloseableHttpClient client = httpClient) {
client.execute(request, response -> {
System.out.println("Status: " + response.getCode());
System.out.println("Body: " + EntityUtils.toString(response.getEntity()));
return null;
});
}
}
}
Jsoup
Jsoup is a Java library for working with real-world HTML. It provides a simple API for fetching URLs and parsing HTML. Jsoup's Connection interface allows direct proxy configuration.
import org.jsoup.Jsoup;
import org.jsoup.Connection;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Proxy;
public class JsoupProxySetup {
public static void main(String[] args) {
String url = "http://httpbin.org/get";
String proxyHost = "proxy.example.com";
int proxyPort = 8080;
try {
// Method 1: Using proxy(host, port)
Connection.Response response1 = Jsoup.connect(url)
.proxy(proxyHost, proxyPort)
.ignoreContentType(true)
.execute();
System.out.println("Jsoup (Method 1) Status: " + response1.statusCode());
System.out.println("Jsoup (Method 1) Body: " + response1.body().substring(0, 100) + "...\n");
// Method 2: Using java.net.Proxy object
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort));
Connection.Response response2 = Jsoup.connect(url)
.proxy(proxy)
.ignoreContentType(true)
.execute();
System.out.println("Jsoup (Method 2) Status: " + response2.statusCode());
System.out.println("Jsoup (Method 2) Body: " + response2.body().substring(0, 100) + "...\n");
} catch (IOException e) {
System.err.println("Error connecting via Jsoup proxy: " + e.getMessage());
e.printStackTrace();
}
// For authenticated proxies, Jsoup relies on global Java system properties and Authenticator.
// See "Global Proxy Setup" section for details.
}
}
For authenticated proxies with Jsoup, the recommended approach is to configure the global Java system properties and java.net.Authenticator as demonstrated in the "Global Proxy Setup" section. Jsoup's internal HttpURLConnection will then pick up these settings.
Selenium WebDriver
Selenium automates web browsers. Proxy configuration for Selenium WebDriver is typically done via browser-specific options.
Chrome WebDriver
import org.openqa.selenium.Proxy;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
public class SeleniumChromeProxySetup {
public static void main(String[] args) {
// Set path to ChromeDriver executable
// System.setProperty("webdriver.chrome.driver", "/path/to/chromedriver");
String proxyHost = "proxy.example.com";
int proxyPort = 8080;
String proxyAddress = proxyHost + ":" + proxyPort;
Proxy proxy = new Proxy();
proxy.setHttpProxy(proxyAddress); // For HTTP traffic
proxy.setSslProxy(proxyAddress); // For HTTPS traffic (often same as HTTP)
proxy.setSocksProxy(proxyAddress); // For SOCKS traffic
// proxy.setSocksUsername("socksuser"); // For SOCKS5 authentication
// proxy.setSocksPassword("sockspass");
ChromeOptions options = new ChromeOptions();
options.setCapability("proxy", proxy);
// options.addArguments("--proxy-server=" + proxyAddress); // Alternative for simple HTTP/HTTPS proxy
WebDriver driver = new ChromeDriver(options);
try {
driver.get("http://httpbin.org/get");
System.out.println("Page title: " + driver.getTitle());
// Further interactions with the page...
} finally {
driver.quit();
}
}
}
Firefox WebDriver
import org.openqa.selenium.Proxy;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxOptions;
public class SeleniumFirefoxProxySetup {
public static void main(String[] args) {
// Set path to GeckoDriver executable
// System.setProperty("webdriver.gecko.driver", "/path/to/geckodriver");
String proxyHost = "proxy.example.com";
int proxyPort = 8080;
String proxyAddress = proxyHost + ":" + proxyPort;
Proxy proxy = new Proxy();
proxy.setHttpProxy(proxyAddress);
proxy.setSslProxy(proxyAddress);
proxy.setSocksProxy(proxyAddress);
// proxy.setSocksUsername("socksuser");
// proxy.setSocksPassword("sockspass");
FirefoxOptions options = new FirefoxOptions();
options.setCapability("proxy", proxy);
WebDriver driver = new FirefoxDriver(options);
try {
driver.get("http://httpbin.org/get");
System.out.println("Page title: " + driver.getTitle());
} finally {
driver.quit();
}
}
}
Selenium Proxy Authentication
Selenium WebDriver itself does not directly handle proxy authentication (username/password for HTTP/HTTPS proxies). Common approaches include:
- Browser Extensions: Use a browser extension that handles proxy authentication. These extensions can be loaded via
ChromeOptionsorFirefoxOptions. - Proxy in URL: For some HTTP proxies, credentials can be embedded in the proxy URL (e.g.,
user:pass@proxy.example.com:8080), but this is not universally supported or secure. - SOCKS5 Authentication: Selenium's
Proxyclass supportssetSocksUsername()andsetSocksPassword()for SOCKS5 proxies. - Upstream Proxy Server: Route traffic through another local proxy server (e.g., BrowserMob Proxy) that manages authentication with the target proxy.
Comparison of Proxy Setup Methods
| Feature | Java System Properties (-D) |
Apache HttpClient (4.x/5.x) | Jsoup (.proxy()) |
Selenium WebDriver (Proxy class) |
|---|---|---|---|---|
| Scope | Global (JVM-wide) | Per HttpClient instance |
Per Connection instance |
Per WebDriver instance |
| Authentication | java.net.Authenticator (global) |
CredentialsProvider (per client) |
java.net.Authenticator (if global) |
SOCKS5 (setSocksUsername/Password); HTTP/HTTPS typically via extensions/external tools |
| Flexibility | Low (affects all java.net connections) |
High (fine-grained control, per-request config) | Moderate (per connection, but relies on global for auth) | High (browser-specific options, SOCKS auth) |
| Ease of Use | High (simple properties) | Moderate to High (requires more code) | High (simple method calls) | Moderate (requires creating Proxy object and setting options) |
| Supported Protocols | HTTP, HTTPS, SOCKS | HTTP, HTTPS, SOCKS | HTTP, HTTPS (via HttpURLConnection) |
HTTP, HTTPS, SOCKS (browser capabilities) |
| Use Cases | Simple, consistent proxy needs across app | Complex HTTP interactions, API clients | Web scraping, simple page fetching | Browser automation, testing, web scraping |