How to Configure Proxies for Selenium in Python: A Complete Guide

Configuring proxies for Selenium in Python requires passing proxy settings through browser-specific options or utilizing middleware libraries like Selenium-wire for authenticated sessions. For standard HTTP or SOCKS5 proxies without authentication, the --proxy-server argument within ChromeOptions is the most efficient method, while authenticated proxies often necessitate custom browser extensions or proxy-aware drivers.

Why Proxy Integration is Essential for Selenium Automation

Selenium is a powerful tool for web automation, but its default behavior—operating from a single, static IP address—makes it highly susceptible to rate-limiting and IP blacklisting. When performing large-scale data extraction or automated testing across different geographical regions, a proxy acts as an intermediary, masking your actual origin and distributing requests across multiple IP addresses.

Using a service like GProxy provides access to residential and data center pools that prevent websites from identifying the traffic as coming from a single automated source. This is particularly critical for:

• Web Scraping: Circumventing anti-bot mechanisms like Cloudflare or Akamai that monitor request frequency per IP.
• Localization Testing: Verifying that a website displays the correct currency, language, and content for users in specific countries.
• Ad Verification: Ensuring that advertisements are being served correctly to the intended audience without being blocked or redirected.
• Load Testing: Simulating traffic from diverse geographical locations to test server response times and CDN efficiency.

Basic Proxy Configuration for Chrome and Firefox

The most straightforward way to implement a proxy in Selenium is through the browser's native options. This method works best for "White-listed IP" proxies, where the proxy provider (such as GProxy) authorizes your server's IP address, removing the need for a username and password in the connection string.

Configuring Chrome with Options

In Chrome, you use the add_argument method. This is the fastest implementation and adds negligible overhead to the browser startup time.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

proxy_ip_port = "gw.gproxy.com:10000"

chrome_options = Options()
chrome_options.add_argument(f'--proxy-server={proxy_ip_port}')

driver = webdriver.Chrome(options=chrome_options)
driver.get("https://api.ipify.org?format=json")
print(driver.page_source)
driver.quit()

Configuring Firefox with the Proxy Object

Firefox handles proxies differently, often utilizing the Proxy object from selenium.webdriver.common.proxy. This allows for more granular control over which protocols (HTTP, SSL, SOCKS) use the proxy.

from selenium import webdriver
from selenium.webdriver.common.proxy import Proxy, ProxyType

proxy_host = "gw.gproxy.com:10000"

proxy = Proxy()
proxy.proxy_type = ProxyType.MANUAL
proxy.http_proxy = proxy_host
proxy.ssl_proxy = proxy_host

options = webdriver.FirefoxOptions()
options.proxy = proxy

driver = webdriver.Firefox(options=options)
driver.get("https://httpbin.org/ip")
driver.quit()

Handling Proxy Authentication in Selenium

Standard Selenium does not natively support proxies that require a username and password via the --proxy-server argument. If you attempt to use http://user:pass@host:port, Chrome will typically ignore the credentials or trigger a native browser pop-up that Selenium cannot easily interact with. To solve this, there are two primary professional approaches.

Method 1: Using Selenium-wire

Selenium-wire is a wrapper for Selenium that extends its capabilities to inspect requests and responses and, most importantly, handle proxy authentication transparently. It achieves this by running a local proxy server that intercepts traffic and injects the necessary authentication headers.

from seleniumwire import webdriver

# GProxy credentials and endpoint
proxy_options = {
    'proxy': {
        'http': 'http://username:password@gw.gproxy.com:10000',
        'https': 'https://username:password@gw.gproxy.com:10000',
        'no_proxy': 'localhost,127.0.0.1'
    }
}

driver = webdriver.Chrome(seleniumwire_options=proxy_options)
driver.get("https://www.gproxy.com/check-ip")
driver.quit()

Method 2: The Custom Extension Approach

For high-performance environments where the overhead of Selenium-wire's MITM (Man-In-The-Middle) proxy is undesirable, creating a temporary Chrome extension on the fly is the industry-standard workaround. This extension uses the chrome.proxy API to set credentials.

import os
import zipfile
from selenium import webdriver

def create_proxy_extension(proxy_host, proxy_port, proxy_user, proxy_pass):
    manifest_json = """
    {
        "version": "1.0.0",
        "manifest_version": 2,
        "name": "GProxy Extension",
        "permissions": [
            "proxy",
            "tabs",
            "unlimitedStorage",
            "storage",
            "<all_urls>",
            "webRequest",
            "webRequestBlocking"
        ],
        "background": {
            "scripts": ["background.js"]
        },
        "minimum_chrome_version":"22.0.0"
    }
    """

    background_js = f"""
    var config = {{
            mode: "fixed_servers",
            rules: {{
              singleProxy: {{
                scheme: "http",
                host: "{proxy_host}",
                port: parseInt({proxy_port})
              }},
              bypassList: ["localhost"]
            }}
          }};

    chrome.proxy.settings.set({{value: config, scope: "regular"}}, function() {{}});

    chrome.webRequest.onAuthRequired.addListener(
                function(details) {{
                    return {{
                        authCredentials: {{
                            username: "{proxy_user}",
                            password: "{proxy_pass}"
                        }}
                    }};
                }},
                {{urls: ["<all_urls>"]}},
                ['blocking']
    );
    """
    
    extension_path = 'proxy_auth_plugin.zip'
    with zipfile.ZipFile(extension_path, 'w') as zp:
        zp.writestr("manifest.json", manifest_json)
        zp.writestr("background.js", background_js)
    
    return extension_path

# Implementation
proxy_ext = create_proxy_extension("gw.gproxy.com", "10000", "my_user", "my_pass")
options = webdriver.ChromeOptions()
options.add_extension(proxy_ext)
driver = webdriver.Chrome(options=options)

Comparison of Proxy Configuration Methods

Choosing the right method depends on your performance requirements, the type of proxy (Residential vs. Datacenter), and whether you need to rotate IPs frequently.

Advanced Proxy Management: Rotation and Sticky Sessions

When using GProxy, you often have the choice between Rotating Proxies and Sticky Sessions. Understanding how to implement these in Python is vital for maintaining session persistence during checkout flows or multi-step form submissions.

Implementing Sticky Sessions

A sticky session ensures that all requests made by your Selenium driver instance stay on the same IP address for a set duration (e.g., 10 to 30 minutes). With GProxy, this is usually handled via a session ID in the username string.

# Example of a sticky session using GProxy residential pool
session_id = "session_82734"
proxy_user = f"user-customer_id-session-{session_id}"
# Now use this user string in your Selenium-wire or Extension config

Managing Headless Mode with Proxies

Running Selenium in headless mode (--headless) is standard for server-side scripts. However, headless Chrome can sometimes leak your real IP or fail to load extensions. If you are using the Extension method for authentication, you must use the newer headless mode available in Chrome 109+ to ensure extensions are loaded correctly:

options.add_argument('--headless=new')

The --headless=new flag ensures that the browser environment more closely mimics a headed browser, improving the reliability of proxy authentication and reducing the chances of detection by advanced bot-mitigation scripts.

Troubleshooting Common Proxy Issues

Even with correct configuration, Selenium proxy setups can encounter issues. Here are the most frequent problems and their technical resolutions:

• ProxyConnectionError: Often caused by an incorrect port or a firewall blocking the outbound connection to the proxy provider. Ensure port 10000 (or your specific provider's port) is open.
• DNS Leaks: By default, Selenium may still resolve DNS queries through your local ISP. To prevent this, especially when using SOCKS5, ensure the browser is configured to perform DNS lookups through the proxy.
• Slow Load Times: Residential proxies are naturally slower than datacenter proxies. If performance is critical, use GProxy's datacenter pool unless you are facing aggressive IP blocking.
• Authentication Pop-ups: If you see a login pop-up, your authentication method (Extension or Selenium-wire) has failed or is not being initialized before the first request.

Key Takeaways

Configuring proxies in Selenium is a multi-tiered process. While simple IP-whitelisted proxies can be handled via command-line arguments, authenticated residential proxies require more robust solutions like custom extensions or Selenium-wire.

• Use GProxy Backconnect: Instead of managing a list of 10,000 IPs in your Python code, use a single backconnect endpoint provided by GProxy to handle rotation automatically.
• Prioritize Extensions: For production environments, the Chrome Extension method is superior to Selenium-wire as it does not introduce a secondary proxy layer, resulting in lower latency.
• Monitor Success Rates: Always wrap your driver.get() calls in try-except blocks to catch WebDriverException, allowing your script to rotate the session ID and retry if a proxy node fails.