How to Create a Proxy Pool with Automatic Checking

How to Create a Proxy Pool with Automatic Checking

Introduction

A proxy pool with automatic checking is a system that stores a list of proxies, regularly checks their availability, and provides working proxies via an API.

Basic Setup

Architecture

Components of a proxy pool:
1. Database — storing proxies (PostgreSQL, Redis)
2. Checker — checking proxy availability
3. API — providing working proxies
4. Scheduler — periodically running checks
5. Web UI — monitoring (optional)

Data Model

For each proxy, store:
- ip, port, type (http/socks5)
- username, password
- status (alive/dead)
- latency (ms)
- country, city
- last_checked, last_alive
- fail_count

Python Checker

Use aiohttp for asynchronous checking of hundreds of proxies simultaneously. For each proxy:
1. Send a request through the proxy to httpbin.org/ip
2. Measure the response time
3. Verify that the IP matches the expected one
4. Update the status in the database

Advanced Configuration

Environment Variables

Most tools support standard environment variables for proxies:
- HTTP_PROXY / http_proxy — proxy for HTTP requests
- HTTPS_PROXY / https_proxy — proxy for HTTPS requests
- NO_PROXY / no_proxy — list of exceptions (addresses that bypass the proxy)
- ALL_PROXY / all_proxy — proxy for all protocols

SSL and Self-Signed Certificates

Corporate proxies often use their own SSL certificates. To work through them, you need to:
1. Obtain the proxy's CA certificate from the administrator
2. Add it to the system certificate store
3. Or disable SSL verification (for testing only)

Authentication

Two main authentication methods are supported:

By username and password — standard HTTP Basic authentication. Credentials are sent with each request. URL format: http://user:pass@proxy_ip:port

By IP address (whitelist) — your IP is added to a whitelist. No credentials need to be sent.

Verifying Operation

After setup, verify:
1. Perform a test request through the proxy
2. Ensure that the IP has changed
3. Check the connection speed
4. Ensure there are no DNS leaks

Troubleshooting

Proxy Not Connecting

• Check the address and port for correctness
• Ensure that the firewall is not blocking the connection
• Check the proxy's accessibility from your network

SSL Errors

• For corporate proxies, add the CA certificate
• For testing — temporarily disable SSL verification
• Update SSL libraries to the latest version

Authentication Errors

• Check the username and password
• Ensure the credentials format is correct
• Try URL encoding special characters in the password

Best Practices

1. Use environment variables — for flexibility and security
2. Do not hardcode credentials — use environment variables or configuration files
3. Configure exceptions — do not proxy localhost and internal addresses
4. Document settings — in the project README or team wiki
5. Test after changes — any configuration change requires verification

Conclusion

Correct proxy configuration ensures stable tool operation and connection security. Follow the recommendations in this guide and always verify functionality after setup.