How to Use Proxies with Scrapy: Complete Guide

TL;DR

Learn how to configure GProxy residential and datacenter proxies with Scrapy for large-scale web scraping. Covers middleware setup, proxy rotation, and error handling.

Get Proxies → Scrapy website →

Language: Python

Prerequisites

✓[
✓"
✓P
✓y
✓t
✓h
✓o
✓n
✓
✓3
✓.
✓8
✓+
✓
✓i
✓n
✓s
✓t
✓a
✓l
✓l
✓e
✓d
✓"
✓,
✓
✓"
✓S
✓c
✓r
✓a
✓p
✓y
✓
✓2
✓.
✓x
✓
✓i
✓n
✓s
✓t
✓a
✓l
✓l
✓e
✓d
✓
✓(
✓p
✓i
✓p
✓
✓i
✓n
✓s
✓t
✓a
✓l
✓l
✓
✓s
✓c
✓r
✓a
✓p
✓y
✓)
✓"
✓,
✓
✓"
✓G
✓P
✓r
✓o
✓x
✓y
✓
✓a
✓c
✓c
✓o
✓u
✓n
✓t
✓
✓w
✓i
✓t
✓h
✓
✓a
✓c
✓t
✓i
✓v
✓e
✓
✓p
✓r
✓o
✓x
✓y
✓
✓p
✓l
✓a
✓n
✓"
✓,
✓
✓"
✓B
✓a
✓s
✓i
✓c
✓
✓f
✓a
✓m
✓i
✓l
✓i
✓a
✓r
✓i
✓t
✓y
✓
✓w
✓i
✓t
✓h
✓
✓S
✓c
✓r
✓a
✓p
✓y
✓
✓s
✓p
✓i
✓d
✓e
✓r
✓s
✓
✓a
✓n
✓d
✓
✓s
✓e
✓t
✓t
✓i
✓n
✓g
✓s
✓"
✓]

Step-by-Step Setup

[

{

"

T

I

T

L

E

"

:

"

I

N

S

T

A

L

S

C

R

A

P

Y

"

,

"

D

E

S

C

R

I

P

T

I

O

N

"

:

"

I

N

S

T

A

L

S

C

R

A

P

Y

A

N

D

C

R

E

A

T

E

A

N

E

W

P

R

O

J

E

C

T

"

,

"

C

O

D

E

"

:

"

P

I

P

100

101

I

102

N

103

S

104

T

105

A

106

L

107

L

108

109

S

110

C

111

R

112

A

113

P

114

Y

115

\

116

N

117

S

118

C

119

R

120

A

121

P

122

Y

123

124

S

125

T

126

A

127

R

128

T

129

P

130

R

131

O

132

J

133

E

134

C

135

T

136

137

M

138

Y

139

P

140

R

141

O

142

J

143

E

144

C

145

T

146

\

147

N

148

C

149

D

150

151

M

152

Y

153

P

154

R

155

O

156

J

157

E

158

C

159

T

160

"

161

}

162

,

163

164

{

165

"

166

T

167

I

168

T

169

L

170

E

171

"

172

:

173

174

"

175

C

176

O

177

N

178

F

179

I

180

G

181

U

182

R

183

E

184

185

P

186

R

187

O

188

X

189

Y

190

191

I

192

N

193

194

S

195

E

196

T

197

T

198

I

199

N

200

G

201

S

202

.

203

P

204

Y

205

"

206

,

207

208

"

209

D

210

E

211

S

212

C

213

R

214

I

215

P

216

T

217

I

218

O

219

N

220

"

221

:

222

223

"

224

A

225

D

226

D

227

228

P

229

R

230

O

231

X

232

Y

233

234

S

235

E

236

T

237

T

238

I

239

N

240

G

241

S

242

243

T

244

O

245

246

Y

247

O

248

U

249

R

250

251

P

252

R

253

O

254

J

255

E

256

C

257

T

258

'

259

S

260

261

S

262

E

263

T

264

T

265

I

266

N

267

G

268

S

269

.

270

P

271

Y

272

"

273

,

274

275

"

276

C

277

O

278

D

279

E

280

"

281

:

282

283

"

284

#

285

286

S

287

E

288

T

289

T

290

I

291

N

292

G

293

S

294

.

295

P

296

Y

297

\

298

N

299

H

300

T

301

T

302

P

303

P

304

R

305

O

306

X

307

Y

308

_

309

E

310

N

311

A

312

B

313

L

314

E

315

D

316

317

=

318

319

T

320

R

321

U

322

E

323

\

324

N

325

D

326

O

327

W

328

N

329

L

330

O

331

A

332

D

333

E

334

R

335

_

336

M

337

I

338

D

339

D

340

L

341

E

342

W

343

A

344

R

345

E

346

S

347

348

=

349

350

{

351

\

352

N

353

354

355

356

357

'

358

M

359

Y

360

P

361

R

362

O

363

J

364

E

365

C

366

T

367

.

368

M

369

I

370

D

371

D

372

L

373

E

374

W

375

A

376

R

377

E

378

S

379

.

380

G

381

P

382

R

383

O

384

X

385

Y

386

M

387

I

388

D

389

D

390

L

391

E

392

W

393

A

394

R

395

E

396

'

397

:

398

399

3

400

5

401

0

402

,

403

\

404

N

405

406

407

408

409

'

410

S

411

C

412

R

413

A

414

P

415

Y

416

.

417

D

418

O

419

W

420

N

421

L

422

O

423

A

424

D

425

E

426

R

427

M

428

I

429

D

430

D

431

L

432

E

433

W

434

A

435

R

436

E

437

S

438

.

439

H

440

T

441

T

442

P

443

P

444

R

445

O

446

X

447

Y

448

.

449

H

450

T

451

T

452

P

453

P

454

R

455

O

456

X

457

Y

458

M

459

I

460

D

461

D

462

L

463

E

464

W

465

A

466

R

467

E

468

'

469

:

470

471

4

472

0

473

0

474

,

475

\

476

N

477

}

478

"

479

}

480

,

481

482

{

483

"

484

T

485

I

486

T

487

L

488

E

489

"

490

:

491

492

"

493

C

494

R

495

E

496

A

497

T

498

E

499

500

P

501

R

502

O

503

X

504

Y

505

506

M

507

I

508

D

509

D

510

L

511

E

512

W

513

A

514

R

515

E

516

"

517

,

518

519

"

520

D

521

E

522

S

523

C

524

R

525

I

526

P

527

T

528

I

529

O

530

N

531

"

532

:

533

534

"

535

C

536

R

537

E

538

A

539

T

540

E

541

542

A

543

544

C

545

U

546

S

547

T

548

O

549

M

550

551

M

552

I

553

D

554

D

555

L

556

E

557

W

558

A

559

R

560

E

561

562

I

563

N

564

565

M

566

I

567

D

568

D

569

L

570

E

571

W

572

A

573

R

574

E

575

S

576

.

577

P

578

Y

579

"

580

,

581

582

"

583

C

584

O

585

D

586

E

587

"

588

:

589

590

"

591

C

592

L

593

A

594

S

595

S

596

597

G

598

P

599

R

600

O

601

X

602

Y

603

M

604

I

605

D

606

D

607

L

608

E

609

W

610

A

611

R

612

E

613

:

614

\

615

N

616

617

618

619

620

D

621

E

622

F

623

624

P

625

R

626

O

627

C

628

E

629

S

630

S

631

_

632

R

633

E

634

Q

635

U

636

E

637

S

638

T

639

(

640

S

641

E

642

L

643

F

644

,

645

646

R

647

E

648

Q

649

U

650

E

651

S

652

T

653

,

654

655

S

656

P

657

I

658

D

659

E

660

R

661

)

662

:

663

\

664

N

665

666

667

668

669

670

671

672

673

P

674

R

675

O

676

X

677

Y

678

679

=

680

681

'

682

H

683

T

684

T

685

P

686

:

687

/

688

/

689

U

690

S

691

E

692

R

693

:

694

P

695

A

696

S

697

S

698

@

699

P

700

R

701

O

702

X

703

Y

704

.

705

G

706

P

707

R

708

O

709

X

710

Y

711

.

712

N

713

E

714

T

715

:

716

1

717

0

718

0

719

0

720

'

721

\

722

N

723

724

725

726

727

728

729

730

731

R

732

E

733

Q

734

U

735

E

736

S

737

T

738

.

739

M

740

E

741

T

742

A

743

[

744

'

745

P

746

R

747

O

748

X

749

Y

750

'

751

]

752

753

=

754

755

P

756

R

757

O

758

X

759

Y

760

"

761

}

762

,

763

764

{

765

"

766

T

767

I

768

T

769

L

770

E

771

"

772

:

773

774

"

775

A

776

D

777

D

778

779

R

780

O

781

T

782

A

783

T

784

I

785

O

786

N

787

788

S

789

U

790

P

791

P

792

O

793

R

794

T

795

"

796

,

797

798

"

799

D

800

E

801

S

802

C

803

R

804

I

805

P

806

T

807

I

808

O

809

N

810

"

811

:

812

813

"

814

F

815

O

816

R

817

818

A

819

U

820

T

821

O

822

M

823

A

824

T

825

I

826

C

827

828

I

829

P

830

831

R

832

O

833

T

834

A

835

T

836

I

837

O

838

N

839

,

840

841

U

842

S

843

E

844

845

T

846

H

847

E

848

849

B

850

A

851

C

852

K

853

C

854

O

855

N

856

N

857

E

858

C

859

T

860

861

E

862

N

863

D

864

P

865

O

866

I

867

N

868

T

869

"

870

,

871

872

"

873

C

874

O

875

D

876

E

877

"

878

:

879

880

"

881

C

882

L

883

A

884

S

885

S

886

887

G

888

P

889

R

890

O

891

X

892

Y

893

R

894

O

895

T

896

A

897

T

898

I

899

N

900

G

901

M

902

I

903

D

904

D

905

L

906

E

907

W

908

A

909

R

910

E

911

:

912

\

913

N

914

915

916

917

918

D

919

E

920

F

921

922

P

923

R

924

O

925

C

926

E

927

S

928

S

929

_

930

R

931

E

932

Q

933

U

934

E

935

S

936

T

937

(

938

S

939

E

940

L

941

F

942

,

943

944

R

945

E

946

Q

947

U

948

E

949

S

950

T

951

,

952

953

S

954

P

955

I

956

D

957

E

958

R

959

)

960

:

961

\

962

N

963

964

965

966

967

968

969

970

971

R

972

E

973

Q

974

U

975

E

976

S

977

T

978

.

979

M

980

E

981

T

982

A

983

[

984

'

985

P

986

R

987

O

988

X

989

Y

990

'

991

]

992

993

=

994

995

'

996

H

997

T

998

T

999

P

1000

:

1001

/

1002

/

1003

U

1004

S

1005

E

1006

R

1007

-

1008

C

1009

O

1010

U

1011

N

1012

T

1013

R

1014

Y

1015

-

1016

U

1017

S

1018

:

1019

P

1020

A

1021

S

1022

S

1023

@

1024

P

1025

R

1026

O

1027

X

1028

Y

1029

.

1030

G

1031

P

1032

R

1033

O

1034

X

1035

Y

1036

.

1037

N

1038

E

1039

T

1040

:

1041

1

1042

0

1043

0

1044

0

1045

'

1046

\

1047

N

1048

1049

1050

1051

1052

\

1053

N

1054

1055

1056

1057

1058

D

1059

E

1060

F

1061

1062

P

1063

R

1064

O

1065

C

1066

E

1067

S

1068

S

1069

_

1070

E

1071

X

1072

C

1073

E

1074

P

1075

T

1076

I

1077

O

1078

N

1079

(

1080

S

1081

E

1082

L

1083

F

1084

,

1085

1086

R

1087

E

1088

Q

1089

U

1090

E

1091

S

1092

T

1093

,

1094

1095

E

1096

X

1097

C

1098

E

1099

P

1100

T

1101

I

1102

O

1103

N

1104

,

1105

1106

S

1107

P

1108

I

1109

D

1110

E

1111

R

1112

)

1113

:

1114

\

1115

N

1116

1117

1118

1119

1120

1121

1122

1123

1124

R

1125

E

1126

T

1127

U

1128

R

1129

N

1130

1131

R

1132

E

1133

Q

1134

U

1135

E

1136

S

1137

T

1138

1139

1140

#

1141

1142

R

1143

E

1144

T

1145

R

1146

Y

1147

1148

W

1149

I

1150

T

1151

H

1152

1153

N

1154

E

1155

W

1156

1157

I

1158

P

1159

"

1160

}

1161

,

1162

1163

{

1164

"

1165

T

1166

I

1167

T

1168

L

1169

E

1170

"

1171

:

1172

1173

"

1174

C

1175

O

1176

N

1177

F

1178

I

1179

G

1180

U

1181

R

1182

E

1183

1184

R

1185

E

1186

T

1187

R

1188

Y

1189

1190

S

1191

E

1192

T

1193

T

1194

I

1195

N

1196

G

1197

S

1198

"

1199

,

1200

1201

"

1202

D

1203

E

1204

S

1205

C

1206

R

1207

I

1208

P

1209

T

1210

I

1211

O

1212

N

1213

"

1214

:

1215

1216

"

1217

A

1218

D

1219

D

1220

1221

R

1222

E

1223

T

1224

R

1225

Y

1226

1227

C

1228

O

1229

N

1230

F

1231

I

1232

G

1233

U

1234

R

1235

A

1236

T

1237

I

1238

O

1239

N

1240

1241

F

1242

O

1243

R

1244

1245

R

1246

E

1247

L

1248

I

1249

A

1250

B

1251

I

1252

L

1253

I

1254

T

1255

Y

1256

"

1257

,

1258

1259

"

1260

C

1261

O

1262

D

1263

E

1264

"

1265

:

1266

1267

"

1268

#

1269

1270

S

1271

E

1272

T

1273

T

1274

I

1275

N

1276

G

1277

S

1278

.

1279

P

1280

Y

1281

\

1282

N

1283

R

1284

E

1285

T

1286

R

1287

Y

1288

_

1289

E

1290

N

1291

A

1292

B

1293

L

1294

E

1295

D

1296

1297

=

1298

1299

T

1300

R

1301

U

1302

E

1303

\

1304

N

1305

R

1306

E

1307

T

1308

R

1309

Y

1310

_

1311

T

1312

I

1313

M

1314

E

1315

S

1316

1317

=

1318

1319

5

1320

\

1321

N

1322

R

1323

E

1324

T

1325

R

1326

Y

1327

_

1328

H

1329

T

1330

T

1331

P

1332

_

1333

C

1334

O

1335

D

1336

E

1337

S

1338

1339

=

1340

1341

[

1342

4

1343

0

1344

3

1345

,

1346

1347

4

1348

0

1349

7

1350

,

1351

1352

4

1353

2

1354

9

1355

,

1356

1357

5

1358

0

1359

0

1360

,

1361

1362

5

1363

0

1364

2

1365

,

1366

1367

5

1368

0

1369

3

1370

]

1371

\

1372

N

1373

D

1374

O

1375

W

1376

N

1377

L

1378

O

1379

A

1380

D

1381

_

1382

T

1383

I

1384

M

1385

E

1386

O

1387

U

1388

T

1389

1390

=

1391

1392

3

1393

0

1394

"

1395

}

1396

,

1397

1398

{

1399

"

1400

T

1401

I

1402

T

1403

L

1404

E

1405

"

1406

:

1407

1408

"

1409

R

1410

U

1411

N

1412

1413

Y

1414

O

1415

U

1416

R

1417

1418

S

1419

P

1420

I

1421

D

1422

E

1423

R

1424

"

1425

,

1426

1427

"

1428

D

1429

E

1430

S

1431

C

1432

R

1433

I

1434

P

1435

T

1436

I

1437

O

1438

N

1439

"

1440

:

1441

1442

"

1443

T

1444

E

1445

S

1446

T

1447

1448

T

1449

H

1450

E

1451

1452

P

1453

R

1454

O

1455

X

1456

Y

1457

1458

C

1459

O

1460

N

1461

F

1462

I

1463

G

1464

U

1465

R

1466

A

1467

T

1468

I

1469

O

1470

N

1471

"

1472

,

1473

1474

"

1475

C

1476

O

1477

D

1478

E

1479

"

1480

:

1481

1482

"

1483

S

1484

C

1485

R

1486

A

1487

P

1488

Y

1489

1490

C

1491

R

1492

A

1493

W

1494

L

1495

1496

M

1497

Y

1498

S

1499

P

1500

I

1501

D

1502

E

1503

R

1504

1505

-

1506

O

1507

1508

O

1509

U

1510

T

1511

P

1512

U

1513

T

1514

.

1515

J

1516

S

1517

O

1518

N

1519

"

1520

}

1521

]

Code Examples

Why Use Proxies with Scrapy?

Scrapy is the most popular Python web scraping framework, powering millions of crawl jobs daily. When scraping at scale, you'll inevitably face IP blocks, CAPTCHAs, and rate limits. GProxy's residential proxy pool of 10M+ IPs across 150+ countries solves these challenges by rotating your exit IP automatically.

Proxy Authentication Methods

GProxy supports two authentication methods with Scrapy: username/password authentication through the proxy URL, and IP whitelist authentication where you add your server's IP to the GProxy dashboard. Both work seamlessly with Scrapy's built-in proxy support.

Basic Proxy Configuration

The simplest way to add GProxy proxies to Scrapy is through the settings.py file. Set the HTTP_PROXY and HTTPS_PROXY environment variables, or configure them directly in your spider's custom settings. For rotating proxies, you'll want to use a custom downloader middleware that assigns a new proxy URL per request.

Custom Proxy Rotation Middleware

For production scraping, a custom middleware gives you full control over proxy assignment. The middleware intercepts each request before it's sent, assigns a proxy endpoint with rotation parameters, and handles proxy-related errors like timeouts or 407 authentication failures. GProxy's backconnect endpoint handles rotation server-side, so each connection through the same endpoint gets a different IP.

Handling Errors and Retries

Robust scraping requires proper error handling. Configure Scrapy's retry middleware to work with your proxy setup: retry on 403, 407, 429, and 503 status codes, and set a reasonable retry count (3-5 attempts). GProxy's sticky sessions let you maintain the same IP across retries when needed — useful for multi-page flows.

Concurrent Requests and Performance

Scrapy excels at concurrent scraping, and GProxy's infrastructure handles high concurrency well. Start with CONCURRENT_REQUESTS = 32 and DOWNLOAD_DELAY = 0.5, then adjust based on target site behavior. With GProxy residential proxies, you can typically run 50-100 concurrent requests without issues.

Geo-Targeting with Scrapy

GProxy supports country and city-level targeting. Pass the target country as part of your proxy credentials (e.g., user-country-us) to route requests through IPs in that location. This is essential for scraping geo-restricted content or localized search results.

Use Cases

[

{

"

T

I

T

L

E

"

:

"

E

-

C

O

M

E

R

C

E

P

R

I

C

E

M

O

N

I

T

O

R

I

N

G

"

,

"

D

E

S

C

R

I

P

T

I

O

N

"

:

"

S

C

R

A

P

E

P

R

O

D

U

C

T

P

R

I

C

E

S

A

C

R

O

S

M

U

L

T

I

P

L

E

R

E

T

A

I

L

E

R

S

W

I

T

H

G

E

O

-

T

A

R

G

E

T

E

D

P

R

O

X

I

E

S

T

O

S

E

L

O

C

A

L

I

Z

E

D

P

R

I

C

I

N

G

.

"

}

,

{

"

T

I

T

L

E

"

:

"

S

E

R

P

T

R

A

C

K

I

N

G

"

,

"

D

E

S

C

R

I

P

T

I

O

N

"

:

"

M

O

N

I

T

O

R

S

E

A

R

C

H

E

N

G

I

N

E

R

A

N

K

I

N

G

S

F

R

O

M

D

I

F

E

R

E

N

T

L

O

C

A

T

I

O

N

S

U

S

I

N

G

C

O

U

N

T

R

Y

-

S

P

E

C

I

F

I

C

R

E

S

I

D

E

N

T

I

A

L

I

P

S

.

"

}

,

{

"

T

I

T

L

E

"

:

"

L

E

A

D

G

E

N

E

R

A

T

I

O

N

"

,

"

D

E

S

C

R

I

P

T

I

O

N

"

:

"

E

X

T

R

A

C

T

B

U

S

I

N

E

S

L

I

S

T

I

N

G

S

A

N

D

C

O

N

T

A

C

T

D

A

T

A

T

S

C

A

L

E

W

I

T

H

R

O

T

A

T

I

N

G

I

P

S

T

O

A

V

O

I

D

E

T

E

C

T

I

O

N

.

"

}

,

{

"

T

I

T

L

E

"

:

"

C

O

N

T

E

N

T

A

G

R

E

G

A

T

I

O

N

"

,

"

D

E

S

C

R

I

P

T

I

O

N

"

:

"

C

O

L

E

C

T

A

R

T

I

C

L

E

S

,

R

E

V

I

E

W

S

,

A

N

D

L

I

S

T

I

N

G

S

F

R

O

M

U

L

T

I

P

L

E

S

O

U

R

C

E

S

W

I

T

H

R

E

L

I

A

B

L

E

P

R

O

X

Y

R

O

T

A

T

I

O

N

.

"

}

]

Pro Tips

Use AUTOTHROTTLE_ENABLED = True for automatic rate adjustment. Set CONCURRENT_REQUESTS_PER_DOMAIN to limit per-site load. Enable HTTP caching during development to avoid burning proxy bandwidth. Use scrapy-rotating-proxies package for advanced pool management.