Residential proxies with Scrapy

Get your credentials

Your username and password are on the Proxy generator page of the dashboard. Use the password exactly as shown there; country, state, city and session options all go in the username.

The gateway is gw.nodelighthouse.com. Port 9000 rotates (a new IP per request), port 10000 is sticky (the username carries -lifetime-N-session-xxxxxxxx), and port 11000 is SOCKS5.

Set the proxy per request

Put the full proxy URL, with credentials, in meta["proxy"]. HttpProxyMiddleware is enabled by default, so no settings change is needed. On the rotating port each request leaves from a new IP.

import scrapy

PROXY = "http://<USERNAME>-type-residential-country-us:<PASSWORD>@gw.nodelighthouse.com:9000"


class IpSpider(scrapy.Spider):
    name = "ip"

    def start_requests(self):
        for _ in range(3):
            yield scrapy.Request(
                "https://api.ipify.org?format=json",
                meta={"proxy": PROXY},
                dont_filter=True,
            )

    def parse(self, response):
        self.logger.info(response.text)

Set it once for every request

A small downloader middleware saves you from repeating meta on every request. Give it a priority below 750 so it runs before HttpProxyMiddleware reads the meta key.

# myproject/middlewares.py
PROXY = "http://<USERNAME>-type-residential-country-us:<PASSWORD>@gw.nodelighthouse.com:9000"


class NodeLighthouseProxyMiddleware:
    def process_request(self, request, spider):
        request.meta.setdefault("proxy", PROXY)


# myproject/settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.NodeLighthouseProxyMiddleware": 350,
}

Use sticky sessions for multi-step flows

When a crawl needs several requests from one IP, such as a login followed by account pages, switch to port 10000 and give that flow its own session id. Pass the same proxy URL along to follow-up requests.

import secrets
import scrapy


def sticky_proxy():
    session_id = secrets.token_hex(4)  # 8 chars, a-f0-9
    return f"http://<USERNAME>-type-residential-country-us-lifetime-10-session-{session_id}:<PASSWORD>@gw.nodelighthouse.com:10000"


class FlowSpider(scrapy.Spider):
    name = "flow"

    def start_requests(self):
        proxy = sticky_proxy()
        yield scrapy.Request("https://api.ipify.org", meta={"proxy": proxy}, cb_kwargs={"proxy": proxy})

    def parse(self, response, proxy):
        self.logger.info("first: %s", response.text)
        yield scrapy.Request(
            "https://api.ipify.org",
            meta={"proxy": proxy},
            dont_filter=True,
            callback=self.parse_second,
        )

    def parse_second(self, response):
        self.logger.info("second (same IP): %s", response.text)

Questions

Does Scrapy support SOCKS5 proxies?

Not with the built-in HttpProxyMiddleware, which handles HTTP proxies. Use the HTTP ports (9000 rotating, 10000 sticky) with Scrapy. Both carry HTTPS traffic through a tunnel.

How many concurrent requests can I run?

Scrapy's CONCURRENT_REQUESTS and CONCURRENT_REQUESTS_PER_DOMAIN control this. Start modestly, watch response codes and adjust. You pay a flat $1.65 per GB of traffic, not per request or per thread.

How do I target a different country per spider?

Change the country in the username, for example -country-de or -country-gb, and keep the rest the same. The username format is <USERNAME>-type-residential[-country-us][-state-california][-city-losangeles][-lifetime-10-session-abc12345]. The quick start lists the options.