Scrapy can use Magnetic Proxy through its standard HTTP proxy support. Set a complete proxy URL on each request through Scrapy's proxy metadata, then choose rotating or sticky behavior in the Magnetic Proxy connection profile. No Magnetic Proxy-specific Scrapy SDK is required.

The implementation has three separate responsibilities: Scrapy controls scheduling and retries, the proxy profile controls route and session behavior, and the spider validates whether the response contains usable data.

What you need before you start

• A permitted target and a documented collection purpose.
• A Magnetic Proxy connection profile with endpoint, username parameters, and password.
• A server-side environment variable for the full proxy URL. Do not hard-code credentials in a spider or commit them to source control.
• A small test set that includes normal pages, pagination, expected errors, and target-specific quality checks.

Scrapy's HttpProxyMiddleware documentation supports a per-request proxy value. For Scrapy 2.18 with the default HTTP/1.1 download handler, use Magnetic Proxy's HTTP endpoint for both HTTP and HTTPS destination URLs, for example http://customer-USERNAME:PASSWORD@rs.magneticproxy.net:1080. Replace the placeholders with the credentials from the active profile and keep the value outside source control. Do not switch that value to an HTTPS or SOCKS proxy URL unless the project's active download handler explicitly supports that proxy scheme.

Choose rotation, sticky sessions, and retries independently

The proxy mode should follow the unit of work, while retry behavior should follow the failure type. On smaller screens, swipe horizontally to view all columns.

Set the proxy on Scrapy requests

The smallest implementation sets the complete proxy URL in request metadata. Keep that value in an environment variable so the connection profile can change without editing spider code.

Export MP_PROXY_URL in the process environment before starting Scrapy. A .env file is not loaded by Scrapy automatically unless the project adds its own environment loader.

import os
import scrapy


class CatalogSpider(scrapy.Spider):
    name = "catalog"
    allowed_domains = ["example.com"]

    async def start(self):
        proxy_url = os.getenv("MP_PROXY_URL")
        if not proxy_url:
            raise RuntimeError(
                "Set MP_PROXY_URL before starting Scrapy."
            )

        yield scrapy.Request(
            "https://example.com/catalog",
            meta={"proxy": proxy_url},
            callback=self.parse,
        )

    def parse(self, response):
        # Validate the fields your workflow requires.
        yield {"title": response.css("h1::text").get()}

For a project-wide setting, use the downloader middleware below instead of repeating meta={"proxy": ...} on every request. Keep per-request proxy metadata only when a request intentionally overrides the project-wide profile.

# myproject/middlewares.py
import os


class MagneticProxyMiddleware:
    def __init__(self):
        try:
            self.proxy_url = os.environ["MP_PROXY_URL"]
        except KeyError as exc:
            raise RuntimeError(
                "Set MP_PROXY_URL before starting Scrapy."
            ) from exc

    def process_request(self, request):
        request.meta.setdefault("proxy", self.proxy_url)

Replace myproject with the Python package name created by scrapy startproject. The middleware order 350 runs this assignment before Scrapy's built-in HttpProxyMiddleware, which parses the URL and applies proxy authentication.

Start with conservative Scrapy settings

Scrapy already includes retry controls. Its current documentation lists two retries by default, in addition to the initial request, and a default set of transient HTTP status codes. Keep the initial policy simple and change it only with target-specific evidence.

# settings.py (Scrapy 2.18+)
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.MagneticProxyMiddleware": 350,
}

RETRY_ENABLED = True
RETRY_TIMES = 2
DOWNLOAD_TIMEOUT = 30

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 30.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
CONCURRENT_REQUESTS_PER_DOMAIN = 2

ROBOTSTXT_OBEY = True

Scrapy's RetryMiddleware documentation is the source of truth for retry defaults. Its AutoThrottle documentation explains how latency and concurrency interact. Respect target rules even when a higher request rate is technically possible.

Configure rotating and sticky profiles

Keep two connection profiles outside the code:

• Rotating profile: omit sessid so independent requests can use rotating routes.
• Sticky profile: include a documented alphanumeric sessid. Add sesstime only when the stateful sequence needs an explicit duration instead of the documented default.

The Scrapy implementation remains the same. The environment variable points to the profile appropriate for that spider or job. If different requests inside one spider need different modes, assign the appropriate profile URL per request rather than mutating one shared string.

Handle retries without corrupting state

• Rotating request: a retry may use another route. Revalidate the full response because route and regional context may change.
• Sticky sequence: keep the same session profile for retryable steps. If route continuity is a hard requirement, detect a changed route and restart or quarantine the sequence.
• Policy response: do not keep retrying a response that indicates the request is not allowed or the data is unavailable by design.
• Parse failure: a changed selector or incomplete page is not automatically a network failure. Store a sample and fix the parser.

Production QA checklist

• Credentials remain server-side and are redacted from logs.
• The proxy URL scheme works with the project's active Scrapy download handler.
• Rotation and sticky behavior are tested separately.
• Retries are capped and classified by reason.
• Concurrency and delay are set per target, not copied blindly.
• Response validation distinguishes transport success from usable data.
• Requested geo parameters and observed page context are both recorded.
• Collection follows applicable terms, privacy duties, and law.

When this setup is the right fit

Use this approach when Scrapy is already the crawler and you need a standard residential proxy layer with configurable rotation, sticky sessions, and location parameters. If the workflow requires browser execution, JavaScript rendering, or a third-party download handler, validate that component's proxy support separately; this guide does not imply a native integration or guaranteed compatibility with every handler.

Review Magnetic Proxy's Web Scraping Capsule and connection documentation, then test the exact profile against a small authorized Scrapy job before scaling.