Proxy Concurrency: Set Limits for Reliable Collection

Proxy concurrency is the number of requests in flight at the same time through your proxy setup. Set it separately from request rate, choose the smallest useful level for your workload, and increase it only while usable results improve without unacceptable errors or latency.

A rotating residential route does not remove a target's traffic limits. Concurrency belongs in the client or job scheduler as an explicit control; changing IPs is not a substitute for it.

Concurrency, request rate and workers are different

Concurrency counts active requests. Request rate counts requests started over a time interval. Worker count counts the processes or tasks able to perform work. A worker may issue several requests, wait on processing, or share connections with other workers, so worker count alone is not a reliable traffic limit.

Consider an illustrative workload with four active requests and an average response time of two seconds. Under steady conditions, that suggests about two completed requests per second. It is arithmetic for sizing a test, not a Magnetic Proxy throughput claim. If responses become slower, the same concurrency yields fewer completions; if responses become faster, it can generate a higher request rate.

Use both an in-flight cap and a rate limiter when the destination requires a start-rate limit. A connection-pool limit adds another constraint: a job can be queued waiting for a connection even when your worker appears busy. Measure that wait separately from time spent fetching the response.

Set limits at the scope that creates the load

• Per destination: share a limit across jobs reaching the same service. Separate URLs or separate proxy IPs do not make the destination's capacity unlimited.
• Per sequence: preserve ordering for work that depends on the previous response. Pagination with session state may need sequential steps even when unrelated tasks run in parallel.
• Across the application: keep a total limit so a growing queue does not exhaust connections, memory or downstream processing.

If several workers run on different machines, a limit that exists only inside one process can multiply accidentally. Decide where the shared count lives and how capacity is released after a crash or deadline. Do not hardcode a provider-wide “safe” concurrency number: confirm your current account terms and test the actual destination and workload.

A controlled test for choosing a limit

1. Define a usable result. Specify required fields, expected page context and checks that distinguish a complete result from an error page with HTTP 200.
2. Choose a small representative workload. Hold destination, response size, region and session mode steady. Run only requests you are permitted to make.
3. Start conservatively. Record active requests, queue wait, response duration, result quality and retries for the initial setting.
4. Change one setting. Increase the in-flight cap by a small deliberate step while holding the request-rate policy and other parameters constant.
5. Stop at the first clear tradeoff. If usable completions stop improving, tail latency grows, or errors increase, return to the preceding stable setting and investigate the constraint.
6. Repeat on a new representative sample. Keep the comparison conditions documented. A short successful burst does not establish a sustained operating limit.

Measure completed business tasks as well as individual HTTP attempts. A crawl that doubles its requests because it retries everything may look busier while producing fewer useful records.

Make retries share the same budget

Every retry consumes request capacity. Send it through the same queue, concurrency cap and rate policy as the original attempt. Creating an unrestricted retry path defeats the controls precisely when the destination or network is struggling.

Classify the response before retrying. A transient failure may justify a bounded retry with backoff; an explicit access refusal, a validation error or a completed negative finding needs a different decision. Respect a destination's documented retry instructions. Record the attempt count and stop after your defined deadline or retry budget.

When a request times out, release its capacity reliably. For actions that can change state, determine whether the first attempt may already have completed before repeating it. The proxy timeout guide covers failure diagnosis separately from capacity tuning.

Keep session routing separate from scheduling

Magnetic Proxy documents a residential gateway with rotating behavior when a session identifier is omitted and sticky routing through documented session parameters. Use the current connection documentation for the exact configuration.

A sticky session addresses route continuity. It does not decide how many requests your application should run at once. Avoid parallelizing dependent steps simply because they share a session. For independent tasks, let the scheduler control concurrency and the connection configuration control routing. The rotating proxy guide explains that distinction in more depth.

Diagnose a plateau before adding workers

• Queue wait grows while requests remain healthy: check whether the queue or connection pool is the intended constraint.
• Response duration grows: inspect the destination, network and selected route conditions before raising load.
• Retries rise: group failures by reason and check whether retry traffic is consuming the useful-work budget.
• Responses arrive but records are missing: inspect parsing and downstream processing. More proxy requests cannot repair an extraction bug.

Track transfer volume alongside usable completions when response sizes vary. Concurrency and bandwidth are separate controls; the proxy bandwidth guide explains how to account for transferred data.

Before you scale the next run

Write down the per-target rate, total in-flight cap, retry budget, timeout, session boundary and definition of a usable result. Keep the tested configuration with its workload and observation window, and review it when the target, geography or response size changes.

For sustained permitted collection, compare the Web Scraping Capsule with the requirements of your workload. Confirm account capacity and routing needs, then validate a small run before expanding it.