Proxy Bandwidth: Estimate Usage and Control Data Transfer

Proxy bandwidth is the data transferred through a proxy connection. For planning, estimate the bytes used by a complete task, multiply by task volume, and compare that estimate with the provider's usage meter. Request counts alone cannot tell you how much data a job will consume.

The most useful unit is often data per usable result: one checked page, one completed workflow, or one accepted record. That captures the extra requests and retries needed to finish the work.

Separate three different measurements

• Application payload: the content your code reads or saves, such as an HTML document or JSON response.
• Transferred data: traffic associated with the requests, including resources and protocol details captured by your measurement method.
• Billable usage: the amount recorded under the provider's billing rules.

These numbers can differ. A downloaded compressed response and its decoded text do not have the same size. A browser page may load images, scripts, fonts, and later requests that a single HTML fetch never requests. A provider may also define accounting differently from your application instrumentation.

Check the current plan terms and usage reporting before treating a local byte count as a bill. This guide does not assume that Magnetic Proxy bills every category of traffic described here in a particular way.

Build a first estimate

For a reasonably uniform job, start with tasks × requests per task × average attempts per request × average transferred bytes per attempt. Divide by 1,000,000,000 to express decimal gigabytes. A gibibyte, or GiB, uses 1,073,741,824 bytes; use the same unit throughout your comparison.

For example, suppose a hypothetical test has 10,000 tasks, three requests per task, 1.2 attempts per request, and 200,000 bytes per attempt. The estimate is 7,200,000,000 bytes, or 7.2 GB. These are illustrative inputs, not measured Magnetic Proxy usage or a recommended allowance.

The average-attempts factor includes the first attempt. Do not add a retry allowance twice. If failed attempts transfer very different amounts from successful ones, calculate them separately instead of multiplying everything by one average.

For mixed workloads, estimate each segment independently. A lightweight API response, an image-heavy page, and a browser session should not share one assumed payload size. Sum the segment estimates after documenting their units.

Measure a representative pilot

1. Define completion. Decide what counts as one usable result and which output checks it must pass.
2. Choose representative tasks. Include normal response sizes, locations, pagination depth, and session behavior.
3. Record a starting meter reading. Isolate the pilot from unrelated activity where possible. Note the account, time window, and reporting delay.
4. Run with bounded retries. Log task counts, attempts, results, and available transfer measurements without storing credentials or sensitive response content.
5. Reconcile the ending reading. Allow for the meter's reporting delay, then compare the usage change with your local estimate.
6. Calculate data per accepted result. Divide the measured usage by the number of outputs that passed the checks, and document excluded work.

If other jobs share the same meter, the difference is not a clean measurement of your pilot. Either isolate the test or label that uncertainty. Repeat when the workload changes materially rather than assuming one sample will always apply.

Find avoidable transfer before reducing coverage

Inspect retries and repeated work

Group failures by cause. Repeatedly retrying invalid credentials or an incorrect endpoint spends effort without fixing the configuration. Use a retry budget, record the final outcome, and investigate persistent errors before increasing concurrency.

Check whether a scheduler, timeout, or queue redelivery is executing the same task twice. Keep a stable task identifier so completed work can be recognized before another full fetch.

Request only what the task needs

Where an approved interface supports selecting fields or limiting records, request the required subset. Cache results when freshness requirements and access terms allow it. A shorter refresh interval should have a reason tied to the use case.

For browser workflows, inspect resource loading before blocking assets. Removing an image may be harmless for a text extraction task but invalidate a visual QA task. Blocking scripts can stop the data from appearing at all. Compare output correctness after every optimization.

Keep routing changes separate from data changes

Switching exit IPs does not by itself make responses smaller. Rotation, session continuity, payload size, and billing are different variables. Configure connections using the current Magnetic Proxy documentation and measure the actual workload after changes.

Turn the measurement into an operating budget

Multiply measured data per task by the expected task volume, then add an explicitly documented allowance for variation. Derive that allowance from observed peaks and failure patterns rather than choosing a universal percentage.

Track both total usage and usable results. Rising usage with stable output can indicate larger pages, extra resources, more retries, or duplicate jobs. Lower usage with missing fields can indicate a broken collection process rather than an efficiency gain.

Before selecting capacity, compare your measured workload with the current pricing and plan terms. A short pilot with clear units, isolated usage, and verified output is a stronger basis for budgeting than a request-count estimate alone.