Proxy Logs: What to Record and What to Redact

Useful proxy logs connect a request to its route configuration, timing and outcome without exposing credentials or unnecessary response data. Record a small set of structured events in your own application, correlate attempts with the business task, and redact sensitive values before those events reach a shared logging system.

The goal is to answer what happened and where to investigate next. A complete response dump often creates more noise and exposure than evidence. Start with the questions a support engineer or job owner actually needs to resolve.

Separate task, request and attempt

A collection task can contain many requests, and a request can have several attempts. Give each level a stable internal reference. Keep the same request reference when an attempt is retried, and assign a new attempt number so a reviewer can reconstruct the sequence.

For example, a catalog job may request one product page twice before obtaining a usable record. That is one intended page observation, two network attempts and one accepted result. Counting all three as interchangeable successes hides retry cost and makes a failure report difficult to follow.

Choose the minimum useful event fields

The following is a recommended application record, not a Magnetic Proxy API schema or a claim that its dashboard exports these exact fields:

• Identity: internal task reference, request reference and attempt number.
• Time: start time with time zone, duration and the relevant deadline.
• Route configuration: protocol, non-secret configuration reference, requested region and session mode.
• Destination: hostname and, only when necessary, a sanitized path or route category.
• Outcome: client error category, HTTP status if one was actually received, bytes received and whether the response passed the application's content checks.
• Next action: accepted result, scheduled retry, review or stopped task, with a short reason.

Keep requested route settings separate from observed exit information. If an approved test collects exit-location evidence, label its source and time. Do not assume that a configured country proves how a destination classified the connection.

Redact before logs leave the client

Proxy URLs can contain usernames and passwords. Request URLs can contain tokens in query parameters, and headers can carry authorization or cookies. Exclude those values from ordinary diagnostics. An exception handler that prints the entire request can undo careful redaction in the normal success path.

Use a non-secret configuration reference that lets an authorized operator find the settings in the appropriate system. Avoid treating a masked username as harmless when it still reveals account or session details. Do not store page bodies by default; capture a restricted sample only when a defined investigation requires it.

OWASP's logging guidance recommends excluding secrets, controlling access and handling untrusted event data safely. Apply those principles to your logging destination and error paths. A redaction rule needs tests with realistic field shapes, not just a search for the word password.

Keep the evidence layers distinct

A request can receive HTTP 200 and still contain a login page, a consent prompt or incomplete content. Record transport outcome separately from the check that decides whether the result is usable. A missing price field is an extraction or content observation; it should not automatically be labeled a proxy connection failure.

Likewise, record a timeout as the client observed it, including the phase when known. Do not invent an HTTP status for an attempt that received no response. Use the proxy timeout guide to investigate the failure and the 407 guide for authentication responses.

Example: explain a retry without duplicating the result

Consider an illustrative request whose first attempt times out and whose second attempt returns usable content. Retain both attempt events under one request reference. Record the retry decision and delay, then attach the accepted result to the intended task once.

Now suppose the second attempt returns HTTP 200 but the required field is missing. Keep that as a separate content-check failure. The log should let a reviewer distinguish network reliability from the application's acceptance rule, without needing a complete page dump.

Make diagnostic access and retention deliberate

Decide who can view ordinary events, who can retrieve restricted samples and how long each category is needed. Give support a concise, sanitized bundle rather than unrestricted production logs. Choose retention for the operational purpose and your organization's requirements; this guide does not prescribe a universal period or establish legal compliance.

Check that logging failures do not expose secrets through a fallback console dump. Also test whether an unavailable logging destination slows or blocks the workload unexpectedly. Define how the application reports missing diagnostics so an empty log is not mistaken for a healthy run.

Validate the record before scaling

• Can you follow one task through its requests and attempts?
• Are timeouts, HTTP responses and content checks distinct?
• Are credentials, cookies and token-bearing URLs absent from both success and error events?
• Does a retried request produce only one accepted business result?
• Can an authorized reviewer find the configuration without receiving its secrets?

Use Magnetic Proxy's current documentation for connection parameters and the Web Scraping Capsule to evaluate the routing needs of a permitted collection workload. Your application owns the diagnostic event design; proxy configuration and observability should support the same investigation without becoming the same system.