What is a web proxy? How it works step by step

You point your browser at a web proxy on port 8080, browse for an hour, then open the access log expecting a list of pages. Every line reads CONNECT github.com:443 and stops there. No URLs, no page titles, nothing.

Three people will now tell you three different things about what happened, because "web proxy" describes at least three separate pieces of software. One of them rewrites the HTML you receive. One never tousches it. One is sitting in your office router whether you configured it or not.

Guessing which one you're dealing with is how engineers end up with scrapers that leak their real IP, filtering rules that quietly stop working over HTTPS, and a 407 they can't explain. In this guide I'll cover what a web proxy is, what happens byte by byte when you use one, how it differs from a VPN and a reverse proxy, and how to run your own.

What is a web proxy?

A web proxy is a server that makes HTTP requests for you. Your client talks to the proxy, the proxy talks to the target site, and the site logs the proxy's IP address rather than yours. Because every request passes through it, the proxy can cache, filter, log, or block whatever it likes.

The word "proxy" means a stand-in with authority to act for someone else, and that's the whole idea. Your laptop asks for example.com/pricing, and a machine in Frankfurt fetches it and hands you the result.

What matters technically is that the proxy is a full participant, not a wire. It terminates your TCP connection, reads your request, then opens a completely separate connection to the origin server. Two connections, two sets of TLS handshakes, two IP addresses in play.

That last part is worth pinning down, because plenty of explainers describe a proxy as "swapping the source IP in your packets." NAT does that, one packet at a time, at layer 3. A web proxy operates at layer 7 and builds its own request from scratch.

The three things people call a web proxy

Search results use the same phrase for three products that behave nothing alike. Sorting them out takes about ninety seconds and saves a lot of debugging.

What it is	How you use it	What it sees	Where you'll meet it
Proxy website (CGI proxy)	Open a site, paste a URL into a form	Everything, including form posts, since it rewrites the page	Free "unblock any site" pages
HTTP forward proxy	Set host:port in your client, browser, or code	Hostnames and ports over HTTPS; full URLs and bodies over plain HTTP	Scraping stacks, proxy providers, squid on a VPS
Interception proxy	Nothing. Your network already routes you through it	Same as above, unless it also installs a CA on your machine	Corporate networks, school wifi, some ISPs

The proxy website is the odd one out. It fetches the target page on its server, rewrites every link and asset URL inside the HTML to point back at itself, and serves you the result from its own domain. Sites with heavy JavaScript, CSRF tokens, or strict cookie policies tend to break under this treatment, which is why logins so often fail on them.

A forward proxy leaves the page untouched. You keep browsing example.com, the URL bar still says example.com, and the proxy just relays bytes.

Take one scenario with all three in it. Suppose you're on an office network, checking how a competitor's pricing page renders in Germany.

The interception proxy at your office has already logged the fact that you hit that domain. You configure a German forward proxy in your scraper so the site returns euro pricing, and its logs show the German exit IP.

Had you reached for a free proxy website instead, the currency selector would probably load and the "add to cart" step would fail, because the rewritten JavaScript lost its origin.

How does a web proxy work, step by step

Five stages, and you can reproduce every one of them on your own machine in about ten minutes. What follows is the actual wire exchange rather than a description of it.

1. Your client opens a connection to the proxy

Nothing exotic happens first. Your client makes a TCP connection to the proxy's IP and port and waits. At this point the target site has no idea you exist.

What the client does next depends entirely on whether you asked for http:// or https://.

How it works in practice: point curl at a proxy and watch it choose. The -v flag prints the request line it sends, which is the fastest way to confirm a proxy is being used at all.

# Plain HTTP through a proxy on port 3128
curl -v -x http://203.0.113.10:3128 http://example.com/

# Same proxy, HTTPS target
curl -v -x http://203.0.113.10:3128 https://example.com/

Run both and compare the output. The first shows GET http://example.com/ HTTP/1.1. The second shows CONNECT example.com:443, and everything after the handshake is opaque.

2. Plain HTTP: the absolute-form request

For an http:// URL, your client sends the whole URL on the request line. That format is called absolute-form, and it exists so the proxy knows which server to contact.

GET http://example.com/products?page=2 HTTP/1.1
Host: example.com
User-Agent: curl/8.4.0
Proxy-Connection: keep-alive

Compare that to a direct request, where the line would read GET /products?page=2 HTTP/1.1. The proxy strips the scheme and host, opens its own connection to example.com:80, and forwards the rest.

How it works in practice: the proxy operator can read that URL, the query string, the cookies, and the response body. On plain HTTP there's no privacy from the machine in the middle. Treat any http:// request through a proxy you don't own as public.

3. HTTPS: the CONNECT tunnel

For https://, your client asks the proxy to open a raw TCP tunnel and then stay out of the way. The method is CONNECT, and the target is a bare host:port pair rather than a path.

CONNECT example.com:443 HTTP/1.1
Host: example.com:443
Proxy-Authorization: Basic dXNlcjpwYXNz

HTTP/1.1 200 Connection established

After that 200, the proxy forwards bytes in both directions without interpreting them. Your TLS handshake happens with example.com directly, through the tunnel. Any 2xx response puts the proxy into tunnel mode, and the port in that request target is mandatory, which is why a pasted proxy string missing its port fails before the site is ever contacted.

How it works in practice: this is why your proxy log showed only CONNECT lines. The proxy sees the hostname, the port, the timing, and the byte counts. URLs, headers, and page content stay inside TLS. For details on the tunnel semantics, MDN's CONNECT reference is the clearest short writeup.

4. Authentication: 407 and Proxy-Authorization

Commercial proxies gate access either by IP allowlist or by username and password. Password auth uses a status code most people meet once and misremember forever.

A 407 Proxy Authentication Required comes from the proxy. A 401 comes from the site. Mixing them up sends people debugging the wrong machine for an hour.

How it works in practice: in Python, credentials go inline in the proxy URL. Note that both map keys point at an http:// proxy address, because the scheme describes how you reach the proxy, not what you're fetching.

import requests

proxies = {
    "http": "http://user:pass@203.0.113.10:3128",
    "https": "http://user:pass@203.0.113.10:3128",  # still http:// here
}

r = requests.get("https://example.com", proxies=proxies, timeout=10)
print(r.status_code)

Getting that second line wrong produces confusing TLS errors rather than a clean failure. Keep credentials in environment variables in anything you commit, since a proxy URL with a password in it ends up in shell history, crash dumps, and log lines.

5. The headers the proxy adds on the way out

On plain HTTP, a proxy can append headers describing where the request came from. X-Forwarded-For carries the original client IP. Via announces that an intermediary handled the message. RFC 9110 defines both, and points out that at the protocol level a network middlebox looks identical to an on-path attacker.

The industry sorts proxies into three tiers based on this behaviour. It's a convention rather than a standard, so verify rather than trust a vendor's label.

Tier	Sends X-Forwarded-For	Sends Via	Site can tell it's a proxy
Transparent	Yes, with your IP	Yes	Yes, and it learns your IP
Anonymous	No	Yes	Yes
Elite / high-anonymity	No	No	Not from headers alone

How it works in practice: don't take anyone's word for the tier. Run a ten-line server that echoes whatever arrives, then hit it through the proxy.

from http.server import BaseHTTPRequestHandler, HTTPServer

class Echo(BaseHTTPRequestHandler):
    def do_GET(self):
        body = f"{self.requestline}\n{self.headers}".encode()
        self.send_response(200)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

HTTPServer(("0.0.0.0", 8000), Echo).serve_forever()

Put that on a box with a public IP, then run curl -x http://your-proxy:3128 http://your-box:8000/ and read what comes back. Anything the proxy added about you is right there in the output. This only works over plain HTTP, which is the point: on HTTPS the proxy can't inject headers into a tunnel it can't read.

Web proxy vs VPN vs reverse proxy

These three get shelved together because all of them sit between two parties. They solve different problems and sit at different ends of the connection.

	Web proxy (forward)	VPN	Reverse proxy
Protects	The client	The client's whole device	The server
Scope	One app, browser, or script	Every connection the OS makes	All inbound traffic to a service
Encryption	Only what you already had	Encrypted tunnel to the VPN server	Usually terminates TLS for the origin
Configured by	You, per application	You, once per device	The site owner
Example	A scraper routing through 40 exit IPs	Company laptop on hotel wifi	Nginx in front of four app servers

A VPN moves your entire device. A web proxy moves one program, which is often what you want: route the scraper through Brazil while your email client stays on your home connection.

A reverse proxy is the same mechanism pointed the other way. It sits in front of web servers and defends the service being delivered, where a forward proxy or VPN defends the user. You've used thousands of them without knowing, because nearly every large site puts one in front of its application servers.

If you're choosing between them for privacy work, the honest summary is that a VPN gives you coverage and a proxy gives you control. Neither one hides you from the operator of the box you're routing through.

Types of web proxy you'll run into

Beyond the forward/reverse split, two axes matter day to day.

By protocol. An HTTP proxy understands HTTP and can cache and filter it. A SOCKS5 proxy relays TCP (and UDP) without caring what's inside, so it handles SSH, game traffic, and torrents that an HTTP proxy would reject. If your traffic isn't HTTP, SOCKS5 is the one you want.

By where the IP comes from. This determines how a target site treats you, and it's the axis that actually decides whether your requests get through:

• Datacenter IPs are fast and cheap, and trivially identifiable by ASN lookup
• ISP IPs are hosted in datacenters but registered to consumer ISPs, giving datacenter speed with a residential-looking registration
• Residential IPs come from real consumer connections, which is why they survive checks that kill datacenter ranges
• Mobile IPs sit behind carrier-grade NAT, so thousands of real users share them and blocking one is expensive for the site

Providers like Floppyblock sell all four, and the choice matters more than any header tweak. For the tradeoffs in detail, see our breakdown of datacenter vs residential proxies.

How to set up your own web proxy

You can rent exit IPs, but the proxy software itself is free and has been for decades. Running your own is the fastest way to understand the mechanism, and it's genuinely useful for debugging.

1. Start with two IPs and a retry loop. Before building anything, confirm you need a proxy at all. A single VPS in the right country plus curl -x answers most geo questions in ten minutes. Rotation frameworks, session pools, and health checkers can wait until something breaks.

2. Install Squid and open exactly one ACL. Squid has run production proxy fleets since 1996 and takes about six lines of config to get going. The forwarded_for delete and via off directives strip the headers that would otherwise advertise the hop.

http_port 3128
acl trusted src 203.0.113.25/32   # your machine, nothing else
http_access allow trusted
http_access deny all
forwarded_for delete
via off

Leaving that ACL open to the internet is how you end up relaying spam within a day. Restrict CONNECT to port 443 too, since an open proxy that tunnels to any port becomes a port scanner for whoever finds it. The Squid configuration reference documents every directive.

3. Verify from the outside. Hit your echo server from step 5 above and read the headers. If you see your own IP in X-Forwarded-For, the config didn't take.

4. Add rotation only when a target starts refusing you. One IP works until it doesn't. When that day comes, the logic lives in your client, not the proxy, and the full build is covered in our guide to using proxies in Python requests.

Web proxies in practice

Corporate egress control

The proxy is the single door through which employee traffic leaves the building. Security teams use it to block categories of site, log which domains were contacted, and keep internal IPs off external server logs.

The IP in the destination's logs belongs to the proxy, so an attacker probing those logs never learns the corporate router's outgoing address. The catch is HTTPS: through a CONNECT tunnel the proxy sees a hostname and nothing more, so content filtering means installing a corporate CA on every device and terminating TLS at the proxy. Most security teams argue about that one for months.

Scraping and geo-verification

Here the proxy exists to change what the target sees. You want a German IP for German prices, or 200 different IPs so one scraper doesn't look like one very enthusiastic visitor.

The rotation logic belongs in your code: pick an exit, use it for a session, retire it on a 403 or a challenge page. That pattern is the whole subject of how rotating proxies work, and the mechanism is the same one described above with a different IP on each CONNECT.

Debugging your own traffic

Point a mobile app or a background service at a local proxy and you get a readable log of every call it makes. This is how you find the analytics SDK phoning home, or the retry storm your client library kicks off after a timeout.

For HTTPS you'll need a tool that MITMs with its own certificate, which is intrusive by design. Use it on devices you own and remove the CA afterwards.

Where web proxies fall short

Encrypt end to end anyway. Over plain HTTP, the proxy operator reads your URLs, cookies, and bodies in full. Free proxy lists exist because that data has value; some of them inject ads into responses, and some collect credentials. If you didn't set up the proxy and you're not on HTTPS, assume someone is reading.

Route DNS through the proxy too. With an HTTP proxy, the proxy resolves the hostname for you, so nothing leaks. With SOCKS5 it depends on the scheme you write: socks5:// resolves locally and tells your ISP exactly which sites you're visiting, while socks5h:// resolves at the proxy. One character, completely different threat model.

Configure the app, not just the OS. Setting a system proxy misses anything that reads HTTP_PROXY on its own terms, ignores it entirely, or uses WebRTC. Browsers in particular can expose your local and public IP through WebRTC while the proxy is working perfectly.

A web proxy doesn't defeat bot detection. This is the one people learn expensively. A fresh residential IP still carries your TLS fingerprint, your header order, your canvas hash, and your mouse movement (or lack of it). Anti-bot systems weigh all of those, so swapping IPs solves rate limiting and geo-blocking while doing nothing about fingerprinting. If a site blocks you after two requests from a clean IP, the IP was never the problem.

FAQ

Is a web proxy the same as a VPN?

No. A VPN encrypts all traffic leaving your device and routes it through one tunnel, while a web proxy handles traffic for a single app or script and adds no encryption of its own. A proxy gives you per-application control; a VPN gives you device-wide coverage.

Can a web proxy see my passwords?

Over plain HTTP, yes, along with everything else in the request. Over HTTPS the proxy only sees the hostname and port from the CONNECT request, unless it's an interception proxy whose certificate authority is installed on your machine, in which case it can read everything again.

Are free web proxies safe to use?

I'd skip them. Running a public proxy costs money and returns nothing, so the operator is monetising something: ad injection, traffic logs, or credentials. For a one-off look at a blocked page the risk is low; for anything involving a login it isn't worth it.

Does a web proxy hide my IP from the website?

It hides it from the website, which logs the proxy's IP instead. It doesn't hide anything from the proxy operator, who sees both ends of the connection. Transparent proxies also forward your real IP in an X-Forwarded-For header, so test before you assume.

Do I need a web proxy for web scraping?

For anything beyond a few hundred requests, yes. Sites rate-limit by IP, so a single address hits a wall quickly no matter how polite your delays are. Start with one proxy and add rotation when a target starts returning 403s.

Wrapping up

The mental model to keep: a web proxy ends your connection and starts a new one, which is why it can cache and filter plain HTTP, and why it goes blind the moment you use CONNECT. Everything else about proxies follows from those two facts.

Go run the echo server against a proxy you're using right now. Five minutes of reading real headers teaches more than any diagram, including mine. When you're ready to put it to work, the Python requests proxy guide picks up where this leaves off.