Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 84 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

proxy-chain is a programmable HTTP/HTTPS proxy server for Node.js with support for SSL/TLS, authentication, upstream proxy chaining (HTTP/HTTPS/SOCKS), custom HTTP responses, and traffic statistics. It's used by Apify Proxy and the Crawlee web scraping library.

## Build & Development Commands

```bash
npm run build # Compile TypeScript to dist/
npm run build:watch # Watch mode compilation
npm run lint # ESLint check
npm run lint:fix # ESLint auto-fix
npm run local-proxy # Run local proxy server for testing
```

## Testing

Docker-based testing is recommended due to Linux/macOS socket handling differences:

```bash
npm run test:docker # Run all tests in Docker
npm run test:docker test/server.js # Run specific test file
npm run test:docker -- --grep "direct ipv6" # Run tests matching pattern
```

Local testing requires `/etc/hosts` entry:
```
127.0.0.1 localhost-test
```

```bash
npm run test # Run all tests locally
npm run test test/anonymize_proxy.js # Run specific test file
```

Tests use Mocha with ts-node. Coverage via nyc.

## Architecture

### Request Flow

1. `Server` class (server.ts) receives HTTP requests or CONNECT tunnels
2. User-provided `prepareRequestFunction` determines authentication and routing
3. Request is dispatched to the appropriate handler based on protocol and upstream type

### Handler Modules

- **direct.ts** - Direct CONNECT tunneling to target (no upstream proxy)
- **forward.ts** - HTTP forwarding to target or upstream HTTP/HTTPS proxy
- **chain.ts** - CONNECT tunneling through upstream HTTP/HTTPS proxy
- **chain_socks.ts** - CONNECT tunneling through SOCKS proxy
- **forward_socks.ts** - HTTP forwarding through SOCKS proxy
- **custom_response.ts** - Generate custom HTTP responses without contacting upstream
- **custom_connect.ts** - Route CONNECT requests to custom HTTP server

### Key Source Files

- **server.ts** - Main `Server` class (EventEmitter), handles connection lifecycle
- **statuses.ts** - Custom HTTP status codes 590-599 for proxy-specific errors
- **request_error.ts** - `RequestError` class for custom error responses
- **anonymize_proxy.ts** - Helper to create local anonymous proxy for authenticated upstreams
- **tcp_tunnel_tools.ts** - `createTunnel`/`closeTunnel` for TCP tunneling

### Connection Tracking

Each connection gets a unique ID. Statistics tracked per connection: `srcTxBytes`, `srcRxBytes`, `trgTxBytes`, `trgRxBytes`. Access via `server.getConnectionStats(connectionId)`.

### Server Types

- `serverType: 'http'` (default) - Standard HTTP proxy
- `serverType: 'https'` - HTTPS proxy requiring `httpsOptions: { key, cert }`

## Custom Status Codes

- 590: Upstream non-200 response
- 593: DNS lookup failed
- 594: Connection refused
- 595: Connection reset
- 596: Broken pipe
- 597: Auth failed
- 599: Generic upstream error
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ For details, read [How to make headless Chrome and Puppeteer use a proxy server

The proxy-chain package is developed by [Apify](https://apify.com/), the full-stack web scraping and data extraction platform, to support their [Apify Proxy](https://apify.com/proxy) product,
which provides an easy access to a large pool of datacenter and residential IP addresses all around the world. The proxy-chain package is also used by [Crawlee](https://crawlee.dev/),
the world's most popular web craling library for Node.js.
the world's most popular web crawling library for Node.js.

The proxy-chain package currently supports HTTP/SOCKS forwarding and HTTP CONNECT tunneling to forward arbitrary protocols such as HTTPS or FTP ([learn more](https://blog.apify.com/tunneling-arbitrary-protocols-over-http-proxy-with-static-ip-address-b3a2222191ff)). The HTTP CONNECT tunneling also supports the SOCKS protocol. Also, proxy-chain only supports the Basic [Proxy-Authorization](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization).

Expand Down Expand Up @@ -64,7 +64,7 @@ const server = new ProxyChain.Server({
// * connectionId - Unique ID of the HTTP connection. It can be used to obtain traffic statistics.
prepareRequestFunction: ({ request, username, password, hostname, port, isHttp, connectionId }) => {
return {
// If set to true, the client is sent HTTP 407 resposne with the Proxy-Authenticate header set,
// If set to true, the client is sent HTTP 407 response with the Proxy-Authenticate header set,
// requiring Basic authentication. Here you can verify user credentials.
requestAuthentication: username !== 'bob' || password !== 'TopSecret',

Expand All @@ -78,7 +78,7 @@ const server = new ProxyChain.Server({

// Applies to HTTPS upstream proxy. If set to true, requests made to the proxy will
// ignore certificate errors. Useful when upstream proxy uses self-signed certificate. By default "false".
ignoreUpstreamProxyCertificate: true
ignoreUpstreamProxyCertificate: true,

// If "requestAuthentication" is true, you can use the following property
// to define a custom error message to return to the client instead of the default "Proxy credentials required"
Expand Down Expand Up @@ -131,7 +131,7 @@ const ProxyChain = require('proxy-chain');
// -> listen for 'connection' events to track raw TCP sockets
//
// https:
// -> listen for 'securedConnection' events (instead of 'connection') to track only post-TLS-handshake sockets
// -> listen for 'secureConnection' events (instead of 'connection') to track only post-TLS-handshake sockets
// -> additionally listen for 'tlsError' events to handle TLS handshake errors
//
// Default value is 'http'
Expand Down Expand Up @@ -263,7 +263,7 @@ Upstream responded with non-200 status code.

### `592 Status Code Out Of Range`

Upstream respondend with status code different than 100-999.
Upstream responded with status code different than 100-999.

### `593 Not Found`

Expand Down Expand Up @@ -482,7 +482,7 @@ server.on('tunnelConnectResponded', ({ proxyChainId, response, socket, head, cus
});
```

Alternatively a [helper function](##helper-functions) may be used:
Alternatively a [helper function](#helper-functions) may be used:

```javascript
listenConnectAnonymizedProxy(anonymizedProxyUrl, ({ response, socket, head }) => {
Expand Down
8 changes: 4 additions & 4 deletions src/anonymize_proxy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ export interface AnonymizeProxyOptions {
*/
export const anonymizeProxy = async (
options: string | AnonymizeProxyOptions,
callback?: (error: Error | null) => void,
callback?: (error: Error | null, result?: string) => void,
): Promise<string> => {
let proxyUrl: string;
let port = 0;
Expand All @@ -32,11 +32,11 @@ export const anonymizeProxy = async (
proxyUrl = options;
} else {
proxyUrl = options.url;
port = options.port;
port = options.port ?? 0;

if (port < 0 || port > 65535) {
throw new Error(
'Invalid "port" option: only values equals or between 0-65535 are valid',
'Invalid "port" option: only values between 0-65535 are valid',
);
}

Expand Down Expand Up @@ -88,7 +88,7 @@ export const anonymizeProxy = async (
/**
* Closes anonymous proxy previously started by `anonymizeProxy()`.
* If proxy was not found or was already closed, the function has no effect
* and its result if `false`. Otherwise the result is `true`.
* and its result is `false`. Otherwise the result is `true`.
* @param closeConnections If true, pending proxy connections are forcibly closed.
*/
export const closeAnonymizedProxy = async (
Expand Down
8 changes: 7 additions & 1 deletion src/chain.ts
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,13 @@ export const chain = (
head: clientHead,
});

sourceSocket.write(isPlain ? '' : `HTTP/1.1 200 Connection Established\r\n\r\n`);
try {
sourceSocket.write(isPlain ? '' : `HTTP/1.1 200 Connection Established\r\n\r\n`);
} catch (error) {
sourceSocket.destroy(error as Error);
targetSocket.destroy();
return;
}

sourceSocket.pipe(targetSocket);
targetSocket.pipe(sourceSocket);
Expand Down
17 changes: 14 additions & 3 deletions src/chain_socks.ts
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ export const chainSocks = async ({

const proxy: SocksProxy = {
host: hostname,
port: Number(port),
port: port ? Number(port) : 1080, // Default SOCKS port is 1080
type: socksProtocolToVersionNumber(handlerOpts.upstreamProxyUrlParsed.protocol),
userId: decodeURIComponent(username),
password: decodeURIComponent(password),
Expand All @@ -74,9 +74,14 @@ export const chainSocks = async ({
}

const url = new URL(`connect://${request.url}`);
let host = url.hostname;
// Strip IPv6 brackets if present (e.g., [::1] -> ::1)
if (host[0] === '[') {
host = host.slice(1, -1);
}
const destination = {
port: Number(url.port),
host: url.hostname,
host,
};

let targetSocket: net.Socket;
Expand All @@ -89,7 +94,13 @@ export const chainSocks = async ({
});
targetSocket = client.socket;

sourceSocket.write(`HTTP/1.1 200 Connection Established\r\n\r\n`);
try {
sourceSocket.write(`HTTP/1.1 200 Connection Established\r\n\r\n`);
} catch (writeError) {
sourceSocket.destroy(writeError as Error);
targetSocket.destroy();
return;
}
} catch (error) {
const socksError = error as SocksClientError;
server.log(proxyChainId, `Failed to connect to upstream SOCKS proxy ${socksError.stack}`);
Expand Down
2 changes: 1 addition & 1 deletion src/custom_response.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ export const handleCustomResponse = async (
throw new Error('The user-provided "customResponseFunction" must return an object.');
}

response.statusCode = customResponse.statusCode || 200;
response.statusCode = customResponse.statusCode ?? 200;

if (customResponse.headers) {
for (const [key, value] of Object.entries(customResponse.headers)) {
Expand Down
17 changes: 15 additions & 2 deletions src/direct.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import net from 'node:net';
import { URL } from 'node:url';

import type { Socket } from './socket';
import { badGatewayStatusCodes, createCustomStatusHttpResponse, errorCodeToStatusCode } from './statuses';
import { countTargetBytes } from './utils/count_target_bytes';

export interface HandlerOpts {
Expand Down Expand Up @@ -67,6 +68,7 @@ export const direct = (
sourceSocket.write(`HTTP/1.1 200 Connection Established\r\n\r\n`);
} catch (error) {
sourceSocket.destroy(error as Error);
targetSocket.destroy();
}
});

Expand Down Expand Up @@ -96,11 +98,22 @@ export const direct = (
});

const { proxyChainId } = sourceSocket;
let connected = false;

targetSocket.on('error', (error) => {
targetSocket.once('connect', () => {
connected = true;
});

targetSocket.on('error', (error: NodeJS.ErrnoException) => {
server.log(proxyChainId, `Direct Destination Socket Error: ${error.stack}`);

sourceSocket.destroy();
// If we haven't connected yet, send an error response to the client
if (!connected && sourceSocket.writable) {
const statusCode = errorCodeToStatusCode[error.code!] ?? badGatewayStatusCodes.GENERIC_ERROR;
sourceSocket.end(createCustomStatusHttpResponse(statusCode, error.code ?? 'Connection Failed'));
} else {
sourceSocket.destroy();
}
});

sourceSocket.on('error', (error) => {
Expand Down
4 changes: 2 additions & 2 deletions src/forward.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ export interface HandlerOpts {

/**
* The request is read from the client and is resent.
* This is similar to Direct / Chain, however it uses the CONNECT protocol instead.
* Forward uses standard HTTP methods.
* Unlike Direct / Chain which use the CONNECT protocol for tunneling,
* Forward uses standard HTTP methods (GET, POST, etc.).
*
* ```
* Client -> Apify (HTTP) -> Web
Expand Down
2 changes: 1 addition & 1 deletion src/forward_socks.ts
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ export const forwardSocks = async (
agent,
};

// Only handling "http" here - since everything else is handeled by tunnelSocks.
// Only handling "http" here - since everything else is handled by chainSocks.
// We have to force cast `options` because @types/node doesn't support an array.
const client = http.request(request.url!, options as unknown as http.ClientRequestArgs, async (clientResponse) => {
try {
Expand Down
12 changes: 7 additions & 5 deletions src/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -130,10 +130,11 @@ export type ServerOptions = HttpServerOptions | HttpsServerOptions;

/**
* Represents the proxy server.
* It emits the 'requestFailed' event on unexpected request errors, with the following parameter `{ error, request }`.
* It emits the 'requestFailed' event on unexpected request errors, with parameter `{ error, request }`.
* It emits the 'connectionClosed' event when connection to proxy server is closed, with parameter `{ connectionId, stats }`.
* It emits the 'tlsError' event on TLS handshake failures (HTTPS servers only), with parameter `{ error, socket }`.
* with parameter `{ connectionId, reason, hasParent, parentType }`.
* It emits the 'tunnelConnectResponded' event on successful CONNECT tunnel establishment, with parameter `{ proxyChainId, response, customTag, socket, head }`.
* It emits the 'tunnelConnectFailed' event when upstream proxy rejects CONNECT request, with parameter `{ proxyChainId, response, customTag, socket, head }`.
*/
export class Server extends EventEmitter {
port: number;
Expand Down Expand Up @@ -474,7 +475,7 @@ export class Server extends EventEmitter {
throw new RequestError(`Target "${request.url}" could not be parsed`, 400);
}

// Only HTTP is supported, other protocols such as HTTP or FTP must use the CONNECT method
// Only HTTP is supported, other protocols such as HTTPS or FTP must use the CONNECT method
if (parsed.protocol !== 'http:') {
throw new RequestError(`Only HTTP protocol is supported (was ${parsed.protocol})`, 400);
}
Expand Down Expand Up @@ -558,7 +559,7 @@ export class Server extends EventEmitter {
try {
handlerOpts.upstreamProxyUrlParsed = new URL(funcResult.upstreamProxyUrl);
} catch (error) {
throw new Error(`Invalid "upstreamProxyUrl" provided: ${error} (was "${funcResult.upstreamProxyUrl}"`);
throw new Error(`Invalid "upstreamProxyUrl" provided: ${error} (was "${funcResult.upstreamProxyUrl}")`);
}

if (!['http:', 'https:', ...SOCKS_PROTOCOLS].includes(handlerOpts.upstreamProxyUrlParsed.protocol)) {
Expand Down Expand Up @@ -740,11 +741,12 @@ export class Server extends EventEmitter {
closeConnections(): void {
this.log(null, 'Closing pending sockets');

const count = this.connections.size;
for (const socket of this.connections.values()) {
socket.destroy();
}

this.log(null, `Destroyed ${this.connections.size} pending sockets`);
this.log(null, `Destroyed ${count} pending sockets`);
}

/**
Expand Down
2 changes: 1 addition & 1 deletion src/statuses.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ export const badGatewayStatusCodes = {
*/
NON_200: 590,
/**
* Upstream respondend with status code different than 100-999.
* Upstream responded with status code different than 100-999.
*/
STATUS_CODE_OUT_OF_RANGE: 592,
/**
Expand Down
8 changes: 7 additions & 1 deletion src/tcp_tunnel_tools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,13 @@ import net from 'node:net';
import { URL } from 'node:url';

import { chain } from './chain';
import type { Socket } from './socket';
import { nodeify } from './utils/nodeify';

const runningServers: Record<string, { server: net.Server, connections: Set<net.Socket> }> = {};

let lastConnectionId = 0;

const getAddress = (server: net.Server) => {
const { address: host, port, family } = server.address() as net.AddressInfo;

Expand Down Expand Up @@ -51,9 +54,12 @@ export async function createTunnel(

server.log = log;

server.on('connection', (sourceSocket) => {
server.on('connection', (sourceSocket: Socket) => {
const remoteAddress = `${sourceSocket.remoteAddress}:${sourceSocket.remotePort}`;

// Assign a unique ID for logging purposes (similar to Server.registerConnection)
sourceSocket.proxyChainId = lastConnectionId++;

const { connections } = runningServers[getAddress(server)];

log(`new client connection from ${remoteAddress}`);
Expand Down
2 changes: 1 addition & 1 deletion src/utils/is_hop_by_hop_header.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// As per HTTP specification, hop-by-hop headers should be consumed but the proxy, and not forwarded
// As per HTTP specification, hop-by-hop headers should be consumed by the proxy, and not forwarded
const hopByHopHeaders = [
'connection',
'keep-alive',
Expand Down
2 changes: 1 addition & 1 deletion src/utils/parse_authorization_header.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import { Buffer } from 'node:buffer';

const splitAt = (string: string, index: number) => {
return [
index === -1 ? '' : string.substring(0, index),
index === -1 ? string : string.substring(0, index),
index === -1 ? '' : string.substring(index + 1),
];
};
Expand Down
Loading
Loading