httpmirror

package module
v0.11.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 2, 2026 License: MIT Imports: 24 Imported by: 0

README

httpmirror

An HTTP reverse proxy and caching server that mirrors remote files with optional cloud storage backend support.

Overview

httpmirror is a lightweight HTTP proxy server designed to cache and serve remote files efficiently. It acts as a caching layer between clients and remote HTTP servers, optionally storing cached content in cloud storage backends (S3, MinIO, etc.) for persistent and distributed caching.

Features

  • HTTP Reverse Proxy: Proxies HTTP GET and HEAD requests to remote servers
  • Intelligent Caching: Caches remote files to reduce bandwidth and latency
  • Cloud Storage Backend: Support for S3-compatible storage backends (via SSS library)
  • CIDN Integration: Optional integration with Content Infrastructure Delivery Network (CIDN) for distributed blob management
  • Configurable Link Expiry: Set custom expiration times for signed URLs
  • Health Checking: Optional sync timeout to verify cached content freshness
  • Flexible Host Mapping: Support for host-from-first-path routing
  • Retry Mechanism: Configurable retry logic for failed requests
  • Suffix Blocking: Block requests for specific file suffixes

Documentation

Index

Constants

This section is empty.

Variables

View Source
var ErrNotOK = fmt.Errorf("http status not ok")

ErrNotOK is returned when an HTTP response status is not 200 OK.

Functions

This section is empty.

Types

type Logger

type Logger interface {
	// Println logs a message with the provided arguments.
	Println(v ...interface{})
}

Logger provides a simple logging interface for the mirror handler.

type MirrorHandler

type MirrorHandler struct {
	// RemoteCache is the cache of the remote file system.
	// When set, files are cached in the storage backend and clients
	// are redirected to signed URLs. When nil, requests are proxied directly.
	RemoteCache *sss.SSS

	// LinkExpires is the expiration duration for signed URLs.
	// Only used when RemoteCache is configured.
	// Default should be 24 hours.
	LinkExpires time.Duration

	// BaseDomain is the domain name suffix to filter requests.
	// If set, only requests to hosts ending with this suffix are processed.
	// Leave empty to process all valid domain requests.
	BaseDomain string

	// Client is the HTTP client used for requests to source servers.
	// If nil, a default client with ProxyDial will be created.
	Client *http.Client

	// ProxyDial specifies the optional proxy dial function for
	// establishing transport connections to source servers.
	ProxyDial func(context.Context, string, string) (net.Conn, error)

	// NotFound is the handler for requests that don't match any proxy rules.
	// If nil, http.NotFound is used.
	NotFound http.Handler

	// Logger is used for error and informational logging.
	// If nil, no logging is performed.
	Logger Logger

	// CheckSyncTimeout is the timeout for checking if cached content
	// is synchronized with the source. When > 0, the handler verifies
	// that cached files match the source size before serving.
	// Set to 0 to disable sync checking.
	CheckSyncTimeout time.Duration

	// Host is the target host for all requests.
	Host string

	// HostFromFirstPath enables extracting the target host from the first
	// path segment instead of the Host header.
	// When true, URLs like /example.com/path/file become requests to
	// https://example.com/path/file
	HostFromFirstPath bool

	// BlockSuffix is a list of file suffixes to block.
	// Requests for files ending with these suffixes return 403 Forbidden.
	// Example: []string{".exe", ".msi"}
	BlockSuffix []string

	// NoRedirect disables HTTP redirects to signed URLs for cached content.
	// When true, the handler serves cached content directly instead of
	// redirecting clients to signed URLs from RemoteCache.
	// This is useful for clients that don't handle redirects well or when
	// you want the proxy to serve all traffic directly.
	NoRedirect bool

	// TeeResponse is used to tee the response body for caching while serving.
	// When true, the handler will write the response to a tee while
	// caching it in RemoteCache. This allows for streaming responses to clients
	// while simultaneously caching them.
	TeeResponse bool

	// CIDNClient is the Kubernetes client for CIDN integration.
	// When set along with RemoteCache, enables distributed blob management.
	CIDNClient versioned.Interface

	// CIDNBlobInformer watches for CIDN Blob resource changes.
	// Used to monitor blob sync status when CIDN is enabled.
	CIDNBlobInformer informers.BlobInformer

	// CIDNDestination is the destination name for CIDN blobs.
	// Typically set to the storage backend scheme (e.g., "s3").
	CIDNDestination string

	// CIDNMaximumRunning is the maximum number of running CIDN blob sync operations.
	CIDNMaximumRunning int64

	// CIDNMinimumChunkSize is the minimum chunk size for CIDN blob sync operations.
	CIDNMinimumChunkSize int64
	// contains filtered or unexported fields
}

MirrorHandler is the main HTTP handler that processes requests and manages caching.

It acts as a reverse proxy, optionally caching responses in a remote storage backend. When RemoteCache is configured, it caches files and redirects clients to signed URLs. When RemoteCache is nil, it proxies requests directly to the source.

The handler supports both simple proxying and advanced features like:

  • Cloud storage caching via RemoteCache
  • CIDN-based distributed blob management
  • Content freshness checking with CheckSyncTimeout
  • Host extraction from URL path with HostFromFirstPath
  • File suffix blocking via BlockSuffix

func (*MirrorHandler) ServeHTTP

func (m *MirrorHandler) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP implements the http.Handler interface. It processes HTTP GET and HEAD requests, optionally caching responses.

Request processing:

  1. Validates request method (only GET and HEAD allowed)
  2. Extracts target host and path
  3. Applies filters (BlockSuffix, BaseDomain, valid domain check)
  4. Routes to cacheResponse if RemoteCache is set, otherwise directResponse

Returns:

  • 405 Method Not Allowed for non-GET/HEAD requests
  • 403 Forbidden for blocked suffixes
  • 404 Not Found for invalid paths or domains
  • 302 Found (redirect) for cached files
  • 500 Internal Server Error for failures
  • 200 OK for successful proxied or cached responses

Directories

Path Synopsis
cmd
httpmirror command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL