Configure web search

This page is for operators. It covers how Helix serves its web search API and the agent web search and browser skills, and how to configure them. All settings are environment variables on the control plane.

How it works

Every web search request, whether it comes from an API client, the helix websearch CLI or an agent skill, goes to one provider. With the default native provider, Helix serves requests from its own stack:

query ──► SearXNG ──► search engines (default: Yandex)
              │
              ▼ candidate URLs
          page cache (Postgres web_pages) ──hit──► extraction
              │ miss
              ▼
          guarded HTTP fetch ──JS-rendered page?──► headless Chrome
              │                                       │ all traffic via the
              │                                       ▼ in-process egress proxy
              └──────────────► extraction (readability, Markdown, PDF/office)
                                   │
                                   ▼
                    embeddings (highlights, findSimilar)
                    LLM (summaries, deep search, outputSchema, answer)

1. Discover. SearXNG queries the configured search engines and returns candidate URLs. Helix pages through results until it has numResults, and applies domain and date filters.
2. Fetch. Pages are loaded from the cache or fetched. Helix first makes a plain HTTP request. If the HTML looks like a JavaScript app (scripts and almost no visible text, or an empty SPA mount node), it renders the page in headless Chrome. Every connection, from the Go client and from Chrome, is checked against the same destination policy.
3. Extract. Main content is extracted and converted to Markdown. PDFs and office documents go through Helix's document extractor.
4. Rank and synthesise. An embedding model ranks highlights and findSimilar results. An LLM writes summaries, deep-search query rewrites, outputSchema output and answers.

With SEARCH_PROVIDER=exa, Helix forwards every request to the Exa API unchanged and none of the stack above is used. Callers and agents see the same API either way.

Choose a provider

The provider applies to the whole deployment, including agent web search and browser tool calls. Callers can't tell which provider served them, and they're charged the same prices either way.

Search engines

SEARCH_SEARXNG_ENGINES is a comma-separated list of the SearXNG engines Helix queries. Helix supports five:

The default is yandex. Any other name, or an empty list, stops the API at startup with an error. Helix's SearXNG settings (searxng/settings.yml in Compose, the searxng-config ConfigMap in Helm) enable all five, so changing the list needs no SearXNG change.

SEARCH_SEARXNG_ENGINES=yandex,brave

SearXNG scrapes these engines' public result pages, and engines throttle scrapers. From datacentre IP addresses you should expect rate limits and CAPTCHAs. In our tests on a datacentre network:

• Yandex returned relevant results.
• Bing returned unrelated pages for every query and language setting, which is why it's no longer in the default.
• Google often returned nothing.
• DuckDuckGo and Brave were CAPTCHA'd.

Results vary by network and over time, so test the engines from your own deployment before adding them. When an engine fails while others answer, responses carry a PARTIAL_RESULTS warning naming it. Engines' terms of service may also forbid automated querying, so check them for the engines you enable. For a guaranteed level of service, use SEARCH_PROVIDER=exa.

SEARCH_SEARXNG_BASE_URL points at SearXNG (default http://searxng:8080; the Helm chart sets it to its own SearXNG service). Helix parses SearXNG's JSON output, so the Compose files and the chart pin the image to searxng/searxng:2025.12.7-9d3ec9a2a. Test before changing that tag.

LLM and embeddings

Several features need a model. Each setting names a provider already configured on the deployment (see Configure LLM providers) and a model it serves.

/findSimilar needs both. Plain search and contents (text, links, images, subpages) need neither.

The LLM has three tiers. SEARCH_LLM_FAST_MODEL and SEARCH_LLM_STRONG_MODEL default to SEARCH_LLM_MODEL:

If SEARCH_LLM_PROVIDER is set, SEARCH_LLM_MODEL is required; the API won't start without it.

Fetching and security

Search requests let any API key make Helix fetch an arbitrary URL, so the fetch path is guarded against server-side request forgery:

• The HTTP client checks the destination of every connection, including each redirect (up to 10).
• Chrome runs each fetch in a fresh browser context whose traffic (navigation, redirects, subresources and scripts' requests) goes through an egress proxy inside the API process. The proxy applies the same policy at connect time and uses credentials generated when the process starts.
• Responses over 20 MB are rejected.

By default, loopback, private-network (RFC 1918, unique-local, CGNAT), link-local and cloud metadata addresses are blocked, and a URL that reaches one fails with BLOCKED_DESTINATION. Blocked fetches are logged at warning level with the caller's user and organisation IDs.

SEARCH_FETCH_ALLOW_PRIVATE_NETWORKS=true allows loopback and private-network addresses. Link-local and cloud metadata addresses stay blocked regardless. Leave it off unless the deployment is self-hosted, every API key holder is trusted, and you want search to read an internal web. With it on, anyone with an API key can read internal services through /contents.

The browser egress proxy

The API serves the egress proxy on SEARCH_BROWSER_PROXY_LISTEN_ADDR (default :8090), and Chrome reaches it at SEARCH_BROWSER_PROXY_URL (default http://api:8090, the Compose service name). Chrome must be able to reach that URL. Nothing else should, so don't publish the port outside the cluster or Compose network.

The Helm chart handles this: the control plane exposes container port 8090, a ClusterIP Service named <release>-browser-proxy points at it, and the chart sets SEARCH_BROWSER_PROXY_URL to that Service.

WebRTC can send UDP around a proxy. When Helix launches Chrome through the rod launcher (the default, RAG_CRAWLER_LAUNCHER_ENABLED=true, as in the Compose files and the chart), it passes --force-webrtc-ip-handling-policy=disable_non_proxied_udp for you. If you disable the launcher and point RAG_CRAWLER_CHROME_URL at a Chrome you run yourself, start that Chrome with this flag.

SEARCH_FETCH_CONCURRENCY (default 8) caps the pages fetched at once per API process, across all requests. Raise it if /contents calls with many URLs queue for too long. Every fetch holds a Chrome page or an HTTP connection, so size it against the Chrome container's resources.

Page cache

Fetched pages are stored in the Postgres web_pages table: the raw response body, final URL, status, content type and fetched_at, keyed by normalised URL. The URL is also the document id clients see.

The cache is shared across organisations. It holds only public web content fetched for a URL, never anything else a caller sent. One consequence: an organisation can tell from a cached status that someone fetched the page earlier.

Callers control freshness per request with maxAgeHours (see Freshness). Without it, a cached copy is used however old it is. Helix doesn't expire rows, so prune the table yourself if it grows too large:

docker exec helix-postgres-1 psql -U postgres -d postgres \
  -c "DELETE FROM web_pages WHERE fetched_at < now() - interval '30 days';"

Billing and prices

When billing is enabled on the deployment (STRIPE_BILLING_ENABLED=true), web search requests are charged in Helix credits from the calling organisation's wallet. The defaults mirror Exa's list prices. Each response reports its cost in costDollars, and a request the wallet can't cover fails with 402 NO_MORE_CREDITS. Without billing enabled, nothing is charged.

Pages that fail to fetch and requests that fail aren't charged. The same prices apply whether the native or exa provider serves the request. Agent web search and browser tool calls also go through this API and are charged like other agent usage.

Environment variable reference

Chrome itself is configured with the existing RAG_CRAWLER_* variables (RAG_CRAWLER_LAUNCHER_URL, RAG_CRAWLER_CHROME_URL, RAG_CRAWLER_LAUNCHER_ENABLED).

Docker Compose

The Compose files already run SearXNG and Chrome on the same network as the API, so the defaults work. Add the model settings to .env, which the api service reads:

SEARCH_SEARXNG_ENGINES=yandex
SEARCH_LLM_PROVIDER=openai
SEARCH_LLM_MODEL=gpt-4o
SEARCH_LLM_FAST_MODEL=gpt-4o-mini
SEARCH_EMBEDDINGS_PROVIDER=openai
SEARCH_EMBEDDINGS_MODEL=text-embedding-3-small

Then recreate the API container so it picks up the new environment (docker compose restart doesn't reload .env):

docker compose up -d api

To use Exa instead:

SEARCH_PROVIDER=exa
SEARCH_EXA_API_KEY=...

Helm

Keep searxng.enabled and chrome.enabled on (both default to true) and pass the settings through controlplane.extraEnv. The chart already sets SEARCH_SEARXNG_BASE_URL and SEARCH_BROWSER_PROXY_URL.

searxng:
  enabled: true
 
chrome:
  enabled: true
 
controlplane:
  extraEnv:
    - name: SEARCH_SEARXNG_ENGINES
      value: "yandex"
    - name: SEARCH_LLM_PROVIDER
      value: "openai"
    - name: SEARCH_LLM_MODEL
      value: "gpt-4o"
    - name: SEARCH_LLM_FAST_MODEL
      value: "gpt-4o-mini"
    - name: SEARCH_EMBEDDINGS_PROVIDER
      value: "openai"
    - name: SEARCH_EMBEDDINGS_MODEL
      value: "text-embedding-3-small"

For Exa, set SEARCH_PROVIDER=exa and read the key from a Secret:

controlplane:
  extraEnv:
    - name: SEARCH_PROVIDER
      value: "exa"
    - name: SEARCH_EXA_API_KEY
      valueFrom:
        secretKeyRef:
          name: exa-api-key
          key: api-key

See Linux & Kubernetes for installing the chart.

Verify

Make a search with the CLI:

export HELIX_URL=https://helix.example.com HELIX_API_KEY=hl-...
helix websearch search "go generics tutorial" -n 3
helix websearch answer "In which Go version were generics introduced?"

For a full check, run the SDK smoke tests from the Helix repository. They drive the pinned exa-py and exa-js SDKs, unmodified, against your deployment: search, contents, findSimilar, answers (plain and streamed) and an error path. They need Docker on the machine you run them from, and the deployment needs SEARCH_LLM_* and SEARCH_EMBEDDINGS_* configured.

HELIX_URL=https://helix.example.com HELIX_API_KEY=hl-... \
  integration-test/websearch-sdk/run.sh

Every line should start with PASS.

Troubleshooting

Responses carry PARTIAL_RESULTS warnings naming an engine, or results are few or irrelevant. The named engine was rate-limited, CAPTCHA'd or didn't respond. Check the SearXNG logs (docker compose logs searxng, or the searxng pod), remove engines that keep failing from SEARCH_SEARXNG_ENGINES, or switch to SEARCH_PROVIDER=exa.

Every search fails with 503 UPSTREAM_ERROR, "search engines unavailable". No engine answered any query. Check that the API can reach SEARCH_SEARXNG_BASE_URL and that SearXNG itself can reach the internet (through global.extraEnv proxy settings, if your network needs them).

Requests fail with 400 FEATURE_DISABLED. The error names the field. For summary, outputSchema, deep search, /answer or /findSimilar, set SEARCH_LLM_PROVIDER and SEARCH_LLM_MODEL. For highlights or /findSimilar, set SEARCH_EMBEDDINGS_PROVIDER and SEARCH_EMBEDDINGS_MODEL. compliance, snapshotAsOf, crawledBeforeDate and beta highlight options aren't supported on the native provider at all.

Pages fail with BLOCKED_DESTINATION. The URL, or a redirect it led to, resolves to a private, loopback, link-local or metadata address. That is intended. Set SEARCH_FETCH_ALLOW_PRIVATE_NETWORKS=true only if you need search to read an internal web and accept the risk described in Fetching and security.

Pages that need JavaScript fail with SOURCE_NOT_AVAILABLE while plain pages work. Chrome probably can't reach the egress proxy. Check that SEARCH_BROWSER_PROXY_URL is reachable from the Chrome container, and in Kubernetes that the <release>-browser-proxy Service exists and selects the control plane pod.

Requests fail with 402 NO_MORE_CREDITS. The calling organisation's wallet balance can't cover the request. Top up the organisation's credits, or adjust the SEARCH_PRICE_* variables if the prices are wrong for your deployment.

The API doesn't start and logs a web search: error. An unknown SEARCH_PROVIDER or engine name, SEARCH_PROVIDER=exa without SEARCH_EXA_API_KEY, or SEARCH_LLM_PROVIDER without SEARCH_LLM_MODEL. The message names the setting.