Back

How ChatGPT Web Works – Considerations and Optimizations

MD Rashid Hussain
MD Rashid Hussain
Jun-2025  -  10 minutes to read
Image source attributed to: https://yourword.in

The web experience of ChatGPT is deceptively simple: you type a message and an AI assistant responds almost instantly. But behind that minimalistic interface is a highly orchestrated web system with strict performance requirements, stateful interactions, streaming pipelines, and resilience constraints — all optimized for responsiveness and scalability.

In this article, we’ll dissect how ChatGPT Web likely works under the hood, zooming into architectural choices, streaming techniques, resource management, and interaction design. This is tailored for engineers who understand the complexity of building systems that appear effortless.

ChatGPT Web is not a typical React app talking to a CRUD backend. Instead, it must:

  • Deliver low-latency responses for a computationally expensive inference task.
  • Manage long-lived sessions with context/history that must persist.
  • Allow streaming tokens back to the user while still in computation.
  • Ensure security and privacy, since prompts may be sensitive.
  • Scale to millions of users in parallel, including Pro users on GPT-4.

OpenAI has migrated ChatGPT Web to Remix, a full-stack React framework optimized for seamless data loading, strong UX primitives, and performance under real-world constraints. This shift significantly impacts how the web interface is architected and optimized.

Remix offers several advantages critical for an application like ChatGPT:

  • First-class support for streaming – ideal for streaming tokens to the user.
  • Nested routes with scoped data loading – enabling UI islands to remain interactive while new content loads.
  • Loaders and actions running at the edge or server – enabling fast, dynamic rendering close to users.
  • Progressive enhancement – allowing forms, navigations, and data interactions to work even with JavaScript disabled.

This aligns with ChatGPT’s goals: fast feedback, resilient interactions, and tight control over user experience.

Remix relies on route loaders for data-fetching and actions for mutations. For ChatGPT:

  • loader() fetches prior conversations or system settings on navigation.
  • action() handles sending a new message.
  • The streaming token response is not a single JSON result but a streamed HTTP response, which Remix supports natively using Response.body.
export async function action({ request }: ActionArgs) {
  const body = await request.formData();
  const prompt = body.get("prompt");
 
  const stream = getLLMTokenStream(prompt); // returns ReadableStream
  return new Response(stream, {
    headers: { "Content-Type": "text/event-stream" },
  });
}

Remix uses nested routes to scope rendering and data loading. For ChatGPT Web:

  • The sidebar (conversations list) is a separate route with its own loader.
  • The message pane listens for streamed tokens and incrementally updates.
  • Independent boundaries allow the app to update conversation state without tearing down or blocking the UI.

This modular routing keeps navigation snappy and state consistent — two things that matter immensely when users are actively chatting.

Chat history and metadata are managed in local state but periodically synced to the server for:

  • Continuity across sessions
  • Reference-based rerendering
  • Retryable requests
const [chatHistory, setChatHistory] = useState<Message[]>([]);
const [currentInput, setCurrentInput] = useState("");

But local state alone is not enough. There's intelligent diff-based rehydration from server-persisted conversations when reloading the page.

flowchart TB A[User Input] B[Debounced Submit] C[WebSocket/HTTP Stream] D[Inference Worker] E[Token Stream] F[UI] A --> B B --> C C --> D D --> E E --> F
  • Debounced Submit: User input is validated, trimmed, and potentially processed client-side (e.g., Markdown escaping).
  • WebSocket or HTTP Stream: OpenAI likely uses text/event-stream or WebSocket to push tokens as they’re generated.
  • Backend Routing: Load balancers route requests to available inference clusters (different for GPT-3.5 vs GPT-4).
  • Inference Engine: A token-by-token stream is generated and sent back as a stream, not a full message.
  • Streaming & Token Handling: Streaming is critical for perceived latency. Instead of waiting for the entire response, tokens are streamed and rendered incrementally.

SSE is a performant and easy-to-use way to implement token streaming. SSE has advantages like:

  • Native browser support
  • No custom reconnection logic
  • Efficient for unidirectional streaming

And the implementation is simple, just set the Content-Type header to text/event-stream and write the token to the response:

res.setHeader("Content-Type", "text/event-stream");
res.write(`data: ${token}\n\n`);

To reduce perceived latency, optimistic UI updates occur before tokens are returned:

setChatHistory((prev) => [...prev, { role: "user", content: input }]);
setChatHistory((prev) => [...prev, { role: "assistant", content: "" }]);

Then as tokens stream in, the assistant’s message is updated in-place.

When ChatGPT encounters a URL — either as part of a prompt or via user input — OpenAI employs several intelligent techniques to extract and interpret relevant content, ensuring both performance and quality. Steps in the Pipeline

  • Content Retrieval

    • Uses HTTP requests to fetch the raw HTML or use third-party APIs (e.g., Bing or internal search index) for web access.
    • For live browsing (e.g., GPT-4 with browsing), links are fetched with appropriate headers and user-agent strings to mimic real user traffic.

  • Content Filtering & Sanitization

    • HTML is parsed and cleaned to strip away ads, navigation bars, cookie banners, and unrelated JavaScript content.
    • This often involves boilerplate removal via libraries like Readability.js, BeautifulSoup, or custom DOM heuristics.

  • Semantic Extraction

    • Extracts high-signal content such as article bodies, structured data (e.g., JSON-LD, Open Graph), and metadata (title, author, date).
    • Breaks down the content into token-efficient chunks suitable for context windows.

  • Summarization & Compression

    • Before injecting into an LLM, the content is Summarized using smaller models (e.g., BART, GPT-3.5) and compressed semantically (vs syntactically) to retain key points
    • Helps reduce token usage in GPT-4-turbo’s 128k context window.

  • Rate Limiting & Caching

    • URLs are cached, and repeated accesses are rate-limited to avoid hitting live servers too often.
    • OpenAI may maintain a pre-fetched index of popular links (e.g., Wikipedia, news sources) for faster retrieval.

Multiple versions of models are supported like GPT-3.5-turbo (fast, cheaper), GPT-4-turbo (more capable, more expensive) etc. The backend must:

  • Route traffic based on user plan
  • Respect rate limits
  • Possibly perform warm-up invocations to avoid cold starts

Context size is crucial. GPT-4-turbo supports up to 128k tokens, but passing large context windows is expensive. Strategies include:

  • Sliding windows: Trim older messages unless pinned
  • Semantic compression: Summarize older chunks using another model
  • Thread IDs + references: Reduce token bloat by using references

When a user asks for sources, citations, or further reading, ChatGPT uses multiple strategies to surface references. These strategies depend on the model variant (e.g., with or without browsing), user plan, and product context.

  1. For most models operating in offline mode (e.g., GPT-4-turbo without browsing), references are generated from:

    • Patterns learned during pretraining (up to the cutoff date)

    • Memorized or paraphrased URLs, papers, and books seen during training

    • Hallucinated but plausible citations — especially if not grounded with a retrieval mechanism

      Caveat: These references may sound credible but might not exist. That’s why citations in offline mode should be verified externally.

  2. Using RAG: When using tools like Browse with Bing, plugins, or custom RAG pipelines, ChatGPT can:

    • Interpret the user’s query semantically
    • Retrieve documents from an external source, such as Bing Search API, vector databases (e.g., Pinecone, Weaviate, FAISS), or PDF/document stores
    • Inject relevant snippets into the model's context window
    • Generate a grounded answer, citing exact sources

    This approach enables accurate citations with titles and links, reduced hallucination, and source-traceable answers.

  3. Internal Tools: browse, code_interpreter, retrieval

    • OpenAI internally uses structured tool calls (via function calling) to enable specific referencing abilities.

  • URLs are cached, and repeated accesses are rate-limited to avoid hitting live servers too often.
  • OpenAI may maintain a pre-fetched index of popular links (e.g., Wikipedia, news sources) for faster retrieval.

  • Keep it fair and sustainable: ChatGPT Web includes token usage stats and caps, session rate limits, and passive timeouts and activity monitoring
  • Threading Model: Conversations can be branched. Each branch creates a new thread with shared ancestry — handled by a thread tree structure internally.
  • Response Rating & Feedback: Each message has a "thumbs up/down" that feeds back to RLHF systems. This feedback is associated with the user + thread + timestamp, and likely sent asynchronously and batched.

LLMs amplify risks due to prompt leakage, jailbreaks, or sensitive data exposure. Key mitigations:

  • Strict input sanitization
  • Output filtering using classifiers
  • Audit logs and monitoring
  • Rate limiting to prevent prompt injection testing at scale

At scale, observability is everything. Some assumed practices:

  • Token-level latency metrics (token/sec)
  • Prompt-class fingerprints (to detect jailbreaking attempts)
  • Heatmaps of API usage (to forecast GPU load)
  • Session-level satisfaction scores (UX quality over time)

ChatGPT Web is a masterpiece of both frontend finesse and backend power. While users see a seamless UI, engineers behind it balance model complexity, latency, memory, and cost.

  • Stream, don’t wait: Streaming improves perceived speed.
  • Think in sessions: LLMs are stateful by nature.
  • UI is not UX: Rendering tokens beautifully matters.
  • Optimize for cost: Not every interaction needs GPT-4.
  • Observe everything: Metrics guide iteration and safety.

If you're building an AI product with LLMs at its core, study ChatGPT’s design patterns. Not everything will be visible, but even what is should teach you a lot about thoughtful system design.

How ChatGPT Web Works – Considerations and Optimizations