Pushing OpenAI’s new flagship model to beat GPT-4.1 and a GPT-4.1 + o3-mini hybrid across Agentic AI (CoT/ReAct and Plan–Execute) workflows for Deep Research.

In my previous post, the rebels (Mistral AI small) held strong. I did a drop-in swap of GPT-5 with zero adaptation (model defaults, no prompt changes), and the result was parity at best.

This time, I came back with a more disciplined benchmark on Deep Research (YT) using two orchestration patterns: Agentic Research (Chain-of-Thought/ReAct-style supervisor) and Deep Agent (Plan–Execute). The goal wasn’t verbosity; it was structured reasoning under constraints.

TL;DR — Short version

GPT-5 wins. Even with minimal reasoning and low verbosity, it solves the tricky case, follows instructions more strictly, executes tool calls and produces structured outputs correctly, and needs no sidekick model. The hybrid GPT-4.1-mini + o3-mini remains strong, but it’s slower, costlier in tokens, and more complex to operate. GPT-4.1-mini alone consistently struggles on the complex planning test. If you’re starting new work on deep research pipelines, pick GPT-5.

---

Why Mistral first, and why a second wave?

In the first wave I wanted a “fair drop-in” scenario:

• Keep GPT-5 default parameters (which are medium reasoning & verbosity; which actually hurt performance in small-model-tuned workflows).
• No prompt surgery or tool routing tweaks.
• Just swap models into existing workflows and watch.

Mistral AI came out as a legit contender under those rules. GPT-5 didn’t collapse, but it also didn’t obviously dominate. That told me: drop-in parity ≠ true capability.

So I launched a second wave to answer: If I meet GPT-5 halfway—just enough to unlock its sweet spot—does it pull ahead decisively?

---

What changed in the second wave (and why)

I kept changes surgical and transparent:

1. Use GPT-5-mini with Agents SDK default values

model_settings = ModelSettings(reasoning=Reasoning(effort="minimal"), verbosity="low")
# or
model_settings = get_default_model_settings(extract_model_name(config.models.search_model))
# optional if you standardize on GPT-5
# OPENAI_DEFAULT_MODEL=gpt-5

I stayed at minimal reasoning / low verbosity intentionally—if GPT-5 is as good as claimed, it shouldn’t need “high-effort” to win.

2. Upgrade evaluation: add LLM-as-judge (content + spec)

My original path evaluation (did the agent call the right tools, in the right order?) wasn’t sufficient. I added an LLM-as-judge to verify content and contract:

• It’s a research report, not a syllabus/podcast/script.
• Required sections are present (Raw Notes → Detailed Agenda → Report).
• Adherence to length & style specs (e.g., 250–300 words for the report).

This also solved a practical gotcha: GPT-5 tends to over-structure outputs (adds parentheses in headings, translates constants like “Raw Notes” into French, etc.). My old grep checks broke on that strict matching. LLM-as-judge made evaluation robust without penalizing readability.

3. Fix the input contract

A subtle bug: I had wrapped the user request in <syllabus/>, so GPT-5 kept returning course outlines instead of research reports. GPT-5 is prompt-strict; if your input is contradictory, it will refuse or follow it literally. That’s a great developer asset (fails fast), but the payload had to be cleaned up.

---

Where GPT-5 immediately pulled ahead: the writer agent

The writer has a dual goal in one turn:

1. Execute function calls / handoffs; and
2. Emit a structured report (three sections: Raw Notes → Agenda → Report).

• GPT-4.1-mini consistently refused to do both simultaneously (wanted JSON or function calling, not both).
• Mistral could do it after iterations.
• GPT-5 did it on the first try.

This mattered later: if a model can’t meet two end goals, you burn time building glue code.

---

Two workflows, one question

I ran the same research ask through two orchestration patterns:

• Agentic Research = CoT/ReAct-style supervision (one supervisor that thinks and acts).
• Deep Research = Plan–Execute (a planner proposes; an executor runs the plan).

Step 1 — Baseline (unit) test

A deliberately easy prompt: “Write a short, neutral, technical report on MIPS + ANN with two algorithms (HNSW, FAISS) and compare with ReWOO; 250–300 words; 4–5 sections; no off-topic multi-agent content.” The goal is format & content sanity across models and both workflows.

Expectation: everyone should pass. If there’s no clear winner here, move to a stressor.

Step 2 — Tricky planning test (stressor)

A planning-first puzzle to force stepwise reasoning during plan generation:

• Hard budget: cap searches to 5–7. If evidence is missing, stop (don’t keep digging forever).
• Split evidence across files: the model must connect facts scattered across different docs in the knowledge base.
• Inject inconsistencies: include contradictory hints that must be reconciled.
• No pre-specified sources: the model must choose the right files on its own.
• Strict output contract: produce the three sections (Raw Notes → Detailed Agenda → Report). No syllabus, no podcast, no script.

Why this is hard: non-reasoning models tend to either (a) over-search (latency/cost blow-ups) or (b) guess and miss cross-file links. A reasoning-capable model should plan, check, course-correct, and still respect the budget.

---

Results

Baseline (unit) test — everyone passes

All models produced valid reports with the right scope and sectioning across both workflows. No material functional gaps here.

Run	Workflow	Model	Content (0–5)	Spec (0–5)	Notes
1	Agentic Research	GPT-5-mini	5	5	Clean, concise; minor stylistic flourishes ok.
2	Deep Research	GPT-5-mini	5	5	Same quality; slightly tighter by default.
3	Deep Research	GPT-4.1-mini	5	4–5	Passes; occasional minor spec drift.
4	Agentic Research	GPT-4.1-mini	5	4	Passes; supervisor sometimes over-elaborates.

Latency & tokens for the baseline were close for this short task, so I focus on the stressor where gaps actually appear.

Tricky planning test — this is where GPT-5 separates

Here the plan quality, budget discipline, and structured output matter. I also added the hybrid arm (GPT-4.1-mini + o3-mini) only for this test to benchmark a “reasoning-assist” strategy.

Run	Workflow	Model	Content (0–5)	Spec (0–5)	Tokens (≈)	Latency (≈)	Notes
1	Agentic Research	GPT-5-mini	5	4	~250k	~40 s	Strong; tiny verbosity drift.
2	Deep Research	GPT-5-mini	5	5	~220k	~35 s	Winner: clean plan; respects 5–7 search cap; nails structure.
3	Deep Research	GPT-4.1-mini	3	1	~500k	~80 s	Off-spec (drifts), misses structure; weaker plan discipline.
4	Agentic Research	GPT-4.1-mini	4	3	~1.88M	~265 s	Loops searches beyond budget; expensive; slower.
5	Agentic Research	GPT-4.1-mini + o3-mini	5	4	~500k	~90 s	Solves the puzzle; slower + costlier; operationally heavier.
6	Deep Research	GPT-4.1-mini + o3-mini	5	4	~430k	~75 s	Strong; still slower than GPT-5-mini (Run 2).

Takeaways

• Baseline: all pass; no big winner → hence the stressor.
• Tricky: GPT-5-mini (Run 2) wins; the hybrid is competitive but slower & pricier; GPT-4.1-mini alone struggles with plan discipline and spec adherence.
• Budget respect: GPT-5 reliably stops at 5–7 searches; GPT-4.1 often keeps digging.
• Dual goal: GPT-5 handled function-calling + structured output; GPT-4.1 refused the dual objective. Here for parity, dual goals was remove (no save_report function calling in Writer agent). This to avoid penalty on latency and token consumption).

Note on Mistral: I excluded Mistral from the global workflow ranking here because our file_search agent relies on OpenAI’s Response API for retrieval, biasing comparisons. It remains a strong contender, but not comparable under this setup.

---

Why GPT-5 pulled ahead

1. Instruction fidelity — It’s prompt-strict. Clean inputs → clean behavior. Contradictory inputs (e.g., a stray <syllabus/>) make it refuse or follow literally, which is a developer asset: failures surface early.
2. Programmability — It reliably does function calls / handoffs and returns structured outputs in the same turn, which is crucial for supervisors and writer agents.
3. Search discipline — It respects budgets (« max 7 searches »), protecting latency and cost; GPT-4.1 tends to chase.
4. Writing quality & structure — Even with low reasoning, outputs are technical and well-structured. GPT-5 also tends to over-structure (parentheses, translated headers), which broke my old grep checks; switching to LLM-as-judge fixed evaluation without penalizing clarity.
5. No sidekick required — The GPT-4.1-mini + o3-mini hybrid is good, but slower, costlier, and more operationally complex. GPT-5 delivers the capability in one model.

---

Notes on integration

• Agents SDK plays nicer with GPT-5 than my older LangChain/LangGraph stacks for these tests. Small prompt fixes + minimal reasoning / low verbosity were enough.
• Evaluation harness: keep path tracing (tool-call traces) and LLM-as-judge (content + spec). It catches “looks-right but wrong-form” failures and protects against syllabus drift.

Latency & token overhead — GPT-5 vs GPT-4.1

From the waterfall traces, GPT-5-mini (low reasoning) shows a slight per-call overhead compared to GPT-4.1-mini:

Model	Total Tokens (≈)	Latency (≈)	Notes
GPT-5-mini	~263k	~163 s	Slower per call, lower token usage.
GPT-4.1-mini	~313k	~135 s	Faster per call, higher token usage.

This gap reinforces the need to optimize workflows (parallel agents, removing deterministic tool calls) if GPT-5 is to be deployed in production-heavy environments.

---

Should you switch?

• Already stable on GPT-4.1 in prod? Don’t rush. You’ll touch prompts and deployment threads to get the most from GPT-5.
• Starting new work or evolving a research pipeline? Choose GPT-5. It’s more reliable, more programmable, more versatile, and it already wins in minimal reasoning + low verbosity.

As always, keep an evaluation harness (path tracing + LLM-as-judge) in place. GPT-5 is powerful; good tests make it predictable.

---

Appendix: test inputs

Baseline prompt

Summarize how autonomous agents use external memory retrieval through Maximum Inner Product Search (MIPS).

Explain why Approximate Nearest Neighbor (ANN) methods are required in this context.

List and briefly describe the main ANN algorithms discussed in the reference (LSH, ANNOY, HNSW, FAISS, ScaNN), keeping each description to a single sentence.

Conclude with a short explanation of the trade-off between latency and recall in such systems.

Tricky planning prompt (stressor)

Write a report explaining how LLM agents use external memory using Maximum Inner Product Search (MIPS), why they use Approximate Nearest Neighbor (ANN), giving two examples of ANN algorithms, and compare this approach with ReWOO.

Why this is a stressor compared to baseline:
While the baseline is essentially an enumeration task (explain, list, conclude), the tricky prompt is synthetic: it requires choice (pick two ANN algorithms), integration across sources (MIPS/ANN vs ReWOO come from different docs), and a comparative analysis under a strict budget of searches. Non-reasoning models tend to either over-search or guess poorly here. Reasoning-capable models (like GPT-5) can plan, reconcile contradictions, and still meet the contract.

---

Appendix B: Notes & reflections

• Reasoning defaults: GPT-5 defaults to medium reasoning, which explains why it underperformed in drop-in tests (over-thinking vs small-model-tuned workflows). Setting to low makes it behave closer to non-reasoning models, but with instruction-following and reasoning intact.
• Prompt strictness: <syllabus/> issue wasn’t malformed but contradictory. GPT-5 refused or followed literally → a developer asset (fail fast).
• Writer agent trajectory test: GPT-5 succeeded immediately; GPT-4 worked too; GPT-4.1 refused (dual-goal error); Mistral managed after iterations. GPT-5 likely benefited from earlier alignment efforts.
• Instruction compatibility: When instructions were updated for GPT-5, Mistral and GPT-4.1 still worked fine → no need for per-model versioning.
• Latency gap: GPT-5-mini slower per call than GPT-4.1-mini (see waterfall table). Next steps: parallelize agents, remove deterministic function calls (reading/writing) from agent loop.
• Agentic vs Deep parity: GPT-5 sometimes parallelized tool calls in Agentic Research, bringing performance closer to Deep Research (plan–execute). A sign it bridges both.
• Evaluation harness: Added LLM-as-judge (content + spec) on top of path eval (tool calls, budget) to catch off-topic drifts (syllabus-style).
• Sidekick vs solo: Hybrid GPT-4.1+o3-mini is strong but operationally heavier; GPT-5 alone is simpler, cheaper, and wins.
• Next step: Optimize workflows to cut runtime overhead; explore parallelization patterns and deterministic tool separation.