github stanfordnlp/dspy 3.4.0b1
on GitHub

pre-release4 hours ago

DSPy 3.4.0 Beta 1

DSPy 3.4.0b1 is the first beta of 3.4. It moves language-model execution to a shared engine interface, adds a local CPython interpreter for trusted code, and brings async execution to ReActV2. It also adds custom Flex code proposals in GEPA and fixes evaluation, streaming, and demonstration-sampling bugs.

This is a prerelease, not the stable 3.4.0 release. APIs and behavior may change before stable. We especially welcome feedback on native versus LiteLLM compatibility, tool calling and streaming, custom LM migration, saved programs, and multi-answer latency and costs. Please include your engine selection and a minimal reproduction when reporting issues.

Install this beta explicitly: 

pip install --upgrade "dspy==3.4.0b1"

3.4 is the LM transition release; 3.5 is the migration deadline. Ordinary DSPy programs and list-returning prompt calls remain supported, but the experimental LM types introduced in 3.3 are replaced in this release. Review the compatibility notes if you use those types, implement a custom LM, pass OpenAI-style messages directly to an LM, or request multiple answers with n.

Highlights

Native LM Engines and a New Custom-Backend Interface — @MaximeRivest

DSPy's LM layer now uses the lm15 request, response, and streaming types bundled with DSPy. Import them from dspy.lm15; no separate lm15 installation is needed.

The default engine="auto" prefers native lm15 execution for supported routes and representable inputs. Unsupported routes, client settings, and ordinary provider-specific inputs that cannot be represented faithfully select LiteLLM before execution. Authentication failures, timeouts, and provider errors do not trigger a switch to another backend. 

import dspy

# Prefer native execution where supported; select compatibility where needed.
lm = dspy.LM("openai/gpt-4o-mini")

# Explicitly retain the LiteLLM compatibility backend.
compat_lm = dspy.LM("openai/gpt-4o-mini", engine="litellm")

Use engine="lm15" to require native execution and reject unsupported mappings. LiteLLM remains installed and supported; it is not deprecated.

Custom backends can now implement complete(Request) -> Response instead of subclassing BaseLM and returning provider-shaped objects: 

from dspy.lm15 import Message, Response, Usage

class EchoEngine:
    def complete(self, request):
        return Response(
            id=None,
            model=request.model,
            message=Message.assistant("hello"),
            finish_reason="stop",
            usage=Usage(),
        )

lm = dspy.LM("custom/echo", engine=EchoEngine())

Supply async_engine= for async calls and implement canonical streaming events when needed. DSPy owns response caching, managed retries, callbacks, history, and usage accounting. Custom engines are caller-owned and are not closed by DSPy.

Ordinary calls retain their existing cache-key format and can read existing SDK-response cache entries. New native cache entries contain plain serialized data and support restricted deserialization; explicit typed requests use a separate cache namespace. Existing dspy.streamify listeners remain supported.

See the LM migration guide and custom-engine tutorial.

PRs: #10359, #10366, #10371

LocalInterpreter: Persistent CPython for Trusted Code — @isaacbmiller

dspy.LocalInterpreter runs generated Python in a persistent local CPython subprocess, using the current Python executable. It provides ordinary Python compatibility without Deno, while separating the worker's memory, stdout, and lifecycle from the DSPy process. 

rlm = dspy.RLM(
    "question: str -> answer: int",
    interpreter_factory=dspy.LocalInterpreter,
)

State and imports persist within an interpreter session. LocalInterpreter supports JSON-compatible inputs and tool results, sync and async host tools, typed SUBMIT, interpreter callbacks, and execution timeouts. It works with both RLM and Flex. The usual factory lifecycle still creates a fresh interpreter for each module invocation.

LocalInterpreter is not a security sandbox. Generated code retains the host user's filesystem, environment, credentials, subprocess, and network access. Use the default PythonInterpreter or a remote sandbox for untrusted code; the default has not changed.

execution_timeout includes host-tool time and terminates the worker when exceeded. It cannot forcibly stop a running host callable; that callable may finish later, but its result is discarded. Guest threads must finish before an execution returns, or the interpreter session becomes terminal.

Host-tool cancellation and interrupts propagate to the caller and shut down the stranded worker, including when no execution timeout is configured. They no longer leave execution waiting indefinitely for a worker reply. Ordinary tool errors remain recoverable.

PRs: #10238, #10379

Async ReActV2 and Explicit Final-Output Failures — @isaacbmiller

The experimental dspy.ReActV2 now supports await agent.acall(...), including async prediction, async tools, and forced final submission. This enables async MCP tools to run through the agent's structured tool-call history, preserving call IDs and tool results across turns.

Tools execute sequentially on the async path. This does not add parallel tool execution or automatically offload blocking synchronous tools.

ReActV2 also no longer returns an incomplete Prediction when final submission fails. Missing or invalid submission raises ValueError; parse and context-window failures from the forced submission propagate. Successful predictions continue to include declared outputs, history, and a termination reason.

The MCP guide includes an async ReActV2 client/server example.

PRs: #10355, #10356

Custom Flex Code Proposals and More Reliable GEPA Evaluation — @dbreunig

dspy.GEPA now accepts code_proposer=, complementing instruction_proposer=. Custom proposers receive the selected Flex code components, candidate source, reflective examples, task descriptions, and context blurbs, and return replacement module source for each component.

This lets applications customize code-generation constraints and domain guidance without monkeypatching the built-in proposer. The built-in proposer remains the default, and programs without Flex components are unaffected.

GEPA's trace-capture evaluation also keeps outputs and scores aligned with the input batch when a program crashes on an example. Failed examples receive failure_score rather than disappearing and causing an indexing error or shifting later results. This addresses the missing/misaligned validation-results issue noted in the 3.3.1 release; it does not require an upstream GEPA upgrade.

PRs: #10212, #10305

API and Compatibility Changes

The Experimental 3.3 LM Types Are Replaced Now

The old dspy.LMRequest, dspy.LMResponse, and related experimental exports are removed. Importing dspy.core.types raises a migration error, and forward_contract="typed_lm" is rejected. These are replacements, not aliases:

Experimental 3.3 API 3.4 replacement
dspy.LMRequest, dspy.LMResponse dspy.lm15.Request, dspy.lm15.Response
dspy.LMMessage, dspy.LMConfig dspy.lm15.Message, dspy.lm15.Config
dspy.System(text) Request(system=text, ...)
dspy.User(text), dspy.Assistant(text) Message.user(text), Message.assistant(text)
response.outputs[0].parts response.message.parts

The new types are frozen dataclasses, not Pydantic models, and their validation and data shapes differ. Old pickles containing removed experimental classes are not automatically migrated; load and export them in their original environment first. Ordinary provider-response caches are a separate compatibility path and remain readable.

experimental=True no longer changes ordinary LM calls into typed responses. Ordinary prompt calls return lists; explicit requests return a Response: 

from dspy.lm15 import Config, Message, Request

lm = dspy.LM("openai/gpt-4o-mini")
response = lm(Request(
    model=lm.model,
    system="Be concise.",
    messages=(Message.user("What is DSPy?"),),
    config=Config(max_tokens=200),
))
print(response.text)

An explicit request's model must match the LM. Set generation options in its Config; LM generation defaults are not added. Config.cache controls provider-side prompt caching, not DSPy's response cache.

DSPy's signature types, including Image, Audio, File, Tool, ToolCalls, and History, are not removed. Existing public DSPy LM error classes remain supported.

Legacy LM Integrations Are Deprecated for Removal in 3.5

The following still execute in 3.4 but emit DeprecationWarning:

  • Public OpenAI-style lm(messages=[...]) calls, including provider SDK message objects. Migrate to explicit lm15 requests.
  • Custom BaseLM.forward() and aforward() integrations. Migrate to the engine interface.
  • LegacyEngine, AsyncLegacyEngine, and custom complete_legacy() shortcuts. These are transition tools, not permanent compatibility interfaces.

lm("hello") remains a list-returning convenience in 3.5. Built-in adapters still use an internal dictionary/list boundary in 3.4; their canonical request/response migration is scheduled for 3.5. Custom adapters must not depend on the removed dspy.clients.openai_format module.

To reveal deprecation warnings during development: 

python -W default::DeprecationWarning your_program.py

Native Multi-Answer Calls, Retries, and Streaming

  • Native n answers use separate sequential requests. This can increase latency and bill input tokens more than once compared with a provider-native multi-answer request. Choose engine="litellm" to retain that backend's native n behavior.
  • DSPy owns managed retries and honors valid provider retry hints. A stream is not retried after a chunk has been emitted, and incomplete responses are not cached as successful results.
  • Native cost estimates remain unknown when reliable pricing is unavailable; unknown cost is not reported as zero.
  • Typed LiteLLM streaming currently supports Chat Completions only. Native Responses streaming is supported.
  • Native listener-facing chunks need not be LiteLLM class instances. Custom chunk consumers should not rely on that identity.

PRs for the LM changes above: #10366, #10371

Other Compatibility Changes

  • Pydantic 2.11.0 is now the minimum supported version. #10271
  • ReActV2 rejects signatures with output fields named history or termination_reason, which collide with its prediction metadata. Rename those outputs. #9852
  • Calling an async tool synchronously from a running event loop raises a descriptive ValueError, even with allow_tool_async_sync_conversion enabled. Use await tool.acall(...). #10146
  • Synchronous streaming now propagates producer failures to the consumer instead of silently ending. #9864
  • Evaluate rejects an empty development set with a descriptive ValueError instead of failing with ZeroDivisionError. #9978

Full PR List

Language Models and Engines

Agents, Tools, and Interpreters

Optimization, Evaluation, and Examples

  • Make Example hashing order-insensitive to match equality by @Kymi808 (#9858).
  • Preserve the input/label split when copying an Example by @iamsharduld (#9946).
  • Sample labeled demonstrations independently from the full available pool for each BootstrapFewShot predictor by @iamsharduld (#9948).
  • Raise a descriptive error for empty evaluation sets by @roli-lpci (#9978).
  • Add GEPA's custom Flex code_proposer hook by @dbreunig (#10212).
  • Keep GEPA trace-capture results aligned after program failures by @dbreunig (#10305).

Adapters, Streaming, and Batching

Documentation

Dependencies, CI, and Release Engineering

Contributors

Thank you to @chuenchen309, @dbreunig, @ellacroix, @he-yufeng, @iamsharduld, @isaacbmiller, @Kymi808, @MaximeRivest, @michaelisaac-dev, @nikolauspschuetz, @NishchayMahor, @roli-lpci, @spjosyula, @tjdharamsi, and @ymxlx for contributing to this release.

Automation contributions were made by @dependabot and @github-actions.

Full Changelog: 3.3.1...3.4.0b1

Check out latest releases or
releases around stanfordnlp/dspy 3.4.0b1

Don't miss a new dspy release

NewReleases is sending notifications on new releases.

Get notifications