langgenius/dify 2.0.0-beta.1

v2.0.0-beta.1 – Orchestrating Knowledge, Powering Workflows

on GitHub

🚀 Introduction

In Dify 2.0, we are introducing two major new capabilities: the Knowledge Pipeline and the Queue-based Graph Engine.

This is a beta release, and we hope to explore these improvements together with you and gather your feedback. The Knowledge Pipeline provides a modularized and extensible workflow for knowledge ingestion and processing, while the Queue-based Graph Engine makes workflow execution more robust and controllable. We believe these will help you build and debug AI applications more smoothly, and we look forward to your experiences to help us continuously improve.

📚 Knowledge Pipeline

✨ Introduction

With the brand-new orchestration interface for knowledge pipelines, we introduce a fundamental architectural upgrade that reshapes how document processing are designed and executed, providing a more modular and flexible workflow that enables users to orchestrate every stage of the pipeline. Enhanced with a wide range of powerful plugins available in the marketplace, it empowers users to flexibly integrate diverse data sources and processing tools. Ultimately, this architecture enables building highly customized, domain-specific RAG solutions that meet enterprises’ growing demands for scalability, adaptability, and precision.

❓ Why Do We Need It?

Previously, Dify's RAG users still encounter persistent challenges in real-world adoption — from inaccurate knowledge retrieval and information loss to limited data integration and extensibility. Common pain points include:

• 🔗 restricted integration of data sources
• 🖼️ missing critical elements such as tables and images
• ✂️ suboptimal chunking results

All of them lead to poor answer quality and hinder the model's overall performance.

In response, we reimagined RAG in Dify as an open and modular architecture, enabling developers, integrators, and domain experts to build document processing pipelines tailored to their specific requirements—from data ingestion to chunk storage and retrieval.

🛠️ Core Capabilities

🧩 Knowledge Pipeline Architecture

The Knowledge Pipeline is a visual, node-based orchestration system dedicated to document ingestion. It provides a customizable way to automate complex document processing, enabling fine-grained transformations and bridging raw content with structured, retrievable knowledge. Developers can build workflows step by step, like assembling puzzle pieces, making document handling easier to observe and adjust.

📑 Templates & Pipeline DSL

• ⚡ Start quickly with official templates
• 🔄 Customize and share pipelines by importing/exporting via DSL for easier reusability and collaboration

🔌 Customizable Data Sources & Tools

Each knowledge base can support multiple data sources. You can seamlessly integrate local files, online documents, cloud drives, and web crawlers through a plugin-based ingestion framework. Developers can extend the ecosystem with new data-source plugins, while marketplace processors handle specialized use cases like formulas, spreadsheets, and image parsing — ensuring accurate ingestion and structured representation.

🧾 New Chunking Strategies

In addition to General and Parent-Child modes, the new Q&A Processor plugin supports Q&A structures. This expands coverage for more use cases, balancing retrieval precision with contextual completeness.

🖼️ Image Extraction & Retrieval

Extract images from documents in multiple formats, store them as URLs in the knowledge base, and enable mixed text-image outputs to improve LLM-generated answers.

🧪 Test Run & Debugging Support

Before publishing a pipeline, you can:

• ▶️ Execute a single step or node independently
• 🔍 Inspect intermediate variables in detail
• 👀 Preview string variables as Markdown in the variable inspector

This provides safe iteration and debugging at every stage.

🔄 One-Click Migration from Legacy Knowledge Bases

Seamlessly convert existing knowledge bases into the Knowledge Pipeline architecture with a single action, ensuring smooth transition and backward compatibility.

🌟 Why It Matters

The Knowledge Pipeline makes knowledge management more transparent, debuggable, and extensible. It is not the endpoint, but a foundation for future enhancements such as multimodal retrieval, human-in-the-loop collaboration, and enterprise-level data governance. We’re excited to see how you apply it and share your feedback.

⚙️ Queue-based Graph Engine

❓ Why Do We Need It?

Previously, designing workflows with parallel branches often led to:

• 🌀 Difficulty managing branch states and reproducing errors
• ❌ Insufficient debugging information
• 🧱 Rigid execution logic lacking flexibility

These issues reduced the usability of complex workflows. To solve this, we redesigned the execution engine around queue scheduling, improving management of parallel tasks.

🛠️ Core Capabilities

📋 Queue Scheduling Model

All tasks enter a unified queue, where the scheduler manages dependencies and order. This reduces errors in parallel execution and makes topology more intuitive.

🎯 Flexible Execution Start Points

Execution can begin at any node, supporting partial runs, resumptions, and subgraph invocations.

🌊 Stream Processing Component

A new ResponseCoordinator handles streaming outputs from multiple nodes, such as token-by-token LLM generation or staged results from long-running tasks.

🕹️ Command Mechanism

With the CommandProcessor, workflows can be paused, resumed, or terminated during execution, enabling external control.

🧩 GraphEngineLayer

A new plugin layer that allows extending engine functionality without modifying core code. It can monitor states, send commands, and support custom monitoring.

Quickstart

1. Prerequisites
   • Dify version: 2.0.0-beta.1 or higher
2. How to Enable
   • Enabled by default, no additional configuration required.
   • Debug mode: set DEBUG=true to enable DebugLoggingLayer.
   • Execution limits:
     • WORKFLOW_MAX_EXECUTION_STEPS=500
     • WORKFLOW_MAX_EXECUTION_TIME=1200
     • WORKFLOW_CALL_MAX_DEPTH=10
   • Worker configuration (optional):
     • WORKFLOW_MIN_WORKERS=1
     • WORKFLOW_MAX_WORKERS=10
     • WORKFLOW_SCALE_UP_THRESHOLD=3
     • WORKFLOW_SCALE_DOWN_IDLE_TIME=30
   • Applies to all workflows.

More Controllable Parallel Branches

Execution Flow:

Start ─→ Unified Task Queue ─→ WorkerPool Scheduling
                          ├─→ Branch-1 Execution
                          └─→ Branch-2 Execution
                                  ↓
                            Aggregator
                                  ↓
                                  End

Improvements:
1. All tasks enter a single queue, managed by the Dispatcher.
2. WorkerPool auto-scales based on load.
3. ResponseCoordinator manages streaming outputs, ensuring correct order.

Example: Command Mechanism

from core.workflow.graph_engine.manager import GraphEngineManager

# Send stop command
GraphEngineManager.send_stop_command(
    task_id="workflow_task_123",
    reason="Emergency stop: resource limit exceeded"
)

Note: pause/resume functionality will be supported in future versions.

Example: GraphEngineLayer

GraphEngineLayer Example

FAQ

1. Is this release focused on performance?
   No. The focus is on stability, clarity, and correctness of parallel branches. Performance improvements are a secondary benefit.

2. What events can be subscribed to?

   • Graph-level: GraphRunStartedEvent, GraphRunSucceededEvent, GraphRunFailedEvent, GraphRunAbortedEvent
   • Node-level: NodeRunStartedEvent, NodeRunSucceededEvent, NodeRunFailedEvent, NodeRunRetryEvent
   • Container nodes: IterationRunStartedEvent, IterationRunNextEvent, IterationRunSucceededEvent, LoopRunStartedEvent, LoopRunNextEvent, LoopRunSucceededEvent
   • Streaming output: NodeRunStreamChunkEvent

3. How can I debug workflow execution?

   • Enable DEBUG=true to view detailed logs.
   • Use DebugLoggingLayer to record events.
   • Add custom monitoring via GraphEngineLayer.

Future Plans

This beta release is just the beginning. Upcoming improvements include:

• Debugging Tools: A visual interface to view execution states and variables in real time.
• Intelligent Scheduling: Optimize scheduling strategies using historical data.
• More Complete Command Support: Add Pause/Resume, breakpoint debugging.
• Human in the Loop: Support human intervention during execution.
• Subgraph Functionality: Enhance modularity and reusability.
• Multimodal Embedding: Support richer content types beyond text.

We look forward to your feedback and experiences to make the engine more practical.

Upgrade Guide

uv run flask transform-datasource-credentials

Docker Compose Deployments

1. Back up your customized docker-compose YAML file (optional)

cd docker
cp docker-compose.yaml docker-compose.yaml.$(date +%s).bak

1. Get the latest code from the main branch

git checkout 2.0.0-beta.1
git pull origin 2.0.0-beta.1

1. Stop the service. Please execute in the docker directory

docker compose down

1. Back up data

tar -cvf volumes-$(date +%s).tgz volumes

1. Upgrade services

docker compose up -d

1. Migrate data after the container starts

docker exec -it docker-api-1 uv run flask transform-datasource-credentials

Source Code Deployments

1. Stop the API server, Worker, and Web frontend Server.

2. Get the latest code from the release branch:

git checkout 2.0.0-beta.1

1. Update Python dependencies:

cd api
uv sync

1. Then, let's run the migration script:

uv run flask db upgrade
uv run flask transform-datasource-credentials

1. Finally, run the API server, Worker, and Web frontend Server again.