apache/airflow 3.1.0

Airflow 3.1.0

on GitHub

We are thrilled to announce the release of Apache Airflow 3.1.0, an update that puts humans at the center of data
workflows.

Read more about what 3.1.0 brings in https://airflow.apache.org/blog/airflow-3.1.0/

📦 PyPI: https://pypi.org/project/apache-airflow/3.1.0/
📚 Core Airflow Docs: https://airflow.apache.org/docs/apache-airflow/3.1.0/
📚 Task SDK Docs: https://airflow.apache.org/docs/task-sdk/1.1.0/
🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/3.1.0/release_notes.html
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.1.0

Apache Airflow 3.1.0 represents an extraordinary community effort, showcasing the vibrant ecosystem that drives this project forward with 163 contributors making this release possible across 1,400+ commits.

Screenshots

New Features

• Add Calendar and Gantt chart views to modern React UI with enhanced filtering (#54252, #51667)
• Add Python 3.13 support for Airflow runtime and dependencies (#46891)
• Add SQLAlchemy 2.0 support with various compatibility fixes for Python 3.13 (#52233, #52518, #54940)
• Add support for the psycopg3 postgres driver (#52976)
• Add ability to track & display user who triggers DAG runs (#51738, #53510, #54164, #55112)
• Add toggle for log grouping in task log viewer for better organization (#51146)
• Add tag filtering improvements with Any/All selection options (#51162)
• Add comprehensive filtering for DAG runs, task instances, and audit logs (#53652, #54210, #55082)
• Add XCom browsing with filtering and improved navigation (#54049)
• Add bulk task instance actions and deletion endpoints (#50443, #50165, #50235)
• Add DAG run deletion functionality through UI (#50368)
• Add test connection button for connection validation (#51055)
• Add hyperlink support for URLs in XCom values (#54288)
• Add pool column to task instances list and improve pool integration (#51185, #51031)
• Add drag-and-drop log grouping and improved log visualization (#51146)
• Add color support for XCom JSON display (#51323)
• Add configuration column to DAG runs page (#51270)
• Add enhanced note visibility and management in task headers (#51764, #54163)
• Introduce React plugin system (AIP-68) for modern UI extensions (#52255)
• Add support for external view plugins via iframe integration (#51003, #51889)
• Add dashboard integration capabilities for custom React apps (#54131, #54144)
• Add comprehensive plugin development tools and documentation (#53643)
• Implement complete HITL operator suite (HITLOperator, ApprovalOperator, HITLEntryOperator) for human decision workflows (#52868)
• Add HITL UI integration with role-based access and form handling (#53035)
• Add HITL API endpoints with filtering and query support (#53376, #53923)
• Add HITL utility functions for generating URLs to required actions page (#54827)
• Improve HITL user experience with bug fixes, UI enhancements, and data model consistency (#55463, #55539, #55575, #55546, #55543, #55536, #55535)
• Add ordering and filtering support for HITL details endpoints (#55217)
• Add "No Response Received" required action state (#55149)
• Add operator filter for HITL task instances (#54773)
• Implement deadline alert system for proactive DAG monitoring (AIP-86) (#53951, #53903, #53201, #55086)
• Add configurable reference points and notification callbacks (#50677, #50093)
• Add deadline calculation and tracking in DAG execution lifecycle (#51638, #50925)
• Add comprehensive UI translation support for 16 languages (#51266, #51038, #51219, #50929, #50981, #51793 and more)
• Add right-to-left (RTL) layout support for Arabic and Hebrew (#51376)
• Add language selection interface and browser preference detection (#51369)
• Add translation completeness validation and automated checks (#51166, #51131)
• Add calendar data API endpoints for DAG execution visualization (#52748)
• Add endpoint to watch DAG runs until completion (#51920, #53346)
• Add DAG run ID pattern search functionality (#52437)
• Add multi-sorting capabilities for improved data navigation (#53408)
• Add bulk connection deletion API and UI (#51201)
• Add task group detail pages across DAG runs (#50412, #50309)
• Add asset event tracking with last event timestamps (#50060, #50279)
• Add has_import_errors filter to Core API GET /dags endpoint (#54563)
• Add dag_version filter to get_dag_runs endpoint (#54882)
• Add pattern search for event log endpoint (#55114)
• Add dry_run support with consistent audit log handling (#55116)
• Add utility functions for generic filter counting (#54817)
• Add keyboard navigation for Grid view interface (#51784)
• Add improved error handling for plugin import failures (#49643)
• Add plugin validation in /plugins API with warnings for invalid plugins (#55673)
• Improve accessibility for screen readers and assistive technologies with proper language detection (#55839)
• Add enhanced variable management with upsert operations (#48547)
• Add favorites/pinning support for DAG dashboard organization (#51264)
• Add system theme support with automatic OS preference detection (#52649)
• Add hotkey shortcut to toggle between Grid and Graph views (#54667)
• Add queued DAGs filter button to DAGs page (#55052)
• Add DAG parsing duration visibility in UI (#54752)
• Add owner links support in DAG Header UI for better navigation (#50627)
• Add dag_display_name aliases for improved API consistency (#50332, #50065, #50014, #49933, #49641)
• Add enhanced search capabilities with SearchParamsKeys constants (#55218)
• Add ALL_DONE_MIN_ONE_SUCCESS trigger rule for flexible task dependencies (#53959)
• Add fail_when_dag_is_paused parameter to TriggerDagRunOperator for better control (#48214)
• Add XCom validation to prevent empty keys in XCom.set() and XCom.get() operations (#46929)
• Add collapsible plugin menu when multiple plugins are present (#55265)
• Add external view plugin categories (admin, browse, docs, user) (#52737)
• Add iframe plugins integration to DAG pages (#52795)
• Add plugin error display in UI with comprehensive error handling (#49643, #49436)
• Add collapsible failed task logs to prevent React error overflow (#54377)
• Add dynamic legend system for calendar view (#55155)
• Add React UI for Edge functionality (#53563)
• Add pending actions display to DAG UI (#55041)
• Add description field for filter parameters (#54903)
• Add Catalan language support to Airflow UI (#55013)
• Add Hungarian language support to Airflow UI (#54716)
• Add map_index validation in categorize_task_instances (#54791)
• Add Grid view UX improvements (#54846)
• Add HITL UX improvements for better user experience (#54990)
• Add async support for Notifiers (AIP-86) (#53831)
• Add filtering capabilities for tasks view (#54484)
• Add asset-based filtering support to DAG API endpoint (#54263)
• Add iframe plugins to navigation (#51706)
• Add RTL (right-to-left) layout support for Arabic and Hebrew (#51376)
• Add test connection button to UI (#51055)
• Add task instance bulk actions endpoint (#50443)
• Add connection bulk deletion functionality (#51201)
• Add pool column to task instances list (#51185)
• Add iframe_views to backend plugin support (#51003)
• Add keyboard shortcuts to clear and mark state for task instances and DAG runs (#50885)
• Add deadline relationship to DAG runs and deadline model (#50925, #50093)
• Add DAG run deletion UI (#50368)
• Add task instance deletion UI and endpoint (#50235, #50165)
• Switch all airflow logging to structlog (#52651, #55434, #55431, #55638)
• Add Filter Bar to Audit Log (#55487)
• Add Filters UI for Asset View (#54640)
• Update color palette and leverage Chakra semantic tokens (#53981, #55739)
• Improve calendar view UI with enhanced tooltips and visual fixes (#55476)

Bug Fixes

• Fix DAG list filtering to include QUEUED runs with null start_date (#52668)
• Fix XCom deletion failure for mapped task instances through bulk deletion API (#51850)
• Fix XCom deletion failure for mapped task instances (#54954)
• Fix task timeout handling within task SDK (#54089)
• Fix task instance tries API duplicate entries (#50597)
• Fix connection validation and type checking during construction (#54759)
• Fix mapped task instance index display in Task Instances tab (#55363)
• Fix Gantt chart state mismatch with Grid view (#55300)
• Fix Gantt chart status color display issues (#55296)
• Fix XCom mapping for dynamically-mapped task groups (#51556)
• Fix missing ti_successes and related metrics in Airflow 3.0 Task SDK (#55322)
• Fix bulk operation permissions for connection, pool and variable (#55278)
• Fix clearTaskInstances API: Restore include_past/future support on UI (#54416)
• Fix migration when XCom has NaN values (#53812)
• Fix HITL related UI schema generated by prek hooks (#55204)
• Fix consistent no-log handling for tasks with try_number=0 in API and UI (#55035)
• Fix timezone conversion in datetime trigger parameters (#54593)
• Fix audit log payload for DAG pause/unpause actions (#55091)
• Fix pushing None as an XCom value (#55080)
• Fix scheduler processing of cleared running tasks stuck in RESTARTING state (#55084)
• Fix XCom deletion failure for mapped task instances (#54954)
• Fix outgoing graph edges should exit opposite of incoming edges (#54789)
• Fix external links in Navigation buttons (#52220)
• Fix Error when viewing DAG details of a no longer configured bundle (#52086)
• Fix compatibility with new numpy and pandas versions (#52071)
• Fix connection recovery from URI when host has protocol (#51953)
• Fix last DAG run not showing on DAG listing (#51115)
• Fix task instance tries API returning duplicate entries (#50597)
• Fix Graph view vanishing and loading issues (#53886, #54756)
• Fix rendered template display formatting for better readability (#53657)
• Fix Grid view expand/collapse button functionality (#54257)
• Fix tooltip visibility and positioning issues (#53913)
• Fix grid keyboard navigation focus management (#54271)
• Fix plugin registration for invalid objects and middleware registration (#55264, #55399)
• Fix external links for plugins with undefined URL routes (#55221)
• Fix language display consistency and flag representation (#51560, #51177)
• Fix RTL layout rendering for Arabic and Hebrew interfaces (#51853)
• Fix graph export cropping when view is partial (#55012)
• Fix log viewer "Toggle Source" to hide only source fields, not all structured log fields (#55474)
• Output on stdout/stderr from within tasks is now filterable in the Sources list in the UI log view (#55508)
• Redact JWT tokens in task logs (#55499)
• Fix grid view to handle long task name (#55332)
• Allow slash characters in Variable keys similar to Airflow 2.x (#55324)
• Fix Grid cache invalidation for multi-run task operations (#55504)
• Fix Gantt chart rendering issues (#55554)
• Fix XCom access in DAG processor callbacks for notifiers (#55542)
• Fix alignment of arrows in RTL mode for right-to-left languages (#55619)
• Fix connection form extras not inferring correct type in UI (#55492)
• Fix incorrect log timestamps in UI when default_timezone is not UTC (#54431)
• Fix handling of priority_weight for DAG processor callbacks (#55436)
• Fix pointless requests from Gantt view when there is no Run ID (#55668)
• Ensure filename and lineno of logger calls are present in Task Logs (#55581)
• Fix DAG disappearing after callback execution in stale detection (#55698)
• Fix DB downgrade to Airflow 2 when fab tables exists (#55738)
• Fix UI stats endpoint causing dashboard loading issues (#55733)
• Fix unintended console output when DAG not found in serialized_dag table (#54972)
• Fix scheduler handling of orphaned tasks from Airflow 2 during upgrade (#55848)
• Fix logging format to respect existing configuration during upgrade to prevent unexpected log format changes (#55824)
• Fix Grid view crashes when DAG version information is missing (#55771)
• Fix compatibility for custom triggers migrating from Airflow 2.x that use synchronous connection calls (#55799)
• Fix DAG runs triggered from UI incorrectly marked as REST API triggers instead of UI triggers (#54650)
• Fix XCom API responses failing when encountering non-serializable objects by falling back to string representation (#55880)
• Fix asset queue display in UI showing incorrect timestamps for deleted queue events (#54652)
• Fix SQLite database migrations failing due to foreign key constraint handling (#55883)
• Fix DAG deserialization failure when using non-default weight_rule values like 'absolute' (#55906)
• Fix async connection retrieval in triggerer context preventing event loop blocking (#55812)
• Fix Airflow downgrade compatibility by handling serialized DAG format conversion from v3 to v2 (#55975)
• Fix 'All Log Levels' filter not working in task log viewer (#55851)
• Fix Grid view scrollbar overlapping issues on Firefox browser (#55960)
• Fix Gantt chart misalignment with Grid view layout (#55995)
• Fix Grid view task names being extremely collapsed and unreadable when displaying many DAG runs (#55997)
• Fix LocalExecutor race condition where tasks could start before database state was committed (#56010)

Miscellaneous

• Move secrets masker to shared distribution for better modularity (#54449)
• Move email notifications from scheduler to DAG processor for better architecture (#55238)
• Add graph UI load optimization with latest run info endpoint (#53429)
• Optimize UI bundle size by moving translations to dynamic loading (#51735)
• Relocate Task SDK components for improved separation (#55174, #54795)
• Refactor trigger rule utilities and weight rule consolidation (#54797, #53393)
• Remove deprecated Airflow 2.x modules and legacy imports (#50482)
• Clean up unused code and improve module organization (#52176, #52173, #53031)
• Add SQLAlchemy 2.0 CI support for future compatibility (#52233)
• Improve test fixtures and SDK communication testing (#54795, #50603)
• Add translation completeness linting and validation tools (#51166)
• Upgrade to latest versions of important dependencies (#55350)
• Move webserver configuration options to API section (#50693, #50656)
• Improve DAG bundle handling and versioning support (#47592)
• Add database management CLI tools for external database operations (#50657)
• Add comprehensive HITL operator documentation and examples (#54618)
• Add guards for registering middlewares from plugins (#55399)
• Optimize Gantt group expansion with de-bouncing and deferred rendering (#55334)
• Differentiate between triggers and watchers currently running for better visibility (#55376)
• Removed unused config: dag_stale_not_seen_duration (#55601, #55684)
• Update UI's query client strategy for improved performance (#55528)
• Unify datetime format across the UI for consistency (#55572)
• Mark React Apps as Experimental for Airflow 3.1 release (#55478)
• Improve OOM error messaging for clearer task failure diagnosis (#55602)
• Display responder username for better audit trail in HITL workflows (#55509)
• The constraint file do not contain developer dependencies anymore (#53631)
• Add hyperlinks to dag_id column in DAG Runs and Task Instances pages for better navigation (#55648)
• Add responsive web design (RWD) support to Grid view (#55745)

Doc Only Changes

• Add comprehensive Human-in-the-Loop operator tutorial and examples (#54618)
• Add deadline alerts configuration and usage documentation (#53727)
• Make term Dag consistent in docs task-sdk (#55100)
• Add migration guide for upgrading from legacy SLA functionality to deadline alerts (#55743)
• Add DAG bundles triggerer limitation documentation (#55232)
• Add deadline alerts usage guides and best practices (#53727)
• Remove --preview flag from ruff check instructions for Airflow 3 upgrade path (#55516)
• Add documentation for context parameter (#55377)