github apache/airflow 2.2.0
Apache Airflow 2.2.0

New Features

  • AIP-39: Add (customizable) Timetable class to Airflow for richer scheduling behaviour (#15397, #16030,
    #16352, #17030, #17122, #17414, #17552, #17755, #17989, #18084, #18088, #18244, #18266, #18420, #18434,
    #18421, #18475, #18499, #18573, #18522, #18729, #18706, #18742, #18786, #18804)
  • AIP-40: Add Deferrable "Async" Tasks (#15389, #17564, #17565, #17601, #17745, #17747, #17748, #17875,
    #17876, #18129, #18210, #18214, #18552, #18728, #18414)
  • Add a Docker Taskflow decorator (#15330, #18739)
  • Add Airflow Standalone command (#15826)
  • Display alert messages on dashboard from local settings (#18284)
  • Advanced Params using json-schema (#17100)
  • Ability to test connections from UI or API (#15795, #18750)
  • Add Next Run to UI (#17732)
  • Add default weight rule configuration option (#18627)
  • Add a calendar field to choose the execution date of the DAG when triggering it (#16141)
  • Allow setting specific cwd for BashOperator (#17751)
  • Show import errors in DAG views (#17818)
  • Add pre/post execution hooks Experimental
  • Added table to view providers in Airflow ui under admin tab (#15385)
  • Adds secrets backend/logging/auth information to provider yaml (#17625)
  • Add date format filters to Jinja environment (#17451)
  • Introduce RESTARTING state (#16681)
  • Webserver: Unpause DAG on manual trigger (#16569)
  • API endpoint to create new user (#16609)
  • Add insert_args for support transfer replace (#15825)
  • Add recursive flag to glob in filesystem sensor (#16894)
  • Add conn to jinja template context (#16686)
  • Add default_args for TaskGroup (#16557)
  • Allow adding duplicate connections from UI (#15574)
  • Allow specifying multiple URLs via the CORS config option (#17941)
  • Implement API endpoint for DAG deletion (#17980)
  • Add DAG run endpoint for marking a dagrun success or failed(#17839)
  • Add support for kinit options [-f|-F] and [-a|-A] (#17816)
  • Queue support for DaskExecutor using Dask Worker Resources (#16829, #18720)
  • Make auto refresh interval configurable (#18107)

Improvements

  • Small improvements for Airflow UI (#18715, #18795)
  • Rename processor_poll_interval to scheduler_idle_sleep_time (#18704)
  • Check the allowed values for the logging level (#18651)
  • Fix error on triggering a dag that doesn't exist using dagrun_conf (#18655)
  • Add muldelete action to TaskInstanceModelView (#18438)
  • Avoid importing DAGs during clean DB installation (#18450)
  • Require can_edit on DAG privileges to modify TaskInstances and DagRuns (#16634)
  • Make Kubernetes job description fit on one log line (#18377)
  • Always draw borders if task instance state is null or undefined (#18033)
  • Inclusive Language (#18349)
  • Improved log handling for zombie tasks (#18277)
  • Adding Variable.update method and improving detection of variable key collisions (#18159)
  • Add note about params on trigger DAG page (#18166)
  • Change TaskInstance and TaskReschedule PK from execution_date to run_id (#17719)
  • Adding TaskGroup support in BaseOperator.chain() (#17456)
  • Advises the kernel to not cache log files generated by Airflow (#18054)
  • Allow filtering DAGS by tags in the REST API (#18090)
  • Optimize imports of Providers Manager (#18052)
  • Adds capability of Warnings for incompatible community providers (#18020)
  • Serialize the template_ext attribute to show it in UI (#17985)
  • Add robots.txt and X-Robots-Tag header (#17946)
  • Refactor BranchDayOfWeekOperator, DayOfWeekSensor (#17940)
  • Update error message to guide the user into self-help mostly (#17929)
  • Update to Celery 5 (#17397)
  • Avoid redirect loop for users with no permissions (#17838)
  • Improve graph view load time for dags with open groups (#17821)
  • Add links to provider's documentation (#17736)
  • Remove Marshmallow schema warnings (#17753)
  • Rename none_failed_or_skipped by none_failed_min_one_success trigger rule (#17683)
  • Remove [core] store_dag_code & use DB to get Dag Code (#16342)
  • Rename task_concurrency to max_active_tis_per_dag (#17708)
  • Import Hooks lazily individually in providers manager (#17682)
  • Adding support for multiple task-ids in the external task sensor (#17339)
  • Replace execution_date with run_id in airflow tasks run command (#16666)
  • Avoid endless redirect loop when user has no roles (#17613)
  • Make output from users cli command more consistent (#17642)
  • Open relative extra links in place (#17477)
  • Move worker_log_server_port option to the logging section (#17621)
  • Use gunicorn to serve logs generated by worker (#17591)
  • Improve validation of Group id (#17578)
  • Simplify 404 page (#17501)
  • Add XCom.clear so it's hookable in custom XCom backend (#17405)
  • Add deprecation notice for SubDagOperator (#17488)
  • Support DAGS folder being in different location on scheduler and runners (#16860)
  • Remove /dagrun/create and disable edit form generated by F.A.B (#17376)
  • Enable specifying dictionary paths in template_fields_renderers (#17321)
  • error early if virtualenv is missing (#15788)
  • Handle connection parameters added to Extra and custom fields (#17269)
  • Fix airflow celery stop to accept the pid file. (#17278)
  • Remove DAG refresh buttons (#17263)
  • Deprecate dummy trigger rule in favor of always (#17144)
  • Be verbose about failure to import airflow_local_settings (#17195)
  • Include exit code in AirflowException str when BashOperator fails. (#17151)
  • Adding EdgeModifier support for chain() (#17099)
  • Only allows supported field types to be used in custom connections (#17194)
  • Secrets backend failover (#16404)
  • Warn on Webserver when using SQLite or SequentialExecutor (#17133)
  • Extend init_containers defined in pod_override (#17537)
  • Client-side filter dag dependencies (#16253)
  • Improve executor validation in CLI (#17071)
  • Prevent running airflow db init/upgrade migrations and setup in parallel. (#17078)
  • Update chain() and cross_downstream() to support XComArgs (#16732)
  • Improve graph view refresh (#16696)
  • When a task instance fails with exception, log it (#16805)
  • Set process title for serve-logs and LocalExecutor (#16644)
  • Rename test_cycle to check_cycle (#16617)
  • Set process title for Worker when using LocalExecutor (#16623)
  • Add schema as DbApiHook instance attribute (#16521, #17423)
  • Improve compatibility with MSSQL (#9973)
  • Add transparency for unsupported connection type (#16220)
  • Call resource based fab methods (#16190)
  • Format more dates with timezone (#16129)
  • Replace deprecated dag.sub_dag with dag.partial_subset (#16179)
  • Treat AirflowSensorTimeout as immediate failure without retrying (#12058)
  • Marking success/failed automatically clears failed downstream tasks (#13037)
  • Add close/open indicator for import dag errors (#16073)
  • Add collapsible import errors (#16072)
  • Always return a response in TI's action_clear view (#15980)
  • Add cli command to delete user by email (#15873)
  • Use resource and action names for FAB permissions (#16410)
  • Rename DAG concurrency ([core] dag_concurrency) settings for easier understanding (#16267, #18730)
  • Calendar UI improvements (#16226)
  • Refactor: SKIPPED should not be logged again as SUCCESS (#14822)
  • Remove version limits for dnspython (#18046, #18162)
  • Accept custom run ID in TriggerDagRunOperator (#18788)

Bug Fixes

  • Make REST API patch user endpoint work the same way as the UI (#18757)
  • Properly set start_date for cleared tasks (#18708)
  • Ensure task_instance exists before running update on its state(REST API) (#18642)
  • Make AirflowDateTimePickerWidget a required field (#18602)
  • Retry deadlocked transactions on deleting old rendered task fields (#18616)
  • Fix retry_exponential_backoff divide by zero error when retry delay is zero (#17003)
  • Improve how UI handles datetimes (#18611, #18700)
  • Bugfix: dag_bag.get_dag should return None, not raise exception (#18554)
  • Only show the task modal if it is a valid instance (#18570)
  • Fix accessing rendered {{ task.x }} attributes from within templates (#18516)
  • Add missing email type of connection (#18502)
  • Don't use flash for "same-page" UI messages. (#18462)
  • Fix task group tooltip (#18406)
  • Properly fix dagrun update state endpoint (#18370)
  • Properly handle ti state difference between executor and scheduler (#17819)
  • Fix stuck "queued" tasks in KubernetesExecutor (#18152)
  • Don't permanently add zip DAGs to sys.path (#18384)
  • Fix random deadlocks in MSSQL database (#18362)
  • Deactivating DAGs which have been removed from files (#17121)
  • When syncing dags to db remove dag_tag rows that are now unused (#8231)
  • Graceful scheduler shutdown on error (#18092)
  • Fix mini scheduler not respecting wait_for_downstream dep (#18338)
  • Pass exception to run_finished_callback for Debug Executor (#17983)
  • Make XCom.get_one return full, not abbreviated values (#18274)
  • Use try/except when closing temporary file in task_runner (#18269)
  • show next run if not none (#18273)
  • Fix DB session handling in XCom.set (#18240)
  • Sort adopted tasks in _check_for_stalled_adopted_tasks method (#18208)
  • Fix external_executor_id not being set for manually run jobs (#17207)
  • Fix deleting of zipped Dags in Serialized Dag Table (#18243)
  • Return explicit error on user-add for duplicated email (#18224)
  • Remove loading dots even when last run data is empty (#18230)
  • Swap dag import error dropdown icons (#18207)
  • Automatically create section when migrating config (#16814)
  • Set encoding to utf-8 by default while reading task logs (#17965)
  • Set task state to failed when pod is DELETED while running (#18095)
  • Apply parent dag permissions to subdags (#18160)
  • Change id collation for MySQL to case-sensitive (#18072)
  • Logs task launch exception in StandardTaskRunner (#17967)
  • Applied permissions to self._error_file (#15947)
  • Hide variable import form if user lacks permission (#18000)
  • Fix blank dag dependencies view (#17990)
  • Add missing menu access for dag dependencies and configurations pages (#17450)
  • Fix passing Jinja templates in DateTimeSensor (#17959)
  • Fixing bug which restricted the visibility of ImportErrors (#17924)
  • Fix grammar in traceback.html (#17942)
  • Fix DagRunState enum query for MySQLdb driver (#17886)
  • Fixed button size in "Actions" group. (#17902)
  • Fix log links on graph TI modal (#17862)
  • Only show import errors for DAGs a user can access (#17835)
  • Show all import_errors from zip files (#17759)
  • fix EXTRA_LOGGER_NAMES param and related docs (#17808)
  • Use one interpreter for Airflow and gunicorn (#17805)
  • Fix: Mysql 5.7 id utf8mb3 (#14535)
  • Fix dag_processing.last_duration metric random holes (#17769)
  • Automatically use utf8mb3_general_ci collation for MySQL (#17729)
  • Fix wrong query on running tis (#17631)
  • Add root to tree refresh url (#17633)
  • Do not delete running DAG from the UI (#17630)
  • Rescue if a DagRun's DAG was removed from db (#17544)
  • fix: filter condition of TaskInstance does not work #17535 (#17548)
  • Regression on pid reset to allow task start after heartbeat (#17333)
  • Dont use TaskInstance in CeleryExecutor.trigger_tasks (#16248)
  • Remove locks for upgrades in MSSQL (#17213)
  • Create virtualenv via python call (#17156)
  • Ensure a DAG is acyclic when running DAG.cli() (#17105)
  • Translate non-ascii characters (#17057)
  • Change the logic of None comparison in model_list template (#16893)
  • Have UI and POST /task_instances_state API endpoint have same behaviour (#16539)
  • ensure task is skipped if missing sla (#16719)
  • Fix direct use of cached_property module (#16710)
  • Fix TI success confirm page (#16650)
  • Cattrs 1.7.0 released by the end of May 2021 break lineage usage (#16173)
  • Fix Celery executor getting stuck randomly because of reset_signals in multiprocessing (#15989)
  • Modify return value check in python virtualenv jinja template (#16049)
  • Fix dag dependency search (#15924)
  • Make custom JSON encoder support Decimal (#16383)
  • Bugfix: Allow clearing tasks with just dag_id and empty subdir (#16513)
  • Convert port value to a number before calling test connection (#16497)
  • Handle missing/null serialized DAG dependencies (#16393)
  • Correctly set dag.fileloc when using the @dag decorator (#16384)
  • Fix TI success/failure links (#16233)
  • Correctly implement autocomplete early return in airflow/www/views.py (#15940)
  • Backport fix to allow pickling of Loggers to Python 3.6 (#18798)
  • Fix bug that Backfill job fail to run when there are tasks run into reschedule state (#17305, #18806)

Doc only changes

  • Update dagbag_size documentation (#18824)
  • Update documentation about bundle extras (#18828)
  • Fix wrong Postgres search_path set up instructions (#17600)
  • Remove AIRFLOW_GID from Docker images (#18747)
  • Improve error message for BranchPythonOperator when no task_id to follow (#18471)
  • Improve guidance to users telling them what to do on import timeout (#18478)
  • Explain scheduler fine-tuning better (#18356)
  • Added example JSON for airflow pools import (#18376)
  • Add sla_miss_callback section to the documentation (#18305)
  • Explain sentry default environment variable for subprocess hook (#18346)
  • Refactor installation pages (#18282)
  • Improves installing from sources pages for all components (#18251)
  • Doc: Use closer.lua script for downloading sources (#18179)
  • Improves quick-start docker-compose warnings and documentation (#18164)
  • Production-level support for MSSQL (#18382)
  • Update non-working example in documentation (#18067)
  • Improve the description of how to handle dynamic task generation (#17963)
  • Remove default_args pattern + added get_current_context() use for Core Airflow example DAGs (#16866)
  • Update max_tis_per_query to better render on the webpage (#17971)
  • Adds Github Oauth example with team based authorization (#17896)
  • Update docker.rst (#17882)
  • Example xcom update (#17749)
  • Add doc warning about connections added via envvars (#17915)
  • fix wrong documents around upgrade-check.rst (#17903)
  • Add Brent to Committers list (#17873)
  • Improves documentation about modules management (#17757)
  • Remove deprecated metrics from metrics.rst (#17772)
  • Make sure "production-readiness" of docker-compose is well explained (#17731)
  • Doc: Update Upgrade to v2 docs with Airflow 1.10.x EOL dates (#17710)
  • Doc: Replace deprecated param from docstrings (#17709)
  • docs(dagowner): describe dag owner more carefully (#17699)
  • docs(impersonation): update note so avoid misintrepretation (#17701)
  • Docs: Make DAG.is_active read-only in API (#17667)
  • Improve cross-links to operators and hooks references (#17622)
  • Update documentation regarding Python 3.9 support (#17611)
  • Fix MySQL database character set instruction (#17603)
  • Document overriding XCom.clear for data lifecycle management (#17589)
  • Path correction in docs for airflow core (#17567)
  • docs(celery): reworded, add actual multiple queues example (#17541)
  • Doc: Add FAQ to speed up parsing with tons of dag files (#17519)
  • Improve image building documentation for new users (#17409)
  • Doc: Strip unnecessary arguments from MariaDB JIRA URL (#17296)
  • Update warning about MariaDB and multiple schedulers (#17287)
  • Doc: Recommend using same configs on all Airflow components (#17146)
  • Move docs about masking to a new page (#17007)
  • Docs: Suggest use of Env vars instead of Airflow Vars in best practises doc (#16926)
  • Docs: Better description for pod_template_file (#16861)
  • Add Aneesh Joseph as Airflow Committer (#16835)
  • Docs: Added new pipeline example for the tutorial docs (#16548)
  • Remove upstart from docs (#16672)
  • Add new committers: Jed and TP (#16671)
  • Docs: Fix flask-ouathlib to flask-oauthlib in Upgrading docs (#16320)
  • Docs: Fix creating a connection docs (#16312)
  • Docs: Fix url for Elasticsearch (#16275)
  • Small improvements for README.md files (#16244)
  • Fix docs for dag_concurrency (#16177)
  • Check syntatic correctness for code-snippets (#16005)
  • Improvements for Docker Image docs (#14843)
  • Add proper link for wheel packages in docs. (#15999)
  • Add Docs for default_pool slots (#15997)
  • Add memory usage warning in quick-start documentation (#15967)
  • Update example KubernetesExecutor git-sync pod template file (#15904)
  • Docs: Fix Taskflow API docs (#16574)
  • Added new pipeline example for the tutorial docs (#16084)
  • Updating the DAG docstring to include render_template_as_native_obj (#16534)
  • Update docs on setting up SMTP (#16523)
  • Docs: Fix API verb from POST to PATCH (#16511)

Misc/Internal

  • Renaming variables to be consistent with code logic (#18685)
  • Simplify strings previously split across lines (#18679)
  • fix exception string of BranchPythonOperator (#18623)
  • Add multiple roles when creating users (#18617)
  • Move FABs base Security Manager into Airflow. (#16647)
  • Remove unnecessary css state colors (#18461)
  • Update boto3 to <1.19 (#18389)
  • Improve coverage for airflow.security.kerberos module (#18258)
  • Fix kinesis test (#18337)
  • Fix provider test acessing importlib-resources (#18228)
  • Silence warnings in tests from using SubDagOperator (#18275)
  • Fix usage of range(len()) to enumerate (#18174)
  • Test coverage on the autocomplete view (#15943)
  • Update version added fields in airflow/config_templates/config.yml (#18128)
  • Add "packaging" to core requirements (#18122)
  • Adds LoggingMixins to BaseTrigger (#18106)
  • Limit colorlog version (6.x is incompatible) (#18099)
  • Fix building docs in main builds (#18035)
  • Remove upper-limit on tenacity (#17593)
  • Remove redundant numpy dependency (#17594)
  • Bump mysql-connector-python to latest version (#17596)
  • Make pandas an optional core dependency (#17575)
  • Add more typing to airflow.utils.helpers (#15582)
  • Chore: Some code cleanup in airflow/utils/db.py (#17090)
  • Refactor: Remove processor_factory from DAG processing (#16659)
  • Remove AbstractDagFileProcessorProcess from dag processing (#16816)
  • Update TaskGroup typing (#16811)
  • Update click to 8.x (#16779)
  • Remove remaining Pylint disables (#16760)
  • Remove duplicated try, there is already a try in create_session (#16701)
  • Removes pylint from our toolchain (#16682)
  • Refactor usage of unneeded function call (#16653)
  • Add type annotations to setup.py (#16658)
  • Remove SQLAlchemy <1.4 constraint (#16630) (Note: our dependencies still have a requirement on <1.4)
  • Refactor dag.clear method (#16086)
  • Use DAG_ACTIONS constant (#16232)
  • Use updated _get_all_non_dag_permissions method (#16317)
  • Add updated-name wrappers for built-in FAB methods (#16077)
  • Pins docutils to <0.17 until breaking behaviour is fixed (#16133)
  • Remove TaskInstance.log_filepath attribute (#15217)
  • Removes unnecessary function call in airflow/www/app.py (#15956)
  • Move plyvel to google provider extra (#15812)
  • Update permission migrations to use new naming scheme (#16400)
  • Use resource and action names for FAB (#16380)
  • Swap out calls to find_permission_view_menu for get_permission wrapper (#16377)
  • Fix deprecated default for fab_logging_level to WARNING (#18783)
  • Allow running tasks from UI when using CeleryKubernetesExecutor (#18441)
latest release: constraints-2.2.0
12 days ago