apache/airflow 2.2.0

Apache Airflow 2.2.0

on GitHub

New Features

• AIP-39: Add (customizable) Timetable class to Airflow for richer scheduling behaviour (#15397, #16030,
  #16352, #17030, #17122, #17414, #17552, #17755, #17989, #18084, #18088, #18244, #18266, #18420, #18434,
  #18421, #18475, #18499, #18573, #18522, #18729, #18706, #18742, #18786, #18804)
• AIP-40: Add Deferrable "Async" Tasks (#15389, #17564, #17565, #17601, #17745, #17747, #17748, #17875,
  #17876, #18129, #18210, #18214, #18552, #18728, #18414)
• Add a Docker Taskflow decorator (#15330, #18739)
• Add Airflow Standalone command (#15826)
• Display alert messages on dashboard from local settings (#18284)
• Advanced Params using json-schema (#17100)
• Ability to test connections from UI or API (#15795, #18750)
• Add Next Run to UI (#17732)
• Add default weight rule configuration option (#18627)
• Add a calendar field to choose the execution date of the DAG when triggering it (#16141)
• Allow setting specific cwd for BashOperator (#17751)
• Show import errors in DAG views (#17818)
• Add pre/post execution hooks [Experimental] (#17576)
• Added table to view providers in Airflow ui under admin tab (#15385)
• Adds secrets backend/logging/auth information to provider yaml (#17625)
• Add date format filters to Jinja environment (#17451)
• Introduce RESTARTING state (#16681)
• Webserver: Unpause DAG on manual trigger (#16569)
• API endpoint to create new user (#16609)
• Add insert_args for support transfer replace (#15825)
• Add recursive flag to glob in filesystem sensor (#16894)
• Add conn to jinja template context (#16686)
• Add default_args for TaskGroup (#16557)
• Allow adding duplicate connections from UI (#15574)
• Allow specifying multiple URLs via the CORS config option (#17941)
• Implement API endpoint for DAG deletion (#17980)
• Add DAG run endpoint for marking a dagrun success or failed(#17839)
• Add support for kinit options [-f|-F] and [-a|-A] (#17816)
• Queue support for DaskExecutor using Dask Worker Resources (#16829, #18720)
• Make auto refresh interval configurable (#18107)

Improvements

• Small improvements for Airflow UI (#18715, #18795)
• Rename processor_poll_interval to scheduler_idle_sleep_time (#18704)
• Check the allowed values for the logging level (#18651)
• Fix error on triggering a dag that doesn't exist using dagrun_conf (#18655)
• Add muldelete action to TaskInstanceModelView (#18438)
• Avoid importing DAGs during clean DB installation (#18450)
• Require can_edit on DAG privileges to modify TaskInstances and DagRuns (#16634)
• Make Kubernetes job description fit on one log line (#18377)
• Always draw borders if task instance state is null or undefined (#18033)
• Inclusive Language (#18349)
• Improved log handling for zombie tasks (#18277)
• Adding Variable.update method and improving detection of variable key collisions (#18159)
• Add note about params on trigger DAG page (#18166)
• Change TaskInstance and TaskReschedule PK from execution_date to run_id (#17719)
• Adding TaskGroup support in BaseOperator.chain() (#17456)
• Advises the kernel to not cache log files generated by Airflow (#18054)
• Allow filtering DAGS by tags in the REST API (#18090)
• Optimize imports of Providers Manager (#18052)
• Adds capability of Warnings for incompatible community providers (#18020)
• Serialize the template_ext attribute to show it in UI (#17985)
• Add robots.txt and X-Robots-Tag header (#17946)
• Refactor BranchDayOfWeekOperator, DayOfWeekSensor (#17940)
• Update error message to guide the user into self-help mostly (#17929)
• Update to Celery 5 (#17397)
• Avoid redirect loop for users with no permissions (#17838)
• Improve graph view load time for dags with open groups (#17821)
• Add links to provider's documentation (#17736)
• Remove Marshmallow schema warnings (#17753)
• Rename none_failed_or_skipped by none_failed_min_one_success trigger rule (#17683)
• Remove [core] store_dag_code & use DB to get Dag Code (#16342)
• Rename task_concurrency to max_active_tis_per_dag (#17708)
• Import Hooks lazily individually in providers manager (#17682)
• Adding support for multiple task-ids in the external task sensor (#17339)
• Replace execution_date with run_id in airflow tasks run command (#16666)
• Avoid endless redirect loop when user has no roles (#17613)
• Make output from users cli command more consistent (#17642)
• Open relative extra links in place (#17477)
• Move worker_log_server_port option to the logging section (#17621)
• Use gunicorn to serve logs generated by worker (#17591)
• Improve validation of Group id (#17578)
• Simplify 404 page (#17501)
• Add XCom.clear so it's hookable in custom XCom backend (#17405)
• Add deprecation notice for SubDagOperator (#17488)
• Support DAGS folder being in different location on scheduler and runners (#16860)
• Remove /dagrun/create and disable edit form generated by F.A.B (#17376)
• Enable specifying dictionary paths in template_fields_renderers (#17321)
• error early if virtualenv is missing (#15788)
• Handle connection parameters added to Extra and custom fields (#17269)
• Fix airflow celery stop to accept the pid file. (#17278)
• Remove DAG refresh buttons (#17263)
• Deprecate dummy trigger rule in favor of always (#17144)
• Be verbose about failure to import airflow_local_settings (#17195)
• Include exit code in AirflowException str when BashOperator fails. (#17151)
• Adding EdgeModifier support for chain() (#17099)
• Only allows supported field types to be used in custom connections (#17194)
• Secrets backend failover (#16404)
• Warn on Webserver when using SQLite or SequentialExecutor (#17133)
• Extend init_containers defined in pod_override (#17537)
• Client-side filter dag dependencies (#16253)
• Improve executor validation in CLI (#17071)
• Prevent running airflow db init/upgrade migrations and setup in parallel. (#17078)
• Update chain() and cross_downstream() to support XComArgs (#16732)
• Improve graph view refresh (#16696)
• When a task instance fails with exception, log it (#16805)
• Set process title for serve-logs and LocalExecutor (#16644)
• Rename test_cycle to check_cycle (#16617)
• Set process title for Worker when using LocalExecutor (#16623)
• Add schema as DbApiHook instance attribute (#16521, #17423)
• Improve compatibility with MSSQL (#9973)
• Add transparency for unsupported connection type (#16220)
• Call resource based fab methods (#16190)
• Format more dates with timezone (#16129)
• Replace deprecated dag.sub_dag with dag.partial_subset (#16179)
• Treat AirflowSensorTimeout as immediate failure without retrying (#12058)
• Marking success/failed automatically clears failed downstream tasks (#13037)
• Add close/open indicator for import dag errors (#16073)
• Add collapsible import errors (#16072)
• Always return a response in TI's action_clear view (#15980)
• Add cli command to delete user by email (#15873)
• Use resource and action names for FAB permissions (#16410)
• Rename DAG concurrency ([core] dag_concurrency) settings for easier understanding (#16267, #18730)
• Calendar UI improvements (#16226)
• Refactor: SKIPPED should not be logged again as SUCCESS (#14822)
• Remove version limits for dnspython (#18046, #18162)
• Accept custom run ID in TriggerDagRunOperator (#18788)

Bug Fixes

• Make REST API patch user endpoint work the same way as the UI (#18757)
• Properly set start_date for cleared tasks (#18708)
• Ensure task_instance exists before running update on its state(REST API) (#18642)
• Make AirflowDateTimePickerWidget a required field (#18602)
• Retry deadlocked transactions on deleting old rendered task fields (#18616)
• Fix retry_exponential_backoff divide by zero error when retry delay is zero (#17003)
• Improve how UI handles datetimes (#18611, #18700)
• Bugfix: dag_bag.get_dag should return None, not raise exception (#18554)
• Only show the task modal if it is a valid instance (#18570)
• Fix accessing rendered {{ task.x }} attributes from within templates (#18516)
• Add missing email type of connection (#18502)
• Don't use flash for "same-page" UI messages. (#18462)
• Fix task group tooltip (#18406)
• Properly fix dagrun update state endpoint (#18370)
• Properly handle ti state difference between executor and scheduler (#17819)
• Fix stuck "queued" tasks in KubernetesExecutor (#18152)
• Don't permanently add zip DAGs to sys.path (#18384)
• Fix random deadlocks in MSSQL database (#18362)
• Deactivating DAGs which have been removed from files (#17121)
• When syncing dags to db remove dag_tag rows that are now unused (#8231)
• Graceful scheduler shutdown on error (#18092)
• Fix mini scheduler not respecting wait_for_downstream dep (#18338)
• Pass exception to run_finished_callback for Debug Executor (#17983)
• Make XCom.get_one return full, not abbreviated values (#18274)
• Use try/except when closing temporary file in task_runner (#18269)
• show next run if not none (#18273)
• Fix DB session handling in XCom.set (#18240)
• Sort adopted tasks in _check_for_stalled_adopted_tasks method (#18208)
• Fix external_executor_id not being set for manually run jobs (#17207)
• Fix deleting of zipped Dags in Serialized Dag Table (#18243)
• Return explicit error on user-add for duplicated email (#18224)
• Remove loading dots even when last run data is empty (#18230)
• Swap dag import error dropdown icons (#18207)
• Automatically create section when migrating config (#16814)
• Set encoding to utf-8 by default while reading task logs (#17965)
• Set task state to failed when pod is DELETED while running (#18095)
• Apply parent dag permissions to subdags (#18160)
• Change id collation for MySQL to case-sensitive (#18072)
• Logs task launch exception in StandardTaskRunner (#17967)
• Applied permissions to self._error_file (#15947)
• Hide variable import form if user lacks permission (#18000)
• Fix blank dag dependencies view (#17990)
• Add missing menu access for dag dependencies and configurations pages (#17450)
• Fix passing Jinja templates in DateTimeSensor (#17959)
• Fixing bug which restricted the visibility of ImportErrors (#17924)
• Fix grammar in traceback.html (#17942)
• Fix DagRunState enum query for MySQLdb driver (#17886)
• Fixed button size in "Actions" group. (#17902)
• Fix log links on graph TI modal (#17862)
• Only show import errors for DAGs a user can access (#17835)
• Show all import_errors from zip files (#17759)
• fix EXTRA_LOGGER_NAMES param and related docs (#17808)
• Use one interpreter for Airflow and gunicorn (#17805)
• Fix: Mysql 5.7 id utf8mb3 (#14535)
• Fix dag_processing.last_duration metric random holes (#17769)
• Automatically use utf8mb3_general_ci collation for MySQL (#17729)
• Fix wrong query on running tis (#17631)
• Add root to tree refresh url (#17633)
• Do not delete running DAG from the UI (#17630)
• Rescue if a DagRun's DAG was removed from db (#17544)
• fix: filter condition of TaskInstance does not work #17535 (#17548)
• Regression on pid reset to allow task start after heartbeat (#17333)
• Dont use TaskInstance in CeleryExecutor.trigger_tasks (#16248)
• Remove locks for upgrades in MSSQL (#17213)
• Create virtualenv via python call (#17156)
• Ensure a DAG is acyclic when running DAG.cli() (#17105)
• Translate non-ascii characters (#17057)
• Change the logic of None comparison in model_list template (#16893)
• Have UI and POST /task_instances_state API endpoint have same behaviour (#16539)
• ensure task is skipped if missing sla (#16719)
• Fix direct use of cached_property module (#16710)
• Fix TI success confirm page (#16650)
• Cattrs 1.7.0 released by the end of May 2021 break lineage usage (#16173)
• Fix Celery executor getting stuck randomly because of reset_signals in multiprocessing (#15989)
• Modify return value check in python virtualenv jinja template (#16049)
• Fix dag dependency search (#15924)
• Make custom JSON encoder support Decimal (#16383)
• Bugfix: Allow clearing tasks with just dag_id and empty subdir (#16513)
• Convert port value to a number before calling test connection (#16497)
• Handle missing/null serialized DAG dependencies (#16393)
• Correctly set dag.fileloc when using the @dag decorator (#16384)
• Fix TI success/failure links (#16233)
• Correctly implement autocomplete early return in airflow/www/views.py (#15940)
• Backport fix to allow pickling of Loggers to Python 3.6 (#18798)
• Fix bug that Backfill job fail to run when there are tasks run into reschedule state (#17305, #18806)

Doc only changes

• Update dagbag_size documentation (#18824)
• Update documentation about bundle extras (#18828)
• Fix wrong Postgres search_path set up instructions (#17600)
• Remove AIRFLOW_GID from Docker images (#18747)
• Improve error message for BranchPythonOperator when no task_id to follow (#18471)
• Improve guidance to users telling them what to do on import timeout (#18478)
• Explain scheduler fine-tuning better (#18356)
• Added example JSON for airflow pools import (#18376)
• Add sla_miss_callback section to the documentation (#18305)
• Explain sentry default environment variable for subprocess hook (#18346)
• Refactor installation pages (#18282)
• Improves installing from sources pages for all components (#18251)
• Doc: Use closer.lua script for downloading sources (#18179)
• Improves quick-start docker-compose warnings and documentation (#18164)
• Production-level support for MSSQL (#18382)
• Update non-working example in documentation (#18067)
• Improve the description of how to handle dynamic task generation (#17963)
• Remove default_args pattern + added get_current_context() use for Core Airflow example DAGs (#16866)
• Update max_tis_per_query to better render on the webpage (#17971)
• Adds Github Oauth example with team based authorization (#17896)
• Update docker.rst (#17882)
• Example xcom update (#17749)
• Add doc warning about connections added via envvars (#17915)
• fix wrong documents around upgrade-check.rst (#17903)
• Add Brent to Committers list (#17873)
• Improves documentation about modules management (#17757)
• Remove deprecated metrics from metrics.rst (#17772)
• Make sure "production-readiness" of docker-compose is well explained (#17731)
• Doc: Update Upgrade to v2 docs with Airflow 1.10.x EOL dates (#17710)
• Doc: Replace deprecated param from docstrings (#17709)
• docs(dagowner): describe dag owner more carefully (#17699)
• docs(impersonation): update note so avoid misintrepretation (#17701)
• Docs: Make DAG.is_active read-only in API (#17667)
• Improve cross-links to operators and hooks references (#17622)
• Update documentation regarding Python 3.9 support (#17611)
• Fix MySQL database character set instruction (#17603)
• Document overriding XCom.clear for data lifecycle management (#17589)
• Path correction in docs for airflow core (#17567)
• docs(celery): reworded, add actual multiple queues example (#17541)
• Doc: Add FAQ to speed up parsing with tons of dag files (#17519)
• Improve image building documentation for new users (#17409)
• Doc: Strip unnecessary arguments from MariaDB JIRA URL (#17296)
• Update warning about MariaDB and multiple schedulers (#17287)
• Doc: Recommend using same configs on all Airflow components (#17146)
• Move docs about masking to a new page (#17007)
• Docs: Suggest use of Env vars instead of Airflow Vars in best practises doc (#16926)
• Docs: Better description for pod_template_file (#16861)
• Add Aneesh Joseph as Airflow Committer (#16835)
• Docs: Added new pipeline example for the tutorial docs (#16548)
• Remove upstart from docs (#16672)
• Add new committers: Jed and TP (#16671)
• Docs: Fix flask-ouathlib to flask-oauthlib in Upgrading docs (#16320)
• Docs: Fix creating a connection docs (#16312)
• Docs: Fix url for Elasticsearch (#16275)
• Small improvements for README.md files (#16244)
• Fix docs for dag_concurrency (#16177)
• Check syntatic correctness for code-snippets (#16005)
• Improvements for Docker Image docs (#14843)
• Add proper link for wheel packages in docs. (#15999)
• Add Docs for default_pool slots (#15997)
• Add memory usage warning in quick-start documentation (#15967)
• Update example KubernetesExecutor git-sync pod template file (#15904)
• Docs: Fix Taskflow API docs (#16574)
• Added new pipeline example for the tutorial docs (#16084)
• Updating the DAG docstring to include render_template_as_native_obj (#16534)
• Update docs on setting up SMTP (#16523)
• Docs: Fix API verb from POST to PATCH (#16511)

Misc/Internal

• Renaming variables to be consistent with code logic (#18685)
• Simplify strings previously split across lines (#18679)
• fix exception string of BranchPythonOperator (#18623)
• Add multiple roles when creating users (#18617)
• Move FABs base Security Manager into Airflow. (#16647)
• Remove unnecessary css state colors (#18461)
• Update boto3 to <1.19 (#18389)
• Improve coverage for airflow.security.kerberos module (#18258)
• Fix kinesis test (#18337)
• Fix provider test acessing importlib-resources (#18228)
• Silence warnings in tests from using SubDagOperator (#18275)
• Fix usage of range(len()) to enumerate (#18174)
• Test coverage on the autocomplete view (#15943)
• Update version added fields in airflow/config_templates/config.yml (#18128)
• Add "packaging" to core requirements (#18122)
• Adds LoggingMixins to BaseTrigger (#18106)
• Limit colorlog version (6.x is incompatible) (#18099)
• Fix building docs in main builds (#18035)
• Remove upper-limit on tenacity (#17593)
• Remove redundant numpy dependency (#17594)
• Bump mysql-connector-python to latest version (#17596)
• Make pandas an optional core dependency (#17575)
• Add more typing to airflow.utils.helpers (#15582)
• Chore: Some code cleanup in airflow/utils/db.py (#17090)
• Refactor: Remove processor_factory from DAG processing (#16659)
• Remove AbstractDagFileProcessorProcess from dag processing (#16816)
• Update TaskGroup typing (#16811)
• Update click to 8.x (#16779)
• Remove remaining Pylint disables (#16760)
• Remove duplicated try, there is already a try in create_session (#16701)
• Removes pylint from our toolchain (#16682)
• Refactor usage of unneeded function call (#16653)
• Add type annotations to setup.py (#16658)
• Remove SQLAlchemy <1.4 constraint (#16630) (Note: our dependencies still have a requirement on <1.4)
• Refactor dag.clear method (#16086)
• Use DAG_ACTIONS constant (#16232)
• Use updated _get_all_non_dag_permissions method (#16317)
• Add updated-name wrappers for built-in FAB methods (#16077)
• Pins docutils to <0.17 until breaking behaviour is fixed (#16133)
• Remove TaskInstance.log_filepath attribute (#15217)
• Removes unnecessary function call in airflow/www/app.py (#15956)
• Move plyvel to google provider extra (#15812)
• Update permission migrations to use new naming scheme (#16400)
• Use resource and action names for FAB (#16380)
• Swap out calls to find_permission_view_menu for get_permission wrapper (#16377)
• Fix deprecated default for fab_logging_level to WARNING (#18783)
• Allow running tasks from UI when using CeleryKubernetesExecutor (#18441)