apache-airflow 1.10.6rc1

Airflow 1.10.6rc1, 2019-10-18

on Python PyPI

New Features

• [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546)
• [AIRFLOW-4741] Optionally report task errors to Sentry (#5407)
• [AIRFLOW-4939] Add default_task_retries config (#5570)
• [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130)
• [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789)
• [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743)

Improvements

• [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738)
• [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737)
• [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736)
• [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330)
• [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034)
• [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194)
• [AIRFLOW-5657] Update the upper bound for dill dependency (#6334)
• [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891)
• [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233)
• [AIRFLOW-5126] Read aws_session_token in extra_config of the aws hook (#6303)
• [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302)
• [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304)
• [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309)
• [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153)
• [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) (#6163)
• [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012)
• [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062)
• [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059)
• [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948)
• [AIRFLOW-3888] HA for Hive metastore connection (#4708)
• [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873)
• [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768)
• [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214)
• [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696)
• [AIRFLOW-2891] allow configurable docker_operator container name (#5689)
• [AIRFLOW-4285] Update task dependency context definition and usage (#5079)
• [AIRFLOW-5142] Fixed flaky Cassandra test (#5758)
• [AIRFLOW-5218] Less polling of AWS Batch job status (#5825)
• [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589)
• [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339)
• [AIRFLOW-5560] Allow no confirmation on reset dags in airflow backfill command (#6195)
• [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879)
• [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159)
• [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154)
• [AIRFLOW-4835] Refactor operator render_template (#5461)

Bug Fixes

• [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator (#6078)
• [Airflow 4923] Fix Databricks hook leaks API secret in logs (#5635)
• [AIRFLOW-5133] Keep original env state in provide_gcp_credential_file (#5747)
• [AIRFLOW-5497] Update docstring in airflow/utils/dag_processing.py (#6314)
• Revert/and then rework "[AIRFLOW-4797] Improve performance and behaviour of zombie detection (#5511)" to improve performance (#5908)
• [AIRFLOW-5634] Don't allow editing of DagModelView (#6308)
• [AIRFLOW-4309] Remove Broken Dag error after Dag is deleted (#6102)
• [AIRFLOW-5387] Fix "show paused" pagination bug (#6100)
• [AIRFLOW-5489] Remove unneeded assignment of variable (#6106)
• [AIRFLOW-5491] mark_tasks pydoc is incorrect (#6108)
• [AIRFLOW-5492] added missing docstrings (#6107)
• [AIRFLOW-5503] Fix tree view layout on HDPI screen (#6125)
• [AIRFLOW-5481] Allow Deleting Renamed DAGs (#6101)
• [AIRFLOW-3857] spark_submit_hook cannot kill driver pod in Kubernetes (#4678)
• [AIRFLOW-4391] Fix tooltip for None-State Tasks in 'Recent Tasks' (#5909)
• [AIRFLOW-5554] Require statsd 3.3.0 minimum (#6185)
• [AIRFLOW-5306] Fix the display of links when they contain special characters (#5904)
• [AIRFLOW-3705] Fix PostgresHook get_conn to use conn_name_attr (#5841)
• [AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe Scheduler shutdown on SIGTERM (#6237)
• [AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView (#6307)
• [AIRFLOW-4833] Allow to set Jinja env options in DAG declaration (#5943)
• [AIRFLOW-5408] Fix env variable name in Kubernetes template (#6016)
• [AIRFLOW-5102] Worker jobs should terminate themselves if they can't heartbeat (#6284)
• [AIRFLOW-5572] Clear task reschedules when clearing task instances (#6217)
• [AIRFLOW-5543] Fix tooltip disappears in tree and graph view (RBAC UI) (#6174)
• [AIRFLOW-5444] Fix action_logging so that request.form for POST is logged (#6064)
• [AIRFLOW-5484] fix PigCliHook has incorrect named parameter (#6112)
• [AIRFLOW-5342] Fix MSSQL breaking task_instance db migration (#6014)
• [AIRFLOW-5556] Add separate config for timeout from scheduler dag processing (#6186)
• [AIRFLOW-4858] Deprecate "Historical convenience functions" in airflow.configuration (#5495) (#6144)
• [AIRFLOW-774] Fix long-broken DAG parsing Statsd metrics (#6157)
• [AIRFLOW-5419] Use sudo to kill cleared tasks when running with impersonation (#6026) (#6176)
• [AIRFLOW-5537] Yamllint is not needed as dependency on host
• [AIRFLOW-5536] Better handling of temporary output files
• [AIRFLOW-5535] Fix name of VERBOSE parameter
• [AIRFLOW-5519] Fix sql_to_gcs operator missing multi-level default args by adding apply_defaults decorator (#6146)
• [AIRFLOW-5210] Make finding template files more efficient (#5815)
• [AIRFLOW-5447] Scheduler stalls because second watcher thread in default args (#6129)

Doc-only changes

• [AIRFLOW-5574] Fix Google Analytics script loading (#6218)
• [AIRFLOW-5588] Add Celery's architecture diagram (#6247)
• [AIRFLOW-5521] Fix link to GCP documentation (#6150)
• [AIRFLOW-5398] Update contrib example DAGs to context manager (#5998)
• [AIRFLOW-5268] Apply same DAG naming conventions as in literature (#5874)
• [AIRFLOW-5101] Fix inconsistent owner value in examples (#5712)
• [AIRFLOW-XXX] Fix typo - AWS DynamoDB Hook (#6319)
• [AIRFLOW-XXX] Fix Documentation for adding extra Operator Links (#6301)
• [AIRFLOW-XXX] Add section on task lifecycle & correct casing in docs (#4681)
• [AIRFLOW-XXX] Make it clear that 1.10.5 wasn't accidentally omitted from UPDATING.md (#6240)
• [AIRFLOW-XXX] Improve format in code-block directives (#6242)
• [AIRFLOW-XXX] Format Sendgrid docs (#6245)
• [AIRFLOW-XXX] Update to new logo (#6066)
• [AIRFLOW-XXX] Typo in FAQ - schedule_interval (#6291)
• [AIRFLOW-XXX] Add message about breaking change in DAG#get_task_instances in 1.10.4 (#6226)
• [AIRFLOW-XXX] Fix incorrect units in docs for metrics using Timers (#6152)
• [AIRFLOW-XXX] Fix backtick issues in .rst files & Add Precommit hook (#6162)
• [AIRFLOW-XXX] Update documentation about variables forcing answer (#6158)
• [AIRFLOW-XXX] Add a third way to configure authorization (#6134)
• [AIRFLOW-XXX] Add example of running pre-commit hooks on single file (#6143)
• [AIRFLOW-XXX] Add information about default pool to docs (#6019)
• [AIRFLOW-XXX] Make Breeze The default integration test environment (#6001)

Misc/Internal

• [AIRFLOW-5687] Upgrade pip to 19.0.2 in CI build pipeline (#6358) (#6361)
• [AIRFLOW-5533] Fixed failing CRON build (#6167)
• [AIRFLOW-5130] Use GOOGLE_APPLICATION_CREDENTIALS constant from library (#5744)
• [AIRFLOW-5369] Adds interactivity to pre-commits (#5976)
• [AIRFLOW-5531] Replace deprecated log.warn() with log.warning() (#6165)
• [AIRFLOW-4686] Make dags Pylint compatible (#5753)
• [AIRFLOW-4864] Remove calls to load_test_config (#5502)
• [AIRFLOW-XXX] Pin version of mypy so we are stable over time (#6198)
• [AIRFLOW-XXX] Add tests that got missed from #5127
• [AIRFLOW-4928] Move config parses to class properties inside DagBag (#5557)
• [AIRFLOW-5003] Making AWS Hooks pylint compatible (#5627)
• [AIRFLOW-5580] Add base class for system test (#6229)