apache-airflow 1.10.7

Airflow 1.10.7, 2019-12-24

on Python PyPI

New Features

• [AIRFLOW-5088] [AIP-24] Persisting serialized DAG in DB for webserver scalability (#5992)
• [AIRFLOW-6083] Adding ability to pass custom configuration to AWS Lambda client. (#6678)
• [AIRFLOW-5117] Automatically refresh EKS API tokens when needed (#5731)
• [AIRFLOW-5118] Add ability to specify optional components in DataprocClusterCreateOperator (#5821)
• [AIRFLOW-5681] Allow specification of a tag or hash for the git_sync init container (#6350)
• [AIRFLOW-6025] Add label to uniquely identify creator of Pod (#6621)
• [AIRFLOW-4843] Allow orchestration via Docker Swarm (SwarmOperator) (#5489)
• [AIRFLOW-5751] add get_uri method to Connection (#6426)
• [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id (#6655)
• [AIRFLOW-2694] Declare permissions in DAG definition (#4642)
• [AIRFLOW-4940] Add DynamoDB to S3 operator (#5663)
• [AIRFLOW-4161] BigQuery to Mysql Operator (#5711)
• [AIRFLOW-6041] Add user agent to the Discovery API client (#6636)
• [AIRFLOW-6089] Reorder setup.py dependencies and add ci (#6681)
• [AIRFLOW-5921] Add bulk_load_custom to MySqlHook (#6575)
• [AIRFLOW-5854] Add support for tty parameter in Docker related operators (#6542)
• [AIRFLOW-4758] Add GcsToGDriveOperator operator (#5822)

Improvements

• [AIRFLOW-3656] Show doc link for the current installed version (#6690)
• [AIRFLOW-5665] Add path_exists method to SFTPHook (#6344)
• [AIRFLOW-5729] Make InputDataConfig optional in Sagemaker's training config (#6398)
• [AIRFLOW-5045] Add ability to create Google Dataproc cluster with custom image from a different project (#5752)
• [AIRFLOW-6132] Allow to pass in tags for the AzureContainerInstancesOperator (#6694)
• [AIRFLOW-5945] Make inbuilt OperatorLinks work when using Serialization (#6715)
• [AIRFLOW-5947] Make the json backend pluggable for DAG Serialization (#6630)
• [AIRFLOW-6239] Filter dags return by last_dagruns (to only select visible dags, not all dags) (#6804)
• [AIRFLOW-6095] Filter dags returned by task_stats (to only select visible dags, not all dags) (#6684)
• [AIRFLOW-4482] Add execution_date to "trigger DagRun" API response (#5260)
• [AIRFLOW-1076] Add get method for template variable accessor (#6793)
• [AIRFLOW-5194] Add error handler to action log (#5883)
• [AIRFLOW-5936] Allow explicit get_pty in SSHOperator (#6586)
• [AIRFLOW-5474] Add Basic auth to Druid hook (#6095)
• [AIRFLOW-5726] Allow custom filename in RedshiftToS3Transfer (#6396)
• [AIRFLOW-5834] Option to skip serve_logs process with airflow worker (#6709)
• [AIRFLOW-5583] Extend the 'DAG Details' page to display the start_date / end_date (#6235)
• [AIRFLOW-6250] Ensure on_failure_callback always has a populated context (#6812)
• [AIRFLOW-6222] http hook logs response body for any failure (#6779)
• [AIRFLOW-6260] Drive _cmd config option by env var (AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD for example) (#6801)
• [AIRFLOW-6168] Allow proxy_fix middleware of webserver to be configurable (#6723)
• [AIRFLOW-5931] Use os.fork when appropriate to speed up task execution. (#6627)
• [AIRFLOW-4145] Allow RBAC roles permissions, ViewMenu to be over-ridable (#4960)
• [AIRFLOW-5928] Hive hooks load_file short circuit (#6582)
• [AIRFLOW-5313] Add params support for awsbatch_operator (#5900)
• [AIRFLOW-2227] Add delete method to Variable class (#4963)
• [AIRFLOW-5082] Add subject in AwsSnsHook (#5694)
• [AIRFLOW-5715] Make email, owner context available (#6385)
• [AIRFLOW-5345] Allow SqlSensor's hook to be customized by subclasses (#5946)
• [AIRFLOW-5417] Fix DB disconnects during webserver startup (#6023)
• [AIRFLOW-5730] Enable get_pandas_df on PinotDbApiHook (#6399)
• [AIRFLOW-3235] Add list function in AzureDataLakeHook (#4070)
• [AIRFLOW-5442] implementing get_pandas_df method for druid broker hook (#6057)
• [AIRFLOW-5883] Improve count() queries in a few places (#6532)
• [AIRFLOW-5811] Add metric for externally killed task count (#6466)
• [AIRFLOW-5758] Support the custom cursor classes for the PostgreSQL hook (#6432)
• [AIRFLOW-5766] Use httpbin.org in http_default (#6438)
• [AIRFLOW-5798] Set default ExternalTaskSensor.external_task_id (#6431)
• [AIRFLOW-5643] Reduce duplicated logic in S3Hook (#6313)
• [AIRFLOW-5562] Skip grant single DAG permissions for Admin role. (#6199)
• [AIRFLOW-6192] Stop creating Hook from SFTPSensor.init (#6748)
• [AIRFLOW-5749][AIRFLOW-4162] Support the "blocks" component for the Slack operators (#6418)
• [AIRFLOW-5693] Support the "blocks" component for the Slack messages (#6364)
• [AIRFLOW-5714] Collect SLA miss emails only from tasks missed SLA (#6384)
• [AIRFLOW-5049] Add validation for src_fmt_configs in bigquery hook (#5671)
• [AIRFLOW-6177] Log DAG processors timeout event at error level, not info (#6731)
• [AIRFLOW-6180] Improve kerberos init in pytest conftest (#6735)
• [AIRFLOW-6159] Change logging level of the heartbeat message to DEBUG (#6716)
• [AIRFLOW-6144] Improve the log message of airflow scheduler (#6710)
• [AIRFLOW-6099] Add host name to task runner log (#6688)
• [AIRFLOW-6045] Error on failed execution of compile_assets (#6640)
• [AIRFLOW-5144] Add confirmation on delete button click (#6745)
• [AIRFLOW-6099] Add host name to task runner log (#6688)
• [AIRFLOW-5915] Add support for the new documentation theme (#6563)
• [AIRFLOW-5897] Allow setting -1 as pool slots value in webserver (#6550)
• [AIRFLOW-5888] Use psycopg2-binary for postgres operations (#6533)
• [AIRFLOW-5870] Allow -1 for pool size and optimise pool query (#6520)

Bug Fixes

• [AIRFLOW-XXX] Bump Jira version to fix issue with async
• [AIRFLOW-XXX] Add encoding to fix Cyrillic output when reading back task logs (#6631)
• [AIRFLOW-5304] Fix extra links in BigQueryOperator with multiple queries (#5906)
• [AIRFLOW-6268] Prevent (expensive) ajax calls on home page when no dags visible (#6839)
• [AIRFLOW-6259] Reset page to 1 with each new search for dags (#6828)
• [AIRFLOW-6185] SQLAlchemy Connection model schema not aligned with Alembic schema (#6754)
• [AIRFLOW-3632] Only replace microseconds if execution_date is None in trigger_dag REST API (#6380)
• [AIRFLOW-5458] Bump Flask-AppBuilder to 2.2.0 (for Python >= 3.6) (#6607)
• [AIRFLOW-5072] gcs_hook should download files once (#5685)
• [AIRFLOW-5744] Environment variables not correctly set in Spark submit operator (#6796)
• [AIRFLOW-3189] Remove schema from DbHook.get_uri response if None (#6833)
• [AIRFLOW-6195] Fixed TaskInstance attrs not correct on UI (#6758)
• [AIRFLOW-5889] Make polling for AWS Batch job status more resillient (#6765)
• [AIRFLOW-6043] Fix bug in UI when "filtering by root" to display section of dag (#6638)
• [AIRFLOW-6033] Fix UI Crash at "Landing Times" when task_id is changed (#6635)
• [AIRFLOW-3745] Fix viewer not able to view dag details (#4569)
• [AIRFLOW-6175] Fixes bug when tasks get stuck in "scheduled" state (#6732)
• [AIRFLOW-5463] Make Variable.set when replacing an atomic operation (#6807)
• [AIRFLOW-5582] Add get_autocommit to JdbcHook (#6232)
• [AIRFLOW-5867] Fix webserver unit_test_mode data type (#6517)
• [AIRFLOW-5819] Update AWSBatchOperator default value (#6473)
• [AIRFLOW-5709] Fix regression in setting custom operator resources. (#6331)
• [AIRFLOW-5658] Fix broken navigation links (#6374)
• [AIRFLOW-5727] SqoopHook: Build --connect parameter only if port/schema are defined (#6397)
• [AIRFLOW-5695] use RUNNING_DEPS to check run from UI (#6367)
• [AIRFLOW-6254] obscure conn extra in logs (#6817)
• [AIRFLOW-4824] Add charset handling for SqlAlchemy engine for MySqlHook (#6816)
• [AIRFLOW-6091] Add flushing in execute method for BigQueryCursor (#6683)
• [AIRFLOW-6256] Ensure Jobs table is cleared when resetting DB (#6818)
• [AIRFLOW-5224] Add encoding parameter to GoogleCloudStorageToBigQuery (#6297)
• [AIRFLOW-5179] Remove top level init.py (#5818)
• [AIRFLOW-5660] Attempt to find the task in DB from Kubernetes pod labels (#6340)
• [AIRFLOW-6241] Fix typo in airflow/gcp/operator/dataflow.py (#6806)
• [AIRFLOW-6171] Apply .airflowignore to correct subdirectories (#6784)
• [AIRFLOW-6018] Display task instance in table during backfilling (#6612)
• [AIRFLOW-6189] Reduce the maximum test duration to 8 minutes (#6744)
• [AIRFLOW-6141] Remove ReadyToRescheduleDep if sensor mode == poke (#6704)
• [AIRFLOW-6054] Add a command that starts the database consoles (#6653)
• [AIRFLOW-6047] Simplify the logging configuration template (#6644)
• [AIRFLOW-6017] Exclude PULL_REQUEST_TEMPLATE.md from RAT check (#6611)
• [AIRFLOW-4560] Fix Tez queue parameter name in mapred_queue (#5315)
• [AIRFLOW-2143] Fix TaskTries graph counts off-by-1 (#6526)
• [AIRFLOW-5873] KubernetesPodOperator fixes and test (#6523)
• [AIRFLOW-5869] BugFix: Some Deserialized tasks have no start_date (#6519)
• [AIRFLOW-4020] Remove DAG edit permissions from Viewer role (#4845)
• [AIRFLOW-6263] Fix broken WinRM integration (#6832)
• [AIRFLOW-5836] Pin azure-storage-blob version to <12 (#6486)
• [AIRFLOW-4488] Fix typo for non-RBAC UI in max_active_runs_per_dag (#6778)
• [AIRFLOW-5942] Pin PyMSSQL to <3.0 (#6592)
• [AIRFLOW-5451] SparkSubmitHook don't set default namespace (#6072)
• [AIRFLOW-6271] Printing log files read during load_test_config (#6842)
• [AIRFLOW-6308] Unpin Kombu for Python 3

Misc/Internal

• [AIRFLOW-6009] Switch off travis_wait for regular tests (#6600)
• [AIRFLOW-6226] Always reset warnings in tests
• [AIRFLOW-XXX] Remove cyclic imports and pylint hacks in Serialization (#6601)
• [AIRFLOW-XXX] Bump npm from 6.4.1 to 6.13.4 in /airflow/www (#6815)
• [AIRFLOW-XXX] Remove executable permission from file
• [AIRFLOW-XXX] Group AWS & Azure dependencies (old [emr] etc. extra still work)
• [AIRFLOW-5487] Fix unused warning var (#6111)
• [AIRFLOW-5925] Relax funcsigs and psutil version requirements (#6580)
• [AIRFLOW-5740] Fix Transient failure in Slack test (#6407)
• [AIRFLOW-6058] Running tests with pytest (#6472)
• [AIRFLOW-6066] Added pre-commit checks for accidental debug stmts (#6662)
• [AIRFLOW-6060] Improve conf_vars context manager (#6658)
• [AIRFLOW-6044] Standardize the Code Structure in kube_pod_operator.py (#6639)
• [AIRFLOW-4940] Simplify tests of DynamoDBToS3Operator (#6836)
• [AIRFLOW-XXX] Update airflow-jira release management script (#6772)
• [AIRFLOW-XXX] Add simple guidelines to unit test writing (#6846)
• [AIRFLOW-6309] Fix stable build on Travis

Doc only changes

• [AIRFLOW-6211] Doc how to use conda for local virtualenv (#6766)
• [AIRFLOW-5855] Fix broken reference in custom operator doc (#6508)
• [AIRFLOW-5875] Fix typo in example_qubole_operator.py (#6525)
• [AIRFLOW-5702] Fix common docstring issues (#6372)
• [AIRFLOW-5640] Document and test email parameters of BaseOperator (#6315)
• [AIRFLOW-XXX] Improve description OpenFaaS Hook (#6187)
• [AIRFLOW-XXX] GSoD: How to make DAGs production ready (#6515)
• [AIRFLOW-XXX] Use full command in examples (#5973)
• [AIRFLOW-XXX] Update docs to accurately describe the precedence of remote and local logs (#5607)
• [AIRFLOW-XXX] Fix example "extras" field in mysql connect doc (#5285)
• [AIRFLOW-XXX] Fix wrong inline code highlighting in docs (#5309)
• [AIRFLOW-XXX] Group executors in one section (#5834)
• [AIRFLOW-XXX] Add task lifecycle diagram to documentation (#6762)
• [AIRFLOW-XXX] Highlight code blocks (#6243)
• [AIRFLOW-XXX] Documents about task_concurrency and pool (#5262)
• [AIRFLOW-XXX] Fix incorrect docstring parameter (#6649)
• [AIRFLOW-XXX] Add link to XCom section in concepts.rst (#6791)
• [AIRFLOW-XXX] Update kubernetes doc with correct path (#6774)
• [AIRFLOW-XXX] Add information how to configure pytest runner (#6736)
• [AIRFLOW-XXX] more GSOD improvements (#6585)
• [AIRFLOW-XXX] Clarified a grammatically incorrect sentence (#6667)
• [AIRFLOW-XXX] Add notice for Mesos Executor deprecation in docs (#6712)
• [AIRFLOW-XXX] Update list of pre-commits (#6603)
• [AIRFLOW-XXX] Updates to Breeze documentation from GSOD (#6285)
• [AIRFLOW-XXX] Clarify daylight savings time behavior (#6324)
• [AIRFLOW-XXX] GSoD: Adding 'Create a custom operator' doc (#6348)
• [AIRFLOW-XXX] Add resources & links to CONTRIBUTING.rst (#6405)
• [AIRFLOW-XXX] Update chat channel details from gitter to slack (#4149)
• [AIRFLOW-XXX] Add logo info to readme (#6349)
• [AIRFLOW-XXX] Fixed case problem with CONTRIBUTING.rst (#6329)
• [AIRFLOW-XXX] Google Season of Docs updates to CONTRIBUTING doc (#6283)