luigi 2.1.0

Version 2.1.0

on Python PyPI

Finally, a new PyPI release in a rather long while. Thanks @Tarrasch and @erikbern, and all the contributors!

Added

luigi:

• Notifications: more emails and proper coloring (#1471), improved SMTP handling
• EnumParameter (#1479), DictParameter (#1574)
• Support for Python 3.5 (#1494)
• Process locking on Alpine Linux (#1530) and Windows (#1557)
• Visualizer: resources tab (#1566), GUI functionality to disable a worker (#1564)

luigi.contrib:

• ExternalBigqueryTask (#1434), BigqueryCreateViewTask (#1465)
• Luigi tasks for Dataproc, Google's managed Hadoop MapReduce, Spark, Pig, and Hive service (#1601)
• ExternalProgramTask, ExternalPythonProgramTask - commonalities for running any external application or script (#1520)
• Support for SFTP (#1585)
• Sped-up Hive client using Metastore (#1533)
• OpenerTarget, a single Luigi target to open multiple file system types (#1555)
• Query base task (giving rise also to luigi.contrib.redshift.RedshiftQuery and luigi.postgres.PostgresQuery) (#1493)
• RedshiftUnloadTask (#1527)
• UploadToSalesforceTask (#1404)
• Support for S3 assumed role (#1596)

Changed

luigi:

• Semi-opaque, hashed task_id (as opposed to TaskName(param1=value1, param2=foo bar)) (#1444)
• More explicit way to handle timelike parameters (date vs datetime) (#1473)
• Optimizations in scheduler algorithm

Removed

luigi:

• Old deprecated (2014) stuff around scheduler and its state (#1592)

luigi.contrib:

• Deprecated classes SparkJob, Spark1xBackwardCompat, Spark1xJob, PySpark1xJob (#1442)

Fixed

luigi.contrib:

• Ensure that FTP RemoteTarget successfully creates temporary files (meaning, in a directory relative to output) (#1515)
• Remove superfluous init_mapper()/init_reducer() calls in LocalJobRunner (#1475)
• Humanly format HadoopJobError (#1528)
• Broken Redshift table creation (#1453)
• Improved Salesforce reliability (#1597, #1600)
• Missing call to post_copy() (#1502)

...and a slew of other additions, fixes, improvements and documentation.