delta-io/delta v4.0.0-final-rc1

Delta Lake 4.0.0 RC1

on GitHub

We are excited to share the first release candidate (RC1) of the Delta Lake 4.0.0 release! Release notes are coming soon.

How to use the release candidate?

Download Spark 4.0 from https://spark.apache.org/downloads.html.

For this release candidate, we have published the artifacts to a staging repository https://oss.sonatype.org/content/repositories/iodelta-1227. Here's how you can use them:

Spark submit

• Add --repositories https://oss.sonatype.org/content/repositories/iodelta-1227 to the command line arguments.
• Example:

spark-submit --packages io.delta:delta-spark_2.13:4.0.0 --repositories https://oss.sonatype.org/content/repositories/iodelta-1227 examples/examples.py

Currently Spark shells (PySpark and Scala) do not accept the external repositories option. However, once the artifacts have been downloaded to the local cache, the shells can be run with Delta 4.0.0 by just providing the --packages io.delta:delta-spark_2.13:4.0.0 argument.

Spark Shell

bin/spark-shell --packages io.delta:delta-spark_2.13:4.0.0 \
--repositories https://oss.sonatype.org/content/repositories/iodelta-1227 \
--conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
--conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog

Spark SQL

bin/spark-sql --packages io.delta:delta-spark_2.13:4.0.0 \
--repositories https://oss.sonatype.org/content/repositories/iodelta-1227 \
--conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
--conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog

Maven

<repositories>
  <repository>
    <id>staging-repo</id>
    <url>https://oss.sonatype.org/content/repositories/iodelta-1227</url>
  </repository>
</repositories>
<dependency>
  <groupId>io.delta</groupId>
  <artifactId>delta-spark_2.13</artifactId>
  <version>4.0.0</version>
</dependency>

SBT project

libraryDependencies += "io.delta" %% "delta-spark" % "4.0.0"
resolvers += "Delta" at https://oss.sonatype.org/content/repositories/iodelta-1227

(PySpark) Delta-Spark

• Download the two artifacts attached to this release

  • delta-spark-4.0.0.tar.gz
  • delta_spark-4.0.0-py3-none-any.whl

• Keep them in one directory. Lets call that ~/Downloads

• pip install ~/Downloads/delta_spark-4.0.0-py3-none-any.whl