Kotlin/dataframe v1.0.0-Beta5 on GitHub

Kotlin DataFrame 1.0 is closer than ever! But in the meantime, we’re excited to present Beta 5, which includes a lot of fixes and improvements.

What's new?

Gradle and KSP plugins were removed (do not confuse them with the Compiler Plugin). See Migration from Gradle/KSP Plugin.
Significant improvements in Java and Kotlin datetime parsing. See Parsing Date-time Strings.
Improvements in working with DuckDB and Custom SQL Databases Support.
Example projects are now separated and can be downloaded as archives.
Improved Apache Arrow and Parquet IO.

And many other improvements and fixes! See below for a complete list of changes in this release grouped by category.

📚 See the Kotlin DataFrame documentation.

We invite you to try this release:

Use the 1.0.0-Beta5 version in your Gradle or Maven projects.
- See setup instructions for Gradle, Maven and Android projects!
- We also recommend trying the compiler plugin! Now also works in Maven projects. See Maven project example.
Use the special 1.0.0-Beta5n version in Kotlin Notebook.
- The regular version still does not work with statistical functions. The n version includes a patch that resolves this issue.
- You can specify the version in the descriptor or use %useLatestDescriptors.
  If you want to include kandy, list it after dataframe:
```
%useLatestDescriptors
%use dataframe, kandy
// or
%use dataframe(1.0.0-Beta5n), kandy(0.8.4, 0.8.4)
```
- Use kernel version 0.16.0-736 or newer.

Features

Added support auto-detection of .shp file inside the provided directory by @koperagen in #1641
Added fast sorting of columns/table in Kotlin Notebook by @koperagen in #1639
Added custom CSV charsets support by @Jolanrensen in #1665
Added inplace rename for col(index) by @koperagen in #1666
Added lazy statistics by @CarloMariaProietti in #1656
Enabled kotlin.time.Instant by default by @Jolanrensen in #1657
Added support Java Record in toDataFrame() conversion by @koperagen in #1682
Refactored JDBC API and improved DuckDB support by @Jolanrensen in #1632
Added names deriving of generated nested markers from column names by @koperagen in #1702
Added custom SQLite types support by @AndreiKingsley in #1741
Added reading ListVector of primitives and StructVector values by @koperagen in #1807
Improved Java and Kotlin datetime parsing by @Jolanrensen in #1777
Added Apache Arrow/Parquet: ColumnGroup <-> StructVector reading and writing by @koperagen in #1621

Removals and Deprecations

Deprecated insert.under(path) and fixed insert.under { path } by @Jolanrensen in #1739
Added missed OverloadAccessApi deprecations in all.kt, cols.kt by @koperagen in #1697
Removed redundant move overloads where insideGroup argument doesn't affect the result by @koperagen in #1793

Fixes

Added @OptIn experimental UUID in notebooks by @Jolanrensen in #1602
Optimized type inference in joinWith by @koperagen in #1633
Optimized Iterable<Map>.toDataFrame() conversion by @koperagen in #1635
Removed sending unused HTML tables in internal Kotlin Notebook <-> Kernel table widget JSON data exchange by @koperagen in #1638
Added lexicographically ordered IDs for writeGeojson features by @koperagen in #1640
Fixed filename too long error on windows by @Jolanrensen in #1676
Fixed accidental and too strict order check in cast by @koperagen in #1716
Fixed ColumnPath nested columns resolving by @AndreiKingsley in #1730
Fixed importDataSchema() in notebooks by @Jolanrensen in #1714
Prevented escaping of SoftKeywords in code generator by @Allex-Nik in #1731
Fixed concatKeepingSchema turning ColumnGroups into DataColumns by @koperagen in #1764
Fixed NoClassDefFoundException in JDBC when PostgreSQL not present by @Jolanrensen in #1774
Substituted type parameters with type arguments in toDataFrame conversion by @koperagen in #1772
Added support for BIGINT UNSIGNED type in MySQL and MariaDB by @AndreiKingsley in #1735
Fixed null-filled columns creation for remaining names in split.into by @Nidhi-M21 in #1776
Fixed nested columns Formatted HTML rendering by @AndreiKingsley in #1796
Fixed shapefile reading from remote url by @AndreiKingsley in #1820

Documentation and Examples

Updated README.md by @AndreiKingsley in #1606
Documented image rendering and open URI action of the table component in Kotlin Notebooks by @koperagen in #1609
Expanded HTML rendering documentation by @koperagen in #1605
Fixed links to Compiler-Plugin.md by @koperagen in #1618
Updated column operations documentation by @koperagen in #1617
Added "mutually comparable" mention by @Jolanrensen in #1637
Added migration guide by @AndreiKingsley in #1631
Updated SQL docs by @zaleslaw in #1645
Added Maven project example and Maven setup instructions by @AndreiKingsley in #1643
Migrated most of the examples in 40 puzzles from delegate style dataframeOf(Columns) to dataFrameOf(Pair<String, Column>) by @koperagen in #1648
Fixed KDocs for distinct and distinctBy by @Allex-Nik in #1628
Described issue with using compiler plugin in lambdas with DslMarker receivers by @koperagen in #1669
Removed deprecated and unused access api examples and migrate multi-column add/map by @koperagen in #1649
Fixed docs related to sql and dataframeVersion by @Jolanrensen in #1705
Added String API page and updated Access APIs overview by @AndreiKingsley in #1664
Fixed nested column notation and added links by @AndreiKingsley in #1709
Added examples as separate projects by @Jolanrensen in #1677
Migrated example projects with compiler plugin by @Jolanrensen in #1734
Removed usage of deprecated column accessor overload from Youtube.ipynb by @koperagen in #1767
Updated usages of dataFrameOf(vararg columns: AnyBaseCol) in samples by @koperagen in #1770
Updated Migration Guide and added deprecation instructions by @AndreiKingsley in #1780
Added GeoDataFrame IO KDocs (GeoJSON & Shapefile) by @AndreiKingsley in #1766
Added aggregate KDocs by @AndreiKingsley in #1792
Added documentation and tests for data column arithmetics by @Allex-Nik in #1769
Bumped Exposed framework version to 1.0.0 by @zaleslaw in #1679

Compiler Plugin

Development of the compiler plugin happens in Kotlin repository. Change log only includes relevant changes in the library.

Added plugin annotations to some ColumnPath related CS DSL operations by @koperagen in #1659
Added plugin annotations to some string- and index-related CS DSL "col" operations by @koperagen in #1667
Add plugin annotations to AnyCol.cast, SingleColumn.cast by @koperagen in #1724
Added parse compiler plugin annotations by @Jolanrensen in #1728
Annotated DataRow.explode for compiler plugin support by @koperagen in #1794
Added more targets for DisableInterpretation to help combine use of plugin together with library code by @koperagen in #1790

Thank you to all our contributors❤️!

New Contributors

@Nidhi-M21 made their first contribution in #1776

Full Changelog: v1.0.0-Beta4...v1.0.0-Beta5