Kotlin/dataframe v1.0.0-Beta3

on GitHub

We're one step closer to DataFrame 1.0! We've been working hard to iron out bugs and improve documentation and examples throughout.

Try out this release by updating to 1.0.0-Beta3 in your project or use %use dataframe in your notebook!

📚 Learn more: https://kotlin.github.io/dataframe/

🌟 Examples: https://github.com/Kotlin/dataframe/tree/master/examples

See below for a complete list of changes in this release grouped by category.

1.0.0-Beta3 Highlights

• NEW: Reading data from Parquet files
• NEW: Reading data from DuckDB databases
• Our KDocs and website have received many, many additions, as you can see below, but some that stand out are:
  • Docs about the compiler plugin: https://kotlin.github.io/dataframe/compiler-plugin.html
  • A setup guide with instructions for each supported platform: https://kotlin.github.io/dataframe/setup.html
  • (Un)supported Data sources overview: https://kotlin.github.io/dataframe/data-sources.html
  • DataFrame Concepts and Principles: https://kotlin.github.io/dataframe/concepts.html
  • Frequently Asked Questions: https://kotlin.github.io/dataframe/faq.html
  • Migration away from Gradle/KSP Plugin: https://kotlin.github.io/dataframe/migration-from-plugins.html
• New examples:
  • Using the compiler plugin in a project: https://github.com/Kotlin/dataframe/tree/master/examples/kotlin-dataframe-plugin-example
  • DataFrame on Android: https://github.com/Kotlin/dataframe/tree/master/examples/android-example
  • "Unsupported data sources" examples, including Exposed, Spark, Hibernate, and Multik: https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/unsupported-data-sources/src/main/kotlin/org/jetbrains/kotlinx/dataframe/examples
  • DataFrame with Spark and Parquet files: https://github.com/Kotlin/dataframe/tree/master/examples/idea-examples/spark-parquet-dataframe
• format to HTML has had several improvements and bug fixes; though, the API might have breaking changes.
• We continue to improve the compiler plugin to better reflect the schemas of dataframes in compile time. You can see the compile time schema by hovering over a function call in the IDE — the schema will be displayed in the popup. If there are any unexpected results, feel free to open an issue or contact us!

Known Issues

• IN NOTEBOOKS use version 1.0.0-Beta3n instead of 1.0.0-Beta3. This uses the patch of #1435 for issue #1116 avoiding DefinitelyNotNullable errors. This is the version that will be provided when writing %use dataframe. Check dataFrameConfig.version in your notebook to check which version you are using.
• median and percentile require explicit type arguments for non-numeric columns
  • #1189
• FormattedFrame does not render formatted in notebooks
  • This has since been fixed but might require a yet to be released version of IntelliJ.
  • See #1405

Features

• DataFrameSchema.generateCode() by @Jolanrensen in #1230
• add nulls parse option in readExcel by @AndreiKingsley in #1247
• Kotlin 2.2 by @Jolanrensen in #1286
• Parse String to UUID #1006 by @EmmanuelBerkowicz in #1287
• Makes prev()?.newValue() work in Add DSL by @Jolanrensen in #1303
• Added readOnly mode by @zaleslaw in #1325
• Add support for validating SQL queries with the WITH clause by @zaleslaw in #1327
• Add deprecated filterNotNull for discoverability of dropNulls by @koperagen in #1334
• parseExperimentalUuid in ParserOptions by @Jolanrensen in #1306
• Add support for reading parquet file thanks to arrow-dataset #576 by @fb64 in #577
• duckdb support by @Jolanrensen in #1366
• Format colgroups by @Jolanrensen in #1374
• Add support for read-only mode in DuckDB and SQLite by @zaleslaw in #1383
• migration from kotlinx.datetime.Instant -> kotlin.time.Instant by @Jolanrensen in #1368
• Make AnyFrame.renderToString public by @bwjohnson92 in #1395

Docs, Examples, and KDocs

• Website docs landing by @AndrewKis in #1170
• Website docs examples dev by @AndreiKingsley in #1179
• fix snippets iframes by @AndreiKingsley in #1190
• Website docs kotlin notebook by @AndreiKingsley in #1180
• Generate docs by @AndreiKingsley in #1194
• Rename WS docs by @AndreiKingsley in #1191
• Website docs dev by @AndreiKingsley in #1196
• fix kodex iframes by @AndreiKingsley in #1201
• Number unification docs by @Jolanrensen in #1200
• Add initial compiler plugin documentation by @koperagen in #1202
• Add few more pages about compiler plugin by @koperagen in #1204
• Remove some of the outdated info from README.md by @koperagen in #1205
• Minor fixes for the docs by @koperagen in #1208
• Update README.md by @koperagen in #1209
• Update Gradle plugin docs and fix hardcoded versions by @koperagen in #1211
• Update documentation for "create dataframe" by @koperagen in #1212
• Remove examples for deprecated overloads from tests by @koperagen in #1213
• Notebook updates by @Jolanrensen in #1224
• Adding links to column selectors in docs by @Jolanrensen in #1226
• fix: fixed German companies notebook sample by @kikoso in #1234
• fix kodex docs iframes by @AndreiKingsley in #1233
• fix escaping in iframes by @AndreiKingsley in #1237
• add pictures for guides by @AndreiKingsley in #1239
• various docs fixes to lower the number of errors by @Jolanrensen in #1238
• Plugin example by @AndreiKingsley in #1241
• KTNB-1041: Fix resource loading by @ileasile in #1250
• Extension properties docs by @AndreiKingsley in #1246
• Readme update by @AndreiKingsley in #1251
• IDE sample of "unsupported sources"->DataFrame by @Jolanrensen in #1231
• add kdocs by @AndreiKingsley in #1261
• Gather kdocs by @AndreiKingsley in #1272
• Corr kdocs by @AndreiKingsley in #1275
• filter kdocs by @AndreiKingsley in #1288
• columns selector type by @AndreiKingsley in #1274
• explode docs and tests by @AndreiKingsley in #1291
• improve concept docs structure by @AndreiKingsley in #1305
• Simplify compiler plugin setup in documentation by @koperagen in #1317
• modules topic by @AndreiKingsley in #1314
• Flatten navigation of Modify operations by @koperagen in #1312
• faq topic by @AndreiKingsley in #1316
• Docs small fixes by @AndreiKingsley in #1323
• generateCode.kt refactor, KDocs, documentation fixes, and tests by @Jolanrensen in #1311
• Data schemas docs by @AndreiKingsley in #1339
• Data sources docs by @AndreiKingsley in #1352
• Setup docs by @AndreiKingsley in #1357
• format kdocs++ by @Jolanrensen in #1346
• count and countDistinct kdocs by @AndreiKingsley in #1301
• clarify how to create BaseColumn by @koperagen in #1371
• add spelling conventions and name fixes in a whole project by @AndreiKingsley in #1364
• android example by @AndreiKingsley in #1349
• Android example: removing arrow from the examples by @Jolanrensen in #1426
• Add Hibernate example with H2 in-memory database to idea-examples by @zaleslaw in #1367
• Extension property name generation fix & docs & tests by @AndreiKingsley in #1380
• Add SQL-to-Kotlin DataFrame transition guide for backend developers by @zaleslaw in #1377
• [Junie]: docs: update ColumnGroup description with relevant links by @jetbrains-junie[bot] in #1404
• Added Spark-Parquet-KDF example by @zaleslaw in #1391
• Rename kdocs by @AndreiKingsley in #1402
• Insert kdocs by @AndreiKingsley in #1406
• Util functions docs by @AndreiKingsley in #1414

Fixes

• Bump apache poi to 5.4.1 by @Badya in #1198
• fixed compilation on android by @Jolanrensen in #1203
• Android compilation fix by @Jolanrensen in #1218
• IDEA goodness by @Jolanrensen in #1235
• Use DayOfWeek.isoDayNumber instead of DayOfWeek.value in DateTests.kt. by @chaoren in #1232
• Run configs by @AndreiKingsley in #1236
• Reusage schemas fix by @Jolanrensen in #1252
• Remove mandatory mariadb dependency from jdbc by @koperagen in #1267
• GroupBy Jupyter codegen by @Jolanrensen in #1263
• Add more specific explodeLists overload that helps to avoid unnecessary cast by @koperagen in #1269
• Bump gradle to 8.14.2 by @Jolanrensen in #1285
• Removing recursively() remnants from public api by @Jolanrensen in #1302
• Replace static initialization of defaultTimeZone with getter by @koperagen in #1315
• Improve SQL validation logic and add related test by @zaleslaw in #1324
• Migrate publishing by @koperagen in #1331
• Hide mapToColumn in CS DSL in favor of expr by @koperagen in #1360
• Adds tableTypes to DbType by @Jolanrensen in #1283
• format kdocs++ by @Jolanrensen in #1346
• skipKodex gradle parameter in dataframe-csv by @koperagen in #1370
• Fix jupyter logger by @AndreiKingsley in #1361
• Initialize JetBrains Junie 🚀 by @jetbrains-junie[bot] in #1379
• Fixed reading Parquet file on Windows by @zaleslaw in #1381
• Extension property name generation fix & docs & tests by @AndreiKingsley in #1380
• Migrate usages of deprecated API in implementation & warning fixes by @koperagen in #1382
• [Junie]: fix: update deprecation message for DataRow.get method by @jetbrains-junie[bot] in #1399
• Removed deprecatedSchemaGeneratorPlugin Gradle plugin publication by @Jolanrensen in #1400
• migration from kotlinx.datetime.Instant -> kotlin.time.Instant by @Jolanrensen in #1368
• Columngroup map fix by @AndreiKingsley in #1396
• Adds is_formatted to json encoding for DF by @Jolanrensen in #1405
• Util functions fixes by @AndreiKingsley in #1412
• Add API that debugger renderer can rely on to avoid breaking changes by @koperagen in #1421
• Update Gradle to 9.0 by @koperagen in #1428
• What's the version 1727 and 1548? Fixed by @Jolanrensen in #436

Compiler Plugin

Note that many changes to the compiler plugin are made in Kotlin, as this is where the compiler plugin now lives.

• Prepare Split for compiler plugin support by @koperagen in #1197
• Prepare update and implode operations for compiler plugin support by @koperagen in #1207
• Deprecate predicate argument in single, colsInGroups, colsAtAnyDepth by @koperagen in #1176
• Add missed compiler plugin annotations for Split by @koperagen in #1262
• add compiler annotations for dropNA by @koperagen in #1330
• add missed compiler annotations for update by @koperagen in #1333
• Add plugin annotations for a new batch of functions by @koperagen in #1362
• Rollback parameter name to keep compatibility with the compiler plugin by @koperagen in #1358
• cols with default argument deprecation by @AndreiKingsley in #1369
• Add basic tests that API exposed for compiler plugin works without excluded dependencies by @koperagen in #1397
• Remove annotations dependency from compiler-plugin-core by @koperagen in #1409
• Add annotations to support types extending DataRow in the compiler plugin by @koperagen in #1410
• [Compiler plugin] Support more functions in toDataFrame DSL and AddDsl by @koperagen in #1425
• Prepare utility to assert compile time column order by @koperagen in #1427

New Contributors

• @Badya made their first contribution in #1198
• @kikoso made their first contribution in #1234
• @chaoren made their first contribution in #1232
• @EmmanuelBerkowicz made their first contribution in #1287
• @jetbrains-junie[bot] made their first contribution in #1379
• @bwjohnson92 made their first contribution in #1395

Full Changelog: v1.0.0-Beta2...v1.0.0-Beta3