apache/doris 0.9.0-rc02

Apache Doris 0.9.0 (incubating) Release

on GitHub

作为 Doris 进入 Apache 孵化器以来的第一个Release版本，该版本在功能、性能、易用性、稳定性方面做了诸多改进。下面介绍本次发布的一些主要功能

1. 流式导入

之前版本中，Doris只支持批量的异步导入方式。这样的导入方式对Kafka、Spark Streaming等流式系统来说并不友好。另外，批量导入执行过程中有多次落盘操作，导致导入整体效率不高，单机导入速度大概在10-20MB/s。
新的导入方式，采用micro-batch的思想，Doris一方面支持了同步导入数据，方便对接流式数据源；另一方面也极大的提高了导入效率，单节点导入吞吐能够达到100MB/s；小数据量导入延时从分钟级降低到毫秒级。

2. Co-Location Join

通过对表数据分布的管理，支持多表数据的原地Join操作。减少Join操作的网络传输，极大降低Join类查询的延迟。（感谢美团同学的贡献）

3. 高效的数据分片管理

通过引入RocksDB来存储数据分片的元信息，极大地减少随机IO 操作。单机50万分片启动时间从10分钟降到了20秒。

4. 更丰富的集群状态查询和管理

提供多达100多项监控指标，以及丰富的监控Dashboard；提供当前查询和历史查询的资源消耗展示和统计；提供更完善的副本管理和状态查询工具。方便运维人员快速定位系统故障和找出性能瓶颈。

5. 提供Docker 开发环境镜像

开发人员无需再应对复杂的编译依赖，可以在绝大部分环境下快速搭建Doris编译调试环境。

6. Apache License 2.0

License 修改为 Apache License 2.0，任何公司或者个人都可以自由的使用和开发Doris。

7. Bug及稳定性问题修复

多达30余项BUG及稳定性问题修复。助力Doris成为更成熟稳定的数据库产品。

Change Log

Features, Changes and Enhanced:

1. Add streaming load feature. You can execute 'help stream load;' to see more information.
2. Introduce RocksDB to save the header info of tablets in Backends, to reduce the IO operations and increate speeding of restarting.
3. Change project name from Palo to Doris, include variables and namespace name (#268)
4. Change license to Apache License 2.0 (#262)
5. Improve build scripts and add docker dev environment (#301).
6. Support NULLS LAST and NULLS FIRST syntax (#252)
7. Support AnalyticExpr in View (#248)
8. Improve cardinality, avgRowSize, numNodes stat info in OlapScanNode (#256)
9. Transform row-oriented table to columnar-oriented table (#311)
10. Change ByteBuffer to StorageByteBuffer in olap/byte_buffer.h (#341)
11. Improve the Backend's disk info report performance (#349)
12. Add path info of replica in catalog (#327)
13. Change log verbose level to vlog(3) (#325)
14. Change PaloMetrics' name and Catalog's Id generator (#329)
15. Rename Rowset to SegmentGroup (#364)
16. Subsititue ColumnType to Type (#366)
17. Optimize the publish logic of streaming load (#350)
18. Add connection id to CurrentQueryStatisticsProcDir (#355)
19. Change SQL built-in function's symbol (#274)

Bugs:

1. Failed to register equal conjuncts which refer more than three tuples (#266)
2. Rewrite aes encryption (#264)
3. Fix a bug that user can not kill it own connection (#276)
4. Fix lose of be's meta data bug (#318)
5. Fix SHOW BACKENDS return ERROR (#320)
6. Fix Ubuntu llvm compile (#361)
7. Support for custom build toolchains (#330)
8. Modify partition's version name to what it means (#334)
9. Fix compile issue of thirdparty library (#338)
10. Fix a compile issue of DORIS_GCC_HOME (#339)
11. Add special add_column_statistics method for linked_schema_change (#337)
12. Fix wrong query result when column value is Null (#344)
13. Fix bug of using symbolic link dir as storage path (#340)
14. Add distributor which schedule task to be fairly, for routine load job (#333)
15. Fix UnionStmt toSql bug (#249)
16. Fix a bug that user can not kill it own connection (#276)
17. Fix failed cases in regression test (#299)
18. Fix code LICENSE for file modified from LevelDB. (#300)
19. Fix snapshot's making header bug (#362)
20. Fix stream load failure when target table contains HLL and insert failure when it contains subquery (#359)
21. Avoid 'No more data to read' error when handling stream load RPC (#354)
22. Fix cast error in StreamLoadScanNode (#356)
23. Fix insert error when it contains HLL (#358)

Compatibility

This release version DOES NOT support rolling upgrade from 0.8.1 and previous version, and CAN NOT rollback after upgrade. So it is HIGHLY RECOMMENDED to backup your data and meta data before upgrading to this release version. Or upgrade it in your test/pre-online Doris cluster to make sufficient test.

Rolling upgrade from 0.8.2

If your Doris version is less than 0.8.2, it is HIGHLY RECOMMENDED to first upgrade it to 0.8.2.

1. Check whether your Doris cluster has row-oriented table using tools in tools/row_to_column/.
2. Change the PALO_HOME in be.conf and fe.conf to DORIS_HOME.
3. Upgrade one Backend, and check if every thing is OK.
4. Upgrade all Backends and check if every thing is OK.
5. Upgrade all non-master Frontends, and check if every thing is OK.
6. Upgrade master Frontend, and check if every thing is OK.