tarantool/tarantool 3.0.0-alpha2 on GitHub

Date: 2023-08-23
Tag: 3.0.0-alpha2

Overview

3.0.0-alpha2 is the alpha version of the 3.0 release series.

This release introduces 74 new features and resolves 87 bugs since
the 2.11.0 version. There can be bugs in less common areas. If you find any,
feel free to report an issue on GitHub.

Notable changes are:

Added initial support of configuring vshard's storage and router roles in the
declarative configuration (gh-9007).
Fixes and improvements of the configuration process based on early adopters feedback.
New config sections: metrics, feedback, vinyl, flightrec (EE), security (EE) (gh-8861).

Compatibility

Tarantool 3.x is backward compatible with Tarantool 1.10.x in the binary data
layout, client-server protocol, and replication protocol.

Please upgrade using the box.schema.upgrade() procedure to unlock
all the new features of the 2.x series.

Functionality added or changed

Core

[Breaking change] Disabled DDL operations with an old system schema.
Now you have to run box.schema.upgrade() before you can execute any DDL
operations (gh-7149).

The _schema.max_id field in the box.space._schema function is now deprecated.
Now the _space:max() function is used instead of _schema.max_id (gh-5997).
Added the new function box.malloc.info() for reporting memory usage of
Tarantool internal data structures allocated with malloc(). The function
is available only on Linux (gh-7311).
Now index_object.parts contains the following methods, similar to the
key_def Lua module: extract_key(), compare(), compare_with_key(),
merge() (gh-7356).
Introduced the box.watch_once() function to get the value currently
associated with a notification key on the local instance without subscribing
to future changes (gh-6493).
Introduced the new IPROTO_WATCH_ONCE request to get the value currently
associated with a notification key on a remote instance without subscribing
to future changes (gh-6493).
Introduced the conn:watch_once() net.box connection method to get the value
currently associated with a notification key on a remote instance without
subscribing to future changes. The new method is implemented using the
IPROTO_WATCH_ONCE request type (gh-6493).
Introduced the default field values in the space format (gh-8157).
Added support for a functional variant of the default field values (gh-8609).

Memtx

Added memtx read view statistics to box.stat.memtx() (gh-8501).

Replication

Added a new bootstrap_leader configuration option to specify the node from
which a replica should bootstrap. To do this, set box.cfg.bootstrap_strategy
to 'config' and set bootstrap_leader value to either the URI or UUID of
the desired bootstrap leader. For example:
```
  box.cfg{
      bootstrap_strategy = 'config',
      bootstrap_leader = 'localhost:3301',
      replication = {
          'localhost:3301',
          'localhost:3302',
      },
      listen = '3302',
  }
```
(gh-7999).
You may now control which node new replicas choose as a bootstrap leader
without touching node config. To do so, set box.cfg.bootstrap_strategy to
'supervised', and the nodes will only bootstrap off the node on which you
called box.ctl.make_bootstrap_leader() last.
This works on an empty replica set bootstrap as well: start the admin console
before configuring the nodes. Then configure the nodes:
```
box.cfg{
    bootstrap_strategy = 'supervised',
    replication = ...,
    listen = ...,
}
```
Finally, call box.ctl.make_bootstrap_leader() through the admin console
on the node you want to promote. All the nodes will bootstrap off that node
(gh-8509).
A new option box.cfg.cluster_name allows assigning a human-readable name to
the entire cluster. It has to match in all instances and is displayed in
box.info.cluster.name (gh-5029).
A new option box.cfg.replicaset_name allows assigning a human-readable name
to the replicaset. It works the same as box.cfg.replicaset_uuid. Its value
must be the same across all instances of one replicaset. The replicaset name
is displayed in box.info.replicaset.name (gh-5029).
A new option box.cfg.instance_name allows assigning a human-readable name to
the instance. It works the same as box.cfg.instance_uuid. Its value must be
unique in the replicaset. The instance name is displayed in box.info.name.
Names of other replicas in the same replicaset are visible in
box.info.replication[id].name (gh-5029).
Instance at rebootstrap can change its UUID while keeping its numeric ID if it
has the same non-empty instance name (gh-5029).

Added the ability to set the bootstrap_leader configuration option to the
instance name of the desired bootstrap leader:

  box.cfg{
      bootstrap_strategy = 'config',
      bootstrap_leader = 'leader-name',
      replication = {
          ...
      },
      ...
  }

(gh-7999, gh-8539).

Implemented correct recovery of mixed transactions. To do this, set
box.cfg.force_recovery to true. If you need to revert to the old
behavior, don't set the force_recovery option (gh-7932).

LuaJIT

LuaJIT now can handle exceptions on traces (gh-7745).

Lua

Embedded the tarantool/metrics module for metrics aggregation and export (gh-7725).
Metrics can now be configured using the box.cfg() function (gh-7725).
[Breaking change] Added the new varbinary type to Lua. An object of
this type is similar to a plain string but encoded in MsgPack as MP_BIN so
it can be used for storing binary blobs in the database. This also works the
other way round: data fields stored as MP_BIN are now decoded in Lua as
varbinary objects, not as plain strings, as they used to be. Since the latter
may cause compatibility issues, the new compat option binary_data_decoding
was introduced to revert the built-in decoder to the old behavior (gh-1629).
Added the :is_new() and :is_old() helpers to tarantool.compat
options to simplify effective value checks (gh-8807).

Netbox

Introduced the new net.box module function from_fd for creating a new
connection from a socket file descriptor number (gh-8984).

Popen

Introduced new option inherit_fds for popen.new. The option takes
an array of file descriptor numbers that should be left open in the child
process (gh-8926).

Socket

Introduced new socket functions socket.socketpair, socket.from_fd, and
socket:detach (gh-8927).

SQL

Introduced the SHOW CREATE TABLE statement (gh-8098).

Box

Non-system spaces in a snapshot file are now sorted by their IDs.
As before, all non-system spaces are stored after system spaces (gh-7954).
[Breaking change] Dropped support for IPROTO_CALL16 from net.box
(i.e., the call_16 option and the call_16 method for net.box
connections) (gh-8636).
Added support for square brackets in procedure resolution for Lua calls
(gh-8604).
Added support for accepting IPROTO requests with a space or index name instead
of an identifier (gh-8146).
Added a new box.info parameter hostname (gh-8605).
Added the box.cfg.memtx_sort_threads parameter that specifies the number of
threads used to sort indexes keys on loading a memtx database. OpenMP is
not used to sort keys anymore (gh-3389).
Changed the output of the fselect index method from a table of strings to
a multi-line string and dropped the print and use_nbsp options. With the
yaml_pretty_multiline compat option enabled by default, multi-line strings
now look good in the console, so there's no need to return a table of strings
to prettify the fselect output anymore.
Allowed multiple updates to the same tuple field in a single call (gh-8226).
Introduced box.tuple.format that enables format definition for tuples
created via box.tuple.new (standalone tuples) (gh-4693).
[Breaking change] Disabled argument list syntax of box.tuple.new (this
was needed for gh-4693). It is possible to switch to the old behavior using
the compatibility option box_tuple_new_vararg.
Allowed truncation of temporary and local spaces in the read-only mode
(gh-5616).
Sped up index.select and index.pairs with the after option by up to 30%
in a synthetic test by eliminating an extra buffer allocation.
The function box.session.push is now deprecated. Consider using
box.broadcast instead (gh-8802).
Introduced the new function box.session.new for creating a new IPROTO
session from a socket file descriptor number (gh-8801).
Introduced the new lua_eval and lua_call object types for
box.schema.user.grant. Granting the 'execute' privilege on lua_eval
allows the user to execute an arbitrary Lua expression with the
IPROTO_EVAL request. Granting the 'execute' privilege on lua_call
allows the user to execute any global user-defined Lua function with
the IPROTO_CALL request (gh-8803).
[Breaking change] Introduced the new sql object type for
box.schema.user.grant. Now only users with the 'execute' privilege
granted on sql or universe can execute SQL expressions with the
IPROTO_EXECUTE or IPROTO_PREPARE requests. To revert to the old behavior
(no SQL access checks), use the sql_priv compat option (gh-8803).

Cfg

Implemented a way to set a table as box.cfg{} options value via
environment variables (gh-8051).

Cli

[Behavior change] Disabled starting the Lua REPL by default when running
Tarantool. Now, Tarantool yields the message that shows the command usage.
To run the Lua REPL, just set the -i flag. To pass a Lua script contents via
stdin, use dash (-) as the script name. For more information see a help
message by running tarantool -h (gh-8613).
Introduced new command-line options --name and --config (gh-8613).

Compat

The following compatibility options' defaults were switched to new behavior:
- yaml_pretty_multiline
- sql_seq_scan_default
- json_escape_forward_slash
- fiber_channel_close_mode
- fiber_slice_default
- box_cfg_replication_sync_timeout
More information on the new behavior can be found on the Module compat page.

Config

Initial version of the declarative server and cluster configuration
(gh-8724).
All vinyl options are now supported (gh-8861).
All feedback options are now supported (gh-8861).
The memtx.sort_threads option is now supported (gh-8861).
The bootstrap_leader option is now supported (gh-8861).
All flight recorder options are now supported (gh-8861).
All security hardening box.cfg{} options are now supported by
config (gh-8861).
Added a low priority environment configuration source, which looks into
TT_*_DEFAULT variables. It is useful to declare defaults (gh-8862).
It is now possible to access configuration from the application script using
the config:get() method (gh-8862).
All metrics configuration options from box.cfg{} are now supported
in the YAML config (gh-8861).
Added --help-env-list CLI option (gh-8862).
Introduced a non-public API for extending the declarative config
functionality in Tarantool Community Edition (gh-8862).
Improved the credentials applier: now it supports two-way synchronization
of roles and privileges for both users and roles (gh-8861).
Removed the sha1 and sha256 hash entries from the config credentials
schema as not planned for development (gh-8967).
Implemented a full password support in the config.credentials schema,
including a password setting, updating and removal for the chap-sha1
auth type (supported by both Tarantool Community Edition and Tarantool
Enterprise Edition) and the pap-sha256 (just for Enterprise Edition
where it is available) (gh-8967).
A group name from a topology defined by the cluster configuration doesn't
persisted anymore as box.cfg.cluster_name (gh-8862).
The default directory and file paths are changed to var/run/<...>,
var/log/<...>, var/lib/<...> and so on (gh-8862).
The directory that contains the configuration file is now added into the Lua
modules search paths (gh-8862).
Most of vshard options are now added in the config (gh-9007).
Added initial support of vshard (gh-9007).

Console

Now multiline commands can be used by setting the continuation marker
using the \set continuation command.

Debugger

Introduced readline support to tarantool debugger (gh-7738).

Test

Fuzz

Now LuaJIT can be fuzzed using grammar-based fuzzer (gh-4823).
Added a grammar-based SQL fuzzer (gh-4826).

Build

The zstd version was updated to pre-1.5.5 (gh-8391).
Added tt weak dependency. The tt command line utility combines
tarantoolctl and cartridge-cli functionality into one tool.
The change is a part of pushing tt to the public (tarantool/tt#286).
Tarantool does not depend on libgomp anymore (gh-7689).
Hardening against memory corruption attacks is now enabled by default on
FreeBSD (gh-7536).
Hardening against memory corruption attacks is now enabled by default on
AArch64 systems (gh-7536).
Added building static deb and rpm packages.
Dropped building distro-specific deb and rpm Tarantool packages.
Dropped building Tarantool packages with GC32. Now only GC64 packages
are shipped.
Exported all LuaJIT public functions from Tarantool binary (gh-3680).
Added the CMake option FIBER_STACK_SIZE to set the default fiber stack size.

Bugs fixed

Core

Fixed various bugs related to unsafe (i.e., from an unknown source) decoding
and validating of MsgPack extensions (ghs-73).
Fixed a crash in the net.box client when a remote space had
a field type that the client version did not support (gh-4632).
Fixed a bug when a space creation failed with a duplication error.
The issue occurred if the explicit and implicit space IDs were mixed.
Now the actual maximal space id is used to generate a new one (gh-8036).
Relaxed the tuple format requirements on tuples passed as the page starting
position to index:tuple_pos() or to the after option of index:select.
Now, Tarantool validates only the key parts of the index being used and all
primary indexes (gh-8511).
Fixed a crash that could happen when preparing a crash report on macOS
(gh-8445).
Fixed an integer overflow issue in net.box (ghs-121).
An IPROTO_EVENT packet now has the same sync number as the last
corresponding IPROTO_WATCH request (gh-8393).
Fixed a bug because of which a dirty (not committed to WAL) DDL record could
be written to a snapshot and cause a recovery failure (gh-8530).
[Breaking change] The key box.space._schema['cluster'] is renamed to
'replicaset_uuid'. That is not expected to be breaking because _schema is
an internal system space, but the key was visible in public and documented
(gh-5029).
[Breaking change] The table box.info.cluster is renamed to
box.info.replicaset. The behavior can be reverted using the compat option
box_info_cluster_meaning
(https://tarantool.io/compat/box_info_cluster_meaning) (gh-5029).
Fixed a bug when a tuple could be inserted even if it violates a unique
constraint of a functional index that has a nullable part (gh-8587).
Fixed a bug when Tarantool failed to decode a request containing an unknown
IPROTO key. The bug resulted in broken connectivity between Tarantool 2.10
and 2.11 (gh-8745).
Fixed the node writing an empty 00000000000000000000.xlog file regardless of
the actual vclock when interrupted during the initial box.cfg() call
(gh-8704).
Fixed a bug causing the ER_CURSOR_NO_TRANSACTION failure for transactions
on synchronous spaces when the on_commit/on_rollback triggers are set
(gh-8505).
Fixed a bug causing the effective session and user are not propagated to
box.on_commit and box.on_rollback trigger callbacks when transaction
is synchronous (gh-8742).
Eliminated implicit conversion of unprintable utf-8 strings to binary blobs
when encoded in YAML. Now unprintable characters are encoded as escaped utf-8
code points, for example, \x80 or \u200B (gh-8756).
Fixed a bug when MVCC sometimes lost gap record (gh-8326).
Fixed a bug when MVCC rollback of prepared statement could break internal
invariants (gh-8648).
Now MVCC engine automatically aborts a transaction if it reads changes
of a prepared transaction and this transaction is aborted (gh-8654).
Disabled the backtrace collection feature on the AArch64 Linux platform
(gh-8572).
Fixed a crash that could happen when Tarantool is started in the
background mode
(gh-6128).
Fixed the collection of fiber backtraces on the M1/M2 macOS platform
(gh-8074).
Fixed a bug that caused writing incorrect values into the stream_id field
of xlog headers (gh-8783).
Fixed decoding of escape sequences for single-byte character codes from YAML.
Before the fix, single-byte character codes between 0x80 and 0xff would
be erroneously converted to two-byte UTF-8 code points, for example, \x80
would be decoded as \uC280 (gh-8782).
Fixed a bug when a space that is referenced by a foreign key could not
be truncated even if the referring space was empty (gh-8946).
Now foreign keys from non-temporary to temporary and from non-local to local
spaces are prohibited since they can potentially break foreign key consistency
(gh-8936).
Fixed a crash when a collation used by a space was deleted (gh-4544).
Fixed a crash that could happen when Tarantool is compiled by clang
version 15 and above with enabled AddressSanitizer
(tarantool/tarantool-qa#321).
Fixed a bug because of which it was impossible to set the hint option
to true for TREE indexes (gh-8937).
Now index_object.parts.exclude_null always contains false rather than
null when it is actually false (gh-8649).

Memtx

Fixed the ability for the memtx_memory parameter to specify a memory size
less than required for the stable operation of the engine (gh-7389).
Fixed a heap-use-after-free bug in the transaction manager, which could occur
when performing a DDL operation concurrently with a transaction on the same
space (gh-8781).

Vinyl

Vinyl space and index directories are now created on demand.
They are removed as soon as they become empty (gh-8441).
Fixed a heap-use-after-free bug in the Vinyl read iterator caused by a race
between a disk read and a memory dump task. The bug could lead to a crash or
an invalid query result (gh-8852).

Replication

Fixed a bug related to box.info.replication[...].upstream being stuck in the "connecting"
state for several minutes after a replica DNS record change (gh-7294).
Fixed an issue when the nodes synchronizing with
a hung leader reported the leader as alive.
This behavior led to the delay of the new elections (gh-7515).
Fixed a bug that occurred on applier failure: a node could start an election
without having a quorum to do this (gh-8433).
Fixed a bug that allowed to change replicaset UUID via _schema space.
Fixed a bug when new instances could try to register via an anon instance
which previously failed to apply box.cfg{replication_anon = false}.
Fixed a possible failure to promote the desired node by box.ctl.promote() on
a cluster with nodes configured with election_mode = "candidate" (gh-8497).
Fixed nodes configured with election_mode = 'candidate' spuriously detecting
a split-vote when another candidate should win with exactly a quorum of votes
for it (gh-8698).
Fixed a crash when using transactions with the linearizable isolation level during
a replica reconnect (gh-7991).
Fixed a possible crash on bootstrap with box.cfg.bootstrap_strategy = 'auto'
when some of the bootstrapping nodes were stopped (gh-8757).

LuaJIT

Backported patches from the vanilla LuaJIT trunk (gh-8069). The following issues
were fixed as part of this activity:

Fixed successful math.min/math.max call with no args (gh-6163).
Fixed inconsistencies in math.min/math.max calls with a NaN arg (gh-6163).
Fixed pcall() call without arguments on arm64.
Fixed assembling of IR_{AHUV}LOAD specialized to boolean for aarch64.
Fixed constant rematerialization on arm64.
Fixed emit_rma() for the x64/GC64 mode for non-mov instructions.
Limited Lua C library path with the default PATH_MAX value of 4096 bytes.
The JIT engine was disabled by default on macOS platforms to improve
the user experience. If necessary, you can enable it with jit.on (gh-8252).
Backported patches from the vanilla LuaJIT trunk (gh-8516). The following issues
were fixed as part of this activity:
Fixed IR_LREF assembling for the GC64 mode on x86_64.
Fixed canonicalization of +-0.0 keys for IR_NEWREF.
Fixed result truncation for bit.rol on x86_64 platforms.
Fixed saved bytecode encapsulated in ELF objects.
Fixed lua_yield() invocation inside C hooks.
Fixed memory chunk allocation beyond the memory limit.
Fixed TNEW load forwarding with instable types.
Fixed use-def analysis for BC_VARG, BC_FUNCV.
Backported patches from the vanilla LuaJIT trunk (gh-8825). The following issues
were fixed as part of this activity:
Fixed BC_UCLO insertion for returns.
Fixed recording of BC_VARG with unused vararg values.
Initialization instructions on trace are now emitted only for the first
member of a union.

Lua

The box.stat.memtx function is now callable, it returns
all memtx statistics. The box.stat.memtx.tx() function
is now equivalent to the box.stat.memtx().tx function (gh-8448).
Exported the missing AUTH_TYPE key to box.iproto.key (gh-8574).
Stripped the unwanted IPROTO_ prefix from the POSITION, AFTER_POSITION,
and AFTER_TUPLE keys exported to box.iproto.key (gh-8577).
Fixed the xlog reader Lua module to show unknown row header fields. Before
this change the xlog reader silently skipped them.

Fio

The default permission mode for fio.open() was changed for newly
created files to 0666 (before umask) (gh-7981).

Http client

Fixed a bug where a response body cannot be decoded (gh-8363).
Fixed the Transfer-Encoding: chunked setting being enabled even if
the Content-Length header exists for stream requests (gh-8744).

Netbox

Fixed a heap-use-after-free bug in the function creating a tuple format Lua
object for net.box (gh-8889).

SQL

The ANY field type now supports collation in SQL (gh-8070).
Now the ARRAY, MAP, and INTERVAL values cannot be used as
arguments in the ORDER BY clause (gh-6668).
Fixed incorrect conversion of an integer greater than INT64_MAX or
less than 0 to a decimal number during SQL arithmetic operations (gh-8460).
Fixed an assertion when selecting tuples with incomplete internal
format (gh-8418).
Fixed integer overflow issues in built-in functions (ghs-119).
Fixed a Use-After-Free vulnerability in the WITH RECURSIVE clause
(ghs-119).
Fixed a possible assertion or segmentation fault when optimizing
INSERT INTO ... SELECT FROM (gh-8661).
Fixed an integer overflow issue and added a check for the printf() failure due
to too large size (ghs-122).

Box

Fixed a bug where box.cfg.force_recovery doesn't work when there are
no user spaces in a snapshot (gh-7974).
Fixed a bug when large numbers were encoded incorrectly by msgpackffi.
It could lead to wrong select results with large number keys (gh-6119).
Fixed a crash on an unknown option of the function (gh-8463).
The takes_raw_args option is now handled correctly
during downgrading (gh-8457).
Fixed several memory leaks on tuple encoding failures (gh-7939).

Fixed the ability to drop _vinyl_deferred_delete system space (gh-5279).

Fixed a bug (bad error message) in pagination related to the validation of the
after position option of the :select and :pairs methods of space and
index objects (gh-8716).
Fixed a bug when it was not possible to access spaces named _stream and
_stream_space_cache and indexes named _space using the net.box stream
objects (gh-8598).
Fixed the memory leaks caused by the multi-statement transaction errors in the
space index building and the space format checking operations (gh-8773).
[Breaking change] The cord_slab_cache symbol was removed from the public
API export (gh-7124).
Fixed the privilege check when using spaces with functional indexes and
constraints (gh-7873).
Fixed the inability to insert an integral number into a double-formatted
field (gh-7483).
Fixed a bug in the box console implementation because of
which the language parameter was shared between connected
clients (gh-8817).
Fixed the invalid memory access in a corner case of a specialized comparison
function (gh-8899).
Fixed a crash when box.iproto.override was called with an unconfigured box.
Now, an error is raised instead (gh-8975).

Config

Fixed an error message if the cluster configuration was not provided or the
instance was not found in the cluster configuration during reload (gh-8862).
Support parent directories creation for options that accept a directory or a
file (gh-8862).
Create parent directories for console.socket and log.file (gh-8862).
Create the process.work_dir directory (gh-8862).
Consider all the paths as relative to process.work_dir when creating
necessary directories (gh-8862).

Console

Fixed console.local_print() failing on non-string arguments, which led to
some rare errors. For example, when connecting via tarantoolctl to cartridged
tarantool with incorrect credentials, a cdata error was passed through the
local_print(), which failed to interpret it (gh-8374).

Datetime

Fixed a bug with buffer overflow in tnt_strptime (gh-8502).
Fixed an error in datetime.set when timestamp is passed along with nsec,
usec, or msec (gh-8583).
Fixed errors when the string representation of a datetime object had
a negative nanosecond part (gh-8570).
Fixed a bug raising a false positive error when creating new intervals with
range boundary values (gh-8878).

Msgpack

Fixed decoding datetime intervals with fields larger than possible int32
values (gh-8887).

Build

Enabled compiler optimizations for static build dependencies, which were
erroneously disabled in version 2.10.3 (gh-8606).