This is the first release that supports Full-Text Indexes.
Merged PRs
dolt
- 6398: Implemented Full-Text indexes
Technically functioning, although not the most ergonomic implementation. I need add a bit of user convenience and additional checking, such as removing the additional tables fromdolt status
, automatically adding the additional tables when the parent is added viadolt add
, adding a user-facing write barrier on the tables, etc. Also need quite a few more tests, but the basics are in via the engine tests and very, very few bats tests.
I'm definitely going to expand the testing in the future, as well as implementing the aforementioned features and more. But for now, this should at least allow us to release unless there are major problems that I've not caught.
go-mysql-server
- 1871: Full-Text Indexes
This is a partial implementation of Full-Text indexes. There is still quite a bit to finish on the GMS side (as can be seen from the copious amount ofTODO
s), but this shows the broad strokes of how it's implemented, along with most of the "difficult" design choices being implemented. The major choice that has not yet been finalized is how to deal withFTS_DOC_ID
, as it's anAUTO_INCREMENT
column in MySQL, but that would not play well with Dolt merging. I already have ideas on how to handle that (taking into account remotes, etc.), but that would come from a later PR.
https://docs.google.com/document/d/1nGyYg461AhxQjFLzhEEj01XMz0VaTBaBaA44WNu0fc4/edit
Quite a few things have changed from the initial design doc, mostly based on feedback during the meeting, however some of it was post-meeting. There are three tables instead of 1: Config (stores table-specific information shared across all indexes), WordToPos (maps words to an ID and position, not fully used in the default search), and Count (used to calculate relevancy, also not fully used in the default search). I was planning on convertingMATCH ... AGAINST ...
to a join between the tables, which would work when fetching results, butMATCH ... AGAINST ...
may also be used as a result, which necessitated writing all of the functionality anyway, so the join plan was dropped.
Last thing to mention, is that I'm pretty sure that Full-Text indexes actually do a full table scan. It seems weird, but AFAICT the indexes are used to quickly calculate relevancy for each search mode. It seems that, for overly large tables, the search time increases even when other index operations continue to operate nearly instantaneously.
I've tagged two people for review to make it a bit easier. Of course, feel free to take a look at more if you desire.
@reltuk Thesql/fulltext/fulltext.go
file is an expansion of the file you've previously reviewed (all still kept to a single file for now). To complement it and see how it'll be implemented on the Dolt side, you can look atmemory/table.go
. Dolt's table editor will be similar, and the merge paths will only use theFulltextEditor
, which special logic to interface with it from those paths.
@max-hoffman Take a look at the analyzer changes, along with thesql/plan/ddl.go
file. You'll probably need to referencesql/fulltext/fulltext.go
as well. - 1863: Bump google.golang.org/grpc from 1.37.0 to 1.53.0
Bumps google.golang.org/grpc from 1.37.0 to 1.53.0.
Closed Issues
- 6382: mysql_server may panic when performing a join query on dolt_history_$tablename and dolt_commits tables.
- 1902: Valid join query erroneously reported as ambiguous
- 1881: duplicate column names in a table alias should be prevented.
- 1753: "SELECT * FROM tbl" has incorrect schema when table has default values.
- 1745: Allow table names with special characters when MySql allows them
Latency
Read Tests | MySQL | Dolt | Multiple |
---|---|---|---|
covering_index_scan | 2.07 | 3.02 | 1.5 |
groupby_scan | 13.22 | 17.95 | 1.4 |
index_join | 1.25 | 4.74 | 3.8 |
index_join_scan | 1.21 | 2.3 | 1.9 |
index_scan | 32.53 | 57.87 | 1.8 |
oltp_point_select | 0.14 | 0.48 | 3.4 |
oltp_read_only | 2.66 | 8.28 | 3.1 |
select_random_points | 0.3 | 0.81 | 2.7 |
select_random_ranges | 0.37 | 1.18 | 3.2 |
table_scan | 32.53 | 56.84 | 1.7 |
types_table_scan | 74.46 | 167.44 | 2.2 |
reads_mean_multiplier | 2.4 |
Write Tests | MySQL | Dolt | Multiple |
---|---|---|---|
bulk_insert | 0.001 | 0.001 | 1.0 |
oltp_delete_insert | 4.74 | 5.88 | 1.2 |
oltp_insert | 2.22 | 2.97 | 1.3 |
oltp_read_write | 5.88 | 15.83 | 2.7 |
oltp_update_index | 2.3 | 3.02 | 1.3 |
oltp_update_non_index | 2.3 | 3.02 | 1.3 |
oltp_write_only | 3.36 | 7.56 | 2.2 |
types_delete_insert | 4.91 | 6.55 | 1.3 |
writes_mean_multiplier | 1.5 |
Overall Mean Multiple | 2.1 |
---|