Version 6.3.0

Released: May 23rd 2024

➡️➡️➡️ DOWNLOAD HERE ⬅️⬅️⬅️

📢📢📢Announcement blog post📢📢📢

Major changes

• Issue #839 Implemented float_vector data type; implemented vector search.
• Issue #1673 INNER/LEFT JOIN (beta stage).
• Issue #1744 Implemented autodetection of date formats for timestamp fields.
• Issue #1720 Changed Manticore Search license from GPLv2-or-later to GPLv3-or-later.
• Commit 7a55 Running Manticore in Windows now requires Docker to run Buddy.
• Issue #1541 Added a REGEX full-text operator.
• Issue #2091 Ubuntu Noble 24.04 support.
• Commit 514d Revamp of time operations for better performance and new date/time functions:
  • CURDATE() - Returns current date in local timezone
  • QUARTER() - Returns the integer quarter of the year from a timestamp argument
  • DAYNAME() - Returns the weekday name for a given timestamp argument
  • MONTHNAME() - Returns the name of the month for a given timestamp argument
  • DAYOFWEEK() - Returns the integer weekday index for a given timestamp argument
  • DAYOFYEAR() - Returns the integer day of the year for a given timestamp argument
  • YEARWEEK() - Returns the integer year and the day code of the first day of current week for a given timestamp argument
  • DATEDIFF() - Returns the number of days between two given timestamps
  • DATE() - Formats the date part from a timestamp argument
  • TIME() - Formats the time part from a timestamp argument
  • timezone - Timezone used by date/time-related functions.
• Commit 30e7 Added range, histogram, date_range, and date_histogram aggregates to the HTTP interface and similar expressions into SQL.

Minor changes

• Issue #1285 Support of Filebeat versions 8.10 - 8.11.
• Issue #1771 ALTER TABLE ... type='distributed'.
• Issue #1788 Added the ability to copy tables using the CREATE TABLE ... LIKE ... WITH DATA SQL statement.
• Issue #2072 Optimized the table compacting algorithm: Previously, both manual OPTIMIZE and automatic auto_optimize processes would first merge chunks to ensure the count did not exceed the limit, and then expunge deleted documents from all other chunks containing deleted documents. This approach was sometimes too resource-intensive and has been disabled. Now, chunk merging occurs solely according to the progressive_merge setting. However, chunks with a high number of deleted documents are more likely to be merged.
• Commit ce6c Added protection against loading a secondary index of a newer version.
• Issue #1417 Partial replace via REPLACE INTO ... SET.
• Commit 7c16 Updated default merge buffer sizes: .spa (scalar attrs): 256KB -> 8MB; .spb (blob attrs): 256KB -> 8MB; .spc (columnar attrs): 1MB, no change; .spds (docstore): 256KB -> 8MB; .spidx (secondary indexes): 256KB buffer -> 128MB memory limit; .spi (dictionary): 256KB -> 16MB; .spd (doclists): 8MB, no change; .spp (hitlists): 8MB, no change; .spe (skiplists): 256KB -> 8MB.
• Issue #1859 Added composite aggregation via JSON.
• Commit 216b Disabled PCRE.JIT due to issues with some regex patterns and no significant time benefit.
• Commit 55cd Added support for vanilla Galera v.3 (api v25) (libgalera_smm.so from MySQL 5.x).
• Commit 86f9 Changed metric suffix from _rate to _rps.
• Commit c0c1 Improved docs about balancer HA support.
• Commit d1d2 Changed index to table in error messages; fixed bison parser error message fixup.
• Commit fd26 Support manticore.tbl as table name.
• Issue #1105 Support for running indexer via systemd (docs). ❤️ Thank you, @subnix for the PR.
• Issue #1294 Secondary indexes support in GEODIST().
• Issue #1394 Simplified SHOW THREADS.
• Issue #1424 Added support for the default values (agent_connect_timeout and agent_query_timeout) for create distributed table statement.
• Issue #1442 Added expansion_limit search query option that overrides searchd.expansion_limit.
• Issue #1448 Implemented ALTER TABLE for int->bigint conversion.
• Issue #146 Meta information in MySQL response.
• Issue #1494 SHOW VERSION.
• Issue #1582 Support of deleting documents by id array via JSON.
• Issue #1589 Improve error "unsupported value type".
• Issue #1634 Added Buddy version into SHOW STATUS.
• Issue #1641 Match requests optimisation in case of zero docs for a keyword.
• Issue #1712 Added conversion to bool attribute from the string value of "true" and "false" on posting data.
• Issue #1713 Added access_dict table and searchd option.
• Issue #1767 Added new options: expansion_merge_threshold_docs and expansion_merge_threshold_hits to the searchd section of the config; made the threshold for merging tiny terms of the expanded terms configurable.
• Issue #1768 Added display of the last command time in @@system.sessions.
• Issue #1806 Upgraded Buddy Protocol to version 2.
• Issue #1810 Implemented additional request formats to the /sql endpoint to ease integration with libraries.
• Issue #1825 Added an Info section to SHOW BUDDY PLUGINS.
• Issue #1837 Improved memory consumption in CALL PQ with large packets.
• Issue #1853 Switched compiler from Clang 15 to Clang 16.
• Issue #1857 Added string comparison. ❤️ Thank you, @etcd for the PR.
• Issue #1915 Added support for joined stored fields.
• Issue #1937 Log client's host:port in query-log.
• Issue #1981 Fixed wrong error.
• Issue #1983 Introduced support for verbosity levels for the query plan via JSON.
• Issue #2010 Disabled logging of queries from Buddy unless log_level=debug is set.
• Issue #2035 Linux Mint 21.3 suppport.
• Issue #2056 Manticore couldn't be built with Mysql 8.3+.
• Issue #2112 DEBUG DEDUP command for real-time table chunks that can experience duplicate entries after attaching a plain table containing duplicates.
• Issue #212 Added time to SHOW QUERIES.
• Issue #218 Handle @timestamp column as timestamp.
• Issue #252 Implemented logic to enable/disable buddy plugins.
• Issue #254 Updated composer to a fresher version where recent CVEs are fixed.
• Issue #340 Minor optimization in Manticore systemd unit related with RuntimeDirectory.
• Issue #51 Added rdkafka support and updated to PHP 8.3.3.
• Issue #527 Support of attaching an RT table. New command ALTER TABLE ... RENAME.

Breaking changes and deprecations

• ⚠️Issue #1436 Fixed an IDF calculation issue. local_df is now a default. Improved the master-agent search protocol (version updated). If you are running Manticore Search in a distributed environment with multiple instances, make sure to first upgrade the agents, then the masters.
• ⚠️Issue #1572 Added replication of distributed tables and updated the replication protocol. If you are running a replication cluster, you need to:
  • First, cleanly stop all your nodes
  • Then, start the node that was stopped last with --new-cluster, using the tool manticore_new_cluster in Linux.
  • Read about restarting a cluster for more details.
• ⚠️Issue #1763 HTTP API endpoint aliases /json/* have been deprecated.
• ⚠️Issue #1982 Changed profile to plan in JSON, added query profiling for JSON.
• ⚠️Commit e235 manticore-backup doesn't backup plugin_dir anymore.
• ⚠️Issue #171 Migrated Buddy to Swoole to improve performance and stability. When switching to the new version, ensure all Manticore packages are updated.
• ⚠️Issue #196 Merged all core plugins into Buddy and changed the core logic.
• ⚠️Issue #2107 Treating document IDs as numbers in /search responses.
• ⚠️Issue #38 Added Swoole, disabled ZTS, and removed the parallel extension.

Replication-related changes

• Commit 3376 Fixed replication error on SST of large files.
• Commit 6d36 Added a retry mechanism to replication commands; fixed replication join failure on a busy network with packet loss.
• Commit 842e Changed the FATAL message in replication to a WARNING message.
• Commit 8c32 Fixed the calculation of the gcache.page_size for replication clusters without tables or with empty tables; also fixed saving and loading of the Galera options.
• Commit a2af Added functionality to skip the update nodes replication command on the node that joins the cluster.
• Commit c054 Fixed deadlock during replication on updating blob attributes versus replacing documents.
• Commit e80d Added replication_connect_timeout, replication_query_timeout, replication_retry_delay, replication_retry_count searchd config options to control network during replication similar to searchd.agent_* but with different defaults.
• Issue #1356 Fixed replication nodes retry after some nodes are missed and name resolution of these nodes failed.
• Issue #1445 Fixed the replication log verbosity level at the show variables.
• Issue #1482 Fixed a replication issue for a joiner node connecting to a cluster on a pod restarted in Kubernetes.
• Issue #1962 Fixed a long wait for replication to alter on empty cluster with an invalid node name.

Bug fixes

• Commit 8a48 Fixed unused matches cleanup in count distinct which could cause a crash.
• Issue #1569 Binary log is now written with transaction granularity.
• Issue #2089 Fixed a bug associated with 64-bit IDs that could result in a "Malformed packet" error when inserting via MySQL, leading to corrupted tables and duplicate IDs.
• Issue #2160 Fixed dates being inserted as if they were in UTC instead of local time zone.
• Issue #2177 Fixed a crash that occurred when performing a search in a real-time table with a non-empty index_token_filter.
• Issue #2209 Changed duplicate filtering in RT columnar storage to fix crashes and wrong query results.
• Commit 001d Fixed html stipper corrupting memory after processing a joined field.
• Commit 00eb Avoided rewinding stream after flush to prevent miscommunication issues with mysqldump.
• Commit 0553 Don't wait for preread to finish if it has not started.
• Commit 055a Fixed large Buddy output string to split among multiple lines in the searchd log.
• Commit 0a88 Moved MySQL interface warning about failing header debugv verbosity level.
• Commit 150a Fixed race condition on multiple clusters management operations; prohibited creating multiple clusters with the same name or path.
• Commit 2e40 Fixed implicit cutoff in fulltext queries; split MatchExtended into template partD.
• Commit 75f5 Fixed the discrepancy of index_exact_words between indexing and loading the table to the daemon.
• Commit 7643 Fixed missed error message for invalid cluster deletion.
• Commit 7a03 Fixed CBO vs queue union; fixed CBO vs RT pseudo sharding.
• Commit 7b4e When starting without the secondary index (SI) library and parameters in the configuration, the misleading warning message 'WARNING: secondary_indexes set but failed to initialize secondary library' was issued.
• Commit 8496 fixed hit sorting in quorum.
• Commit 8973 Fixed issue with upper case options in the ModifyTable plugin.
• Commit 9935 Fixed restoring from a dump with empty json values (represented as NULL).
• Commit a28f Fixed SST timeout at the joiner node when receiving SST by using pcon.
• Commit b5a5 Fixed a crash on selecting aliased string attribute.
• Commit c556 Added query transform of the term into =term of full-text query with the morphology_skip_fields field.
• Commit cdc3 Added missing config key (skiplist_cache_size).
• Commit cf6e Fixed crash at the expression ranker with large complex query.
• Commit e513 Fixed fulltext CBO vs invalid index hints.
• Commit eb05 Interrupt preread on shutdown for faster shutdown.
• Commit f945 Changed stack calculation for fulltext queries to avoid a crash in case of a complex query.
• Issue #1262 Fixed a crash of the indexer when indexing an SQL source with multiple columns having the same name.
• Issue #1273 Return 0 instead of for non-existing sysvars.
• Issue #1289 Fixed indextool error when checking external files of the RT table.
• Issue #1335 Fixed query parse error due to multi wordform inside the phrase.
• Issue #1364 Added the replay of empty binlog files with old binlog versions.
• Issue #1365 Fixed removal of the last empty binlog file.
• Issue #1372 Fixed incorrect relative paths (converted to absolute from the daemon's start directory) after changes in data_dir affect the current work directory on daemon start.
• Issue #1393 Slowest time degradation in hn_small: fetch/cache cpu info on daemon startup.
• Issue #1395 Fixed warning regarding missing external file during index load.
• Issue #1402 Fixed crash at global groupers on free of data ptr attributes.
• Issue #1403 _ADDITIONAL_SEARCHD_PARAMS is not working.
• Issue #1427 Fixed per table agent_query_timeout being replaced by the default query option agent_query_timeout.
• Issue #1444 Fixed crash at the grouper and ranker when using packedfactors() with multiple values per match.
• Issue #1458 Manticore crashes on frequent index updates.
• Issue #1481 Fixed crash on cleanup of the parsed query after parse error.
• Issue #1484 Fixed HTTP JSON requests not being routed to buddy.
• Issue #1499 JSON attribute root value couldn't be an array. Fixed.
• Issue #1507 Fixed crash on table recreation within the transaction.
• Issue #1515 Fixed expansion of the short forms of the RU lemmas.
• Issue #1579 Fixed JSON and STRING attributes usage in date_format expression.
• Issue #1580 Fixed the grouper for multiple aliases to JSON fields.
• Issue #1594 Wrong total_related in dev: fixed implict cutoff vs limit; added better fullscan detection in json queries.
• Issue #1603 Fixed JSON and STRING attributes usage in all date expression.
• Issue #1609 crash on using LEVENSHTEIN().
• Issue #1612 Fixed memory corruption after a search query parse error with highlight.
• Issue #1614 Disabled wildcard expansion for terms shorter than min_prefix_len / min_infix_len.
• Issue #1617 Altered behavior to not log an error if Buddy handles the request successfully.
• Issue #1635 Fixed total at the meta of the search query for queries with limit set.
• Issue #1640 Impossible to use a table with an upper case via JSON in plain mode.
• Issue #1643 Provided a default SPH_EXTNODE_STACK_SIZE value.
• Issue #1646 Fixed SphinxQL log of negative filter with ALL/ANY on MVA attribute.
• Issue #1660 Fix application of docid killlists from other indexes. ❤️ Thank you, @raxoft for the PR.
• Issue #1668 Fixed missed matches due to early exit on raw index full scan (without any indexes iterators); removed cutoff from the plain row iterator.
• Issue #1671 Fixed FACET error when querying a distributed table with agent and local tables.
• Issue #1690 Fixed crash on histogram estimation for large values.
• Issue #1692 crash on alter table tbl add column col uint.
• Issue #1710 Empty result for condition WHERE json.array IN (<value>).
• Issue #172 Fixed an issue with TableFormatter when sending request to /cli.
• Issue #1742 CREATE TABLE wasn't failing in case of a missing wordforms file.
• Issue #1762 The order of attributes in RT tables now follows the configuration order.
• Issue #1765 HTTP bool query with 'should' condition returns incorrect results.
• Issue #1769 Sorting by string attributes does not work with SPH_SORT_ATTR_DESC and SPH_SORT_ATTR_ASC.
• Issue #177 Disabled the Expect: 100-continue HTTP header for curl requests to Buddy.
• Issue #1791 crash caused by GROUP BY alias.
• Issue #1792 SQL meta summary shows wrong time on windows.
• Issue #1794 Fixed a single-term performance drop with JSON queries.
• Issue #1798 Incompatible filters didn't raise an error on /search.
• Issue #1802 Fixed ALTER CLUSTER ADD and JOIN CLUSTER operations to wait for each other, preventing a race condition where ALTER adds a table to the cluster while the donor sends tables to the joiner node.
• Issue #1811 Incorrect handling of /pq/{table}/* requests.
• Issue #1816 UNFREEZE wasn't working in some cases.
• Issue #183 Fixed an issue with MVA restoration in some cases.
• Issue #1849 Fixed indextool crash on shutdown if used with MCL.
• Issue #1866 Fixed unnecessary url decoding for /cli_json requests.
• Issue #1872 change plugin_dir set logic on daemon start.
• Issue #1874 alter table ... exceptions fails.
• Issue #1891 Manticore crashes with signal 11 when inserting data.
• Issue #1920 Reduced throttling for low_priority.
• Issue #1924 Mysqldump + mysql restore bug.
• Issue #1929 Overriding in charset_table was not working in some cases.
• Issue #1951 Fixed incorrect creation of the distributed table in the case of a missing local table or incorrect agent description; now returns an error message.
• Issue #1972 Implemented a FREEZE counter to avoid freeze/unfreeze issues.
• Issue #1980 Obey query timeout in OR nodes. Previously max_query_time could be not working in some cases.
• Issue #1986 Failed to rename new to current [manticore.json].
• Issue #1988 A full-text query could ignore a SecondaryIndex CBO hint.
• Issue #1990 Fixed expansion_limit to slice final result set for call keywords from multiple disk chunks or RAM chunks.
• Issue #1994 wrong external files.
• Issue #2021 A few manticore-executor processes could be left running after stopping Manticore.
• Issue #2029 Crash using Levenshtein Distance.
• Issue #2037 Got error after multiple max operator ran on an empty index.
• Issue #2052 crash on multi-group with JSON.field.
• Issue #2067 Manticore was crashing on incorrect request to _update.
• Issue #2069 Fixed an issue with string filter comparators for closed ranges in the JSON interface.
• Issue #2082 alter failed when the data_dir path was located on a symlink.
• Issue #2102 Improved special handling of SELECT queries in mysqldump to ensure the resulting INSERT statements are compatible with Manticore.
• Issue #2103 Thai chars were in the wrong charsets.
• Issue #2124 Crash if I use an SQL with a reserved word.
• Issue #2154 Tables with wordforms couldn't be imported.
• Issue #2176 Fixed a crash that occurred when the engine parameter was set to 'columnar' and duplicate IDs were added via JSON.
• Issue #223 Proper error when trying to insert a document w/o schema and w/o column names.
• Issue #239 Auto-schema multi-line insert could fail.
• Issue #399 Added an error message on indexing if an id attribute is declared at the data source.
• Issue #59 Manticore cluster breakdown.
• Issue #68 optimize.php would crash if percolate table was present.
• Issue #77 Fixed errors when deploying on Kubernetes.

Related with manticore-backup

• Issue #97 Set VIP HTTP port as default when available.
  Various improvements: improved versions check and streaming ZSTD decompression; added user prompts for version mismatches during restore; fixed incorrect prompting behavior for different versions on restore; enhanced decompression logic to read directly from the stream rather than into working memory; added --force flag
• Commit 3b35 Added backup version display after Manticore search start to identify issues at this stage.
• Commit ad2e Updated error message for failed connections to the daemon.
• Commit ce5e Fixed issue with converting absolute root backup paths to relative and removed writeable check on restore to enable restoration from different paths.
• Commit db7e Added sorting to the file iterator to ensure consistency across various situations.
• Issue #106 Backup and restore of multiple configurations.
• Issue #91 Added defattr to prevent unusual user permissions in files after installation on RHEL.
• Issue #91 Added extra chown to ensure files default to the root user in Ubuntu.

Related with MCL (columnar, secondary, knn libs)

• Commit f104 Vector search support.
• Commit 2169 Fixed cleanup of temporary files during the interrupted setup of the secondary index build. This resolves the issue where the daemon exceeded the open files limit when creating tmp.spidx files.
• Commit 709b Use separate streamvbyte library for columnar and SI.
• Commit 1c26 Added a warning that columnar storage doesn't support json attrs.
• Commit 3acd Fixed data unpacking in SI.
• Commit 574c Fixed a crash on saving a disk chunk with mixed rowwise and columnar storage.
• Commit e87f Fixed SI iterator being hinted at an already processed block.
• Issue #1474 Update is broken for rowwise MVA column with columnar engine.
• Issue #1510 Fixed crash when aggregating to a columnar attribute used in HAVING.
• Issue #1519 Fixed crash in expr ranker on using columnar attribute.

Related with Docker

• ❗Issue #42 Support of plain indexation via environment variables.
• ❗Issue #47 Improved flexibility of configuration via environment vars.
• Issue #54 Improved the backup and restore processes for Docker.
• Issue #77 Improved entrypoint to handle backup restoration on first start only.
• Commit a27c Fixed query logging to stdout.
• Issue #38 Mute BUDDY warnings if EXTRA is not set.
• Issue #71 Fixed hostname in manticore.conf.sh.