This is the release note of v3.1.0.

This is not something you have to read from top to bottom to learn about the summary of Optuna v3.1. The recommended way is reading the release blog.

Highlights

New Features

CMA-ES with Margin

CMA-ES CMA-ES with Margin

“The animation is referred from https://github.com/EvoConJP/CMA-ES_with_Margin, which is distributed under the MIT license.”

CMA-ES achieves strong performance for continuous optimization, but there is still room for improvement in mixed-integer search spaces. To address this, we have added support for the "CMA-ES with Margin" algorithm to our CmaEsSampler, which makes it more efficient in these cases. You can see the benchmark results here. For more detailed information about CMA-ES with Margin, please refer to the paper “CMA-ES with Margin: Lower-Bounding Marginal Probability for Mixed-Integer Black-Box Optimization - arXiv”, which has been accepted for presentation at GECCO 2022.

import optuna
from optuna.samplers import CmaEsSampler

def objective(trial):
    x = trial.suggest_float("y", -10, 10, step=0.1)
    y = trial.suggest_int("x", -100, 100)
    return x**2 + y
 
study = optuna.create_study(sampler=CmaEsSampler(with_margin=True))
study.optimize(objective)

Distributed Optimization via NFS

JournalFileStorage, a file storage backend based on JournalStorage, supports NFS (Network File System) environments. It is the easiest option for users who wish to execute distributed optimization in environments where it is difficult to set up database servers such as MySQL, PostgreSQL or Redis (e.g. #815, #1330, #1457 and #2216).

import optuna
from optuna.storages import JournalStorage, JournalFileStorage

def objective(trial):
    x = trial.suggest_float("x", -100, 100)
    y = trial.suggest_float("y", -100, 100)
    return x**2 + y
 
storage = JournalStorage(JournalFileStorage("./journal.log"))
study = optuna.create_study(storage=storage)
study.optimize(objective)

For more information on JournalFileStorage, see the blog post “Distributed Optimization via NFS Using Optuna’s New Operation-Based Logging Storage” written by @wattlebirdaz.

A Brand-New Redis Storage

We have replaced the Redis storage backend with a JournalStorage-based one. The experimental RedisStorage class has been removed in v3.1. The following example shows how to use the new JournalRedisStorage class.

import optuna
from optuna.storages import JournalStorage, JournalRedisStorage

def objective(trial):
    …
 
storage = JournalStorage(JournalRedisStorage("redis://localhost:6379"))
study = optuna.create_study(storage=storage)
study.optimize(objective)

Dask.distributed Integration

DaskStorage, a new storage backend based on Dask.distributed, is supported. It allows you to leverage distributed capabilities in similar APIs with concurrent.futures. DaskStorage can be used with InMemoryStorage, so you don't need to set up a database server. Here's a code example showing how to use DaskStorage:

import optuna
from optuna.storages import InMemoryStorage
from optuna.integration import DaskStorage
from distributed import Client, wait

def objective(trial):
    ...

with Client("192.168.1.8:8686") as client:
    study = optuna.create_study(storage=DaskStorage(InMemoryStorage()))
    futures = [
        client.submit(study.optimize, objective, n_trials=10, pure=False)
        for i in range(10)
    ]
    wait(futures)
    print(f"Best params: {study.best_params}")

Setting up a Dask cluster is easy: install dask and distributed, then run the dask scheduler and dask worker commands, as detailed in the Quick Start Guide in the Dask.distributed documentation.

$ pip install optuna dask distributed

$ dark scheduler
INFO - Scheduler at: tcp://192.168.1.8:8686
INFO - Dashboard at:                  :8687
…

$ dask worker tcp://192.168.1.8:8686
$ dask worker tcp://192.168.1.8:8686
$ dask worker tcp://192.168.1.8:8686

See the documentation for more information.

Brute-force Sampler

BruteForceSampler, a new sampler for brute-force search, tries all combinations of parameters. In contrast to GridSampler, it does not require passing the search space as an argument and works even with branches. This sampler constructs the search space with the define-by-run style, so it works by just adding sampler=optuna.samplers.BruteForceSampler().

import optuna

def objective(trial):
    c = trial.suggest_categorical("c", ["float", "int"])
    if c == "float":
        return trial.suggest_float("x", 1, 3, step=0.5)
    elif c == "int":
        a = trial.suggest_int("a", 1, 3)
        b = trial.suggest_int("b", a, 3)
        return a + b

study = optuna.create_study(sampler=optuna.samplers.BruteForceSampler())
study.optimize(objective)

Other Improvements

Bug Fix for TPE’s `constant_liar` Option

The constant_liar option of TPESampler is an option for the distributed optimization or batch optimization. It has been introduced in v2.8.0, but suffers from performance degradation in specific situations. In this release, we have detected the cause of the problem, and resolve it with fruitful performance verification. See #4073 for more details.

Make Scipy Dependency Optional

50% time of import optuna is consumed by SciPy-related modules. Also, it consumes 110MB of storage space, which is really problematic in environments with limited resources such as serverless computing.
We decided to implement scientific functions on our own to make the SciPy dependency optional. Thanks to contributors' effort on performance optimization, our implementation is as fast as the code with SciPy although ours is written in pure Python. See #4105 for more information.
Note that QMCSampler still depends on SciPy. If you use QMCSampler, please explicitly specify SciPy as your dependency.

The New UI for Optuna Dashboard

We are developing a new UI for Optuna Dashboard that is available as an opt-in feature from the beta release - simply launch the dashboard as usual and click the link to the new UI. Please try it out and share your thoughts with us.

$ pip install "optuna-dashboard>=0.9.0b2"

Feedback Survey: The New UI for Optuna Dashboard

Change Supported Python Versions

We have changed the supported Python versions. Specifically, Python 3.6 has been removed from the supported versions and Python 3.11 has been added. See #3021 and #3964 for more details.

Breaking Changes

Allow users to call study.optimize() in multiple threads (#4068)
Use all trials in TPESampler even when multivariate=True (#4079)
Drop Python 3.6 (#4150)
Remove RedisStorage (#4156)
Deprecate set_system_attr in Study and Trial (#4188)
Add a directions arg to storage.create_new_study (#4189)
Deprecate system_attrs in Study class (#4250)
Deprecate Trial.system_attrs property method (#4264)
Remove device argument of TorchDistributedTrial (#4266)

New Features

Add Dask integration (#2023, thanks @jrbourbeau!)
Add journal-style log storage (#3854)
Support CMA-ES with margin in CmaEsSampler (#4016)
Add journal redis storage (#4086)
Add device argument to BoTorchSampler (#4101)
Add the feature to JournalStorage of Redis backend to resume from a snapshot (#4102)
TorchDistributedTrial uses group as parameter instead of device (#4106, thanks @reyoung!)
Added user_attrs to print by Optuna studies in cli.py (#4129, thanks @gonzaload!)
Add BruteForceSampler (#4132, thanks @semiexp!)
Add __getstate__ and __setstate__ to RedisStorage (#4135, thanks @shu65!)
Make journal redis storage picklable (#4139, thanks @shu65!)
Support for qNoisyExpectedHypervolumeImprovement acquisition function from Botorch (Issue#4014) (#4186)
Show best trial number and value in progress bar (#4205)

Enhancements

Change the log message format for failed trials (#3857, thanks @erentknn!)
Move default logic of get_trial_id_from_study_id_trial_number() method to BaseStorage (#3910)
Fix the data migration script for v3 release (#4020)
Convert search_space values of GridSampler explicitly (#4062)
Add single exception catch to study optimize (#4098)
Remove scipy dependencies from TPESampler (#4105)
Add validation in enqueue_trial (#4126)
Speed up tests/samplers_tests/test_nsgaii.py::test_fast_non_dominated_sort_with_constraints (#4128, thanks @mist714!)
Add getstate and setstate to journal storage (#4130, thanks @shu65!)
Support None in slice plot (#4133, thanks @belldandyxtq!)
Add marker to matplotlib plot_intermediate_value (#4134, thanks @belldandyxtq!)
Add overloads for type narrowing in suggest_categorical (#4143, thanks @ConnorBaker!)
Cache study.directions to reduce the number of get_study_directions() calls (#4146)
Add an in-memory cache in Trial class (#4240)
Use CMAwM class even when there is no discrete params (#4289)
Refer OPTUNA_STORAGE environment variable in Optuna CLI (#4299, thanks @Hakuyume!)
Apply @overload to ChainerMNTrial and TorchDistributedTrial (Follow-up of #4143) (#4300)
Make OPTUNA_STORAGE environment variable experimental (#4316)

Bug Fixes

Fix infinite loop bug in TPESampler (#3953, thanks @gasin!)
Fix GridSampler (#3957)
Fix an import error of sqlalchemy.orm.declarative_base (#3967)
Skip to add intermediate_value_type and value_type columns if exists (#4015)
Fix duplicated sampling of SkoptSampler (#4023)
Avoid parse errors of datetime.isoformat strings (#4025)
Fix a concurrency bug of JournalStorage set_trial_state_values (#4033)
Specify object type to numpy array init to avoid unintended str cast (#4035)
Make TPESampler reproducible (#4056)
Fix bugs in constant_liar option (#4073)
Add a flush to JournalFileStorage.append_logs (#4076)
Add a lock to MLflowCallback (#4097)
Reject deprecated distributions in OptunaSearchCV (#4120)
Stop using hash function in _get_bracket_id in HyperbandPruner (#4131, thanks @zaburo-ch!)
Validation for the parameter enqueued in to_internal_repr of FloatDistribution and IntDistribution (#4137)
Fix PartialFixedSampler to handle None correctly (#4147, thanks @halucinor!)
Fix the bug of JournalFileStorage on Windows (#4151)
Fix CmaEs system attribution key (#4184)
Skip constraint check for running trial (#4275)
Fix constrained optimization with TPESampler's constant_liar (#4325)
Fix import of ProcessGroup from torch.distributed (#4344)

Installation

Replace thop with fvcore (#3906)
Use the latest stable scipy (#3959, thanks @gasin!)
Remove GPyTorch version constraint (#3986)
Make typing_extensions optional (#3990)
Add version constraint on importlib-metadata (#4036)
Add a version constraint of matplotlib (#4044)

Documentation

Update cli tutorial (#3902)
Replace thop with fvcore (#3906)
Slightly improve docs of FrozenTrial (#3943)
Refine docs in BaseStorage (#3948)
Remove "Edit on GitHub" button from readthedocs (#3952)
Mention restoring sampler in saving/resuming tutorial (#3992)
Use log_loss instead of deprecated log since sklearn 1.1 (#3993)
Fix script path in benchmarks/README.md (#4021)
Ignore ConvergenceWarning in the ask-and-tell tutorial (#4032)
Update docs to let users know the concurrency problem on SQLite3 (#4034)
Fix the time complexity of NSGAIISampler (#4045)
Fix sampler comparison table (#4082)
Add BruteForceSampler in the samplers' list (#4152)
Remove markup from NaN in FAQ (#4155)
Remove the document of the multi_objective module (#4167)
Fix a typo in QMCSampler (#4179)
Introduce Optuna Dashboard in tutorial docs (#4226)
Remove RedisStorage from docstring (#4232)
Add the BruteForceSampler example to the document (#4244)
Improve the document of BruteForceSampler (#4245)
Fix an inline markup in distributed tutorial (#4247)
Fix a typo in BruteForceSampler (#4267)
Update FAQ (#4269)
Fix a typo in XGBoostPruningCallback (#4270)
Fix CMAEvolutionStrategy link in integration.PyCmaSampler document (#4284, thanks @hrntsm!)
Resolve warnings by sphinx with nitpicky option and fix typos (#4287)
Fix typos (#4291)
Improve the document of JournalStorage (#4308, thanks @hrntsm!)
Fix typo (#4332, thanks @Jasha10!)
Fix docstring in optuna/integration/dask.py (#4333)
Mention suggest_float in BruteForceSampler (#4334)
Remove verbose_eval argument from lightgbm callback in tutorial pages (#4335)
Use Sphinx 5 until sphinx_rtd_theme supports Sphinx 6 (#4341)

Examples

Add Dask example (optuna/optuna-examples#46, thanks @jrbourbeau!)
Hotfix for botorch example (optuna/optuna-examples#134)
Replace thop with fvcore (optuna/optuna-examples#136)
Add Optuna-distributed to external projects (optuna/optuna-examples#137)
Remove the version constraint of GPyTorch (optuna/optuna-examples#138)
Fix a file path in CONTRIBUTING.md (optuna/optuna-examples#139)
Install scikit-learn instead of sklearn (optuna/optuna-examples#141)
Add constraint on tensorflow to <2.11.0 (optuna/optuna-examples#146)
Specify botorch version (optuna/optuna-examples#151)
Add an example of Optuna Dashboard (optuna/optuna-examples#152)
Pin numpy version to 1.23.x for mxnet examples (optuna/optuna-examples#154)
Use tensorflow 2.11 syntax to fix CI error (optuna/optuna-examples#156)
Add Python 3.11 to version-matrix except for integration (optuna/optuna-examples#160)
Use latest stable version of actions (optuna/optuna-examples#161)
Use Monitor to resolve stable_baselines3's warning (optuna/optuna-examples#162)
Add Python 3.11 to a part of CI jobs (optuna/optuna-examples#163)

Tests

Suppress warnings in tests/test_distributions.py (#3912)
Suppress warnings and minor code fixes in tests/trial_tests (#3914)
Reduce warning messages by tests/study_tests/ (#3915)
Remove dynamic search space based objective from a parallel job test (#3916)
Remove all warning messages from tests/integration_tests/test_sklearn.py (#3922)
Remove out-of-range related warning messages from MLflowCallback and WeightsAndBiasesCallback (#3923)
Ignore RuntimeWarning when nanmin and nanmax take an array only containing nan values from pruners_tests (#3924)
Remove warning messages from test files for pytorch_distributed and chainermn modules (#3927)
Remove warning messages from tests/integration_tests/test_lightgbm.py (#3944)
Resolve warnings in tests/visualization_tests/test_contour.py (#3954)
Reduced warning messages from tests/visualization_tests/test_slice.py (#3970, thanks @jmsykes83!)
Remove warning from a few visualization tests (#3989)
Deselect integration tests in Tests CI (#4013)
Remove warnings from tests/visualization_tests/test_optimization_history.py (#4024)
Unset PYTHONHASHSEED for the hash-depedenet test (#4031)
Call study.tell from another process (#4039, thanks @Abelarm!)
Improve test for heartbeat: Add test for the case that trial state should be kept running (#4055)
Remove warnings in the test of Pareto front (#4072)
Remove matplotlib's get_cmap warning from tests/visualization_tests/test_param_importances.py (#4095)
Reduce tests' n_trials for CI time reduction (#4117)
Skip test_pop_waiting_trial_thread_safe on RedisStorage (#4119)
Simplify the test of BruteForceSampler for infinite search space (#4153)
Add sep-CMA-ES in parametrize_sampler (#4154)
Fix a broken test for dask.distributed integration (#4170)
Add DaskStorage to existing storage tests (#4176, thanks @jrbourbeau!)
Fix a test error in test_catboost.py (#4190)
Remove test/integration_tests/test_sampler.py (#4204)
Fix mypy error abut PyTorch Lightning in Checks (integration) (#4279)
Remove OPTUNA_STORAGE environment variable to check missing storage errors (#4306)
Use Trial not FrozenTrial in a test of WeightsAndBiasesCallback (#4309)
Activate a test case for _set_alembic_revision (#4319)
Add test to check error_score is stored (#4337)

Code Fixes

Refactor _tell.py (#3841)
Make log message user-friendly when objective returns a sequence of unsupported values (#3868)
Gather mask of None parameter in TPESampler (#3886)
Migrate CLI from cliff to argparse (#4100)
Enable mypy --no-implicit-reexport option (#4110)
Remove unused function: find_any_distribution (#4127)
Remove object inheritance from base classes (#4161)
Use mlflow 2.0.1 syntax (#4173)
Simplify implementation of _preprocess_argv in CLI (#4187)
Move _solve_hssp to _hypervolume/utils.py (#4227, thanks @jpbianchi!)
Add tests for CmaEsSampler (#4233)
Remove an obsoleted logic from CmaEsSampler (#4239)
Avoid to decode log string in JournalRedisStorage (#4246)
Fix a typo in TorchDistributedTrial (#4271)
Fix mypy error abut Chainer in Checks (integration) (#4276)
Fix mypy error abut BoTorch in Checks (integration) (#4278)
Fix mypy error abut dask.py in Checks (integration) (#4280)
Avoid to use features that will be removed in SQLAlchemy v2.0 (#4304)

Continuous Integration

Hotfix botorch module by adding the version constraint of gpytorch (#3950)
Drop Python 3.6 from integration CIs (#3983)
Use PyTorch 1.11 for consistency and fix a typo (#3987)
Support Python 3.11 (#4018)
Remove # type: ignore for mypy 0.981 (#4019)
Fix metric inconsistency between bayesmark plots and report (#4077)
Pin Ubuntu version to 20.04 in Tests and Tests (Storage with server) (#4118)
Add workflow to test Optuna with lower versions of constraints (#4125)
Mark some tests slow and ignore in pull request trigger (#4138, thanks @mist714!)
Allow display names to be changed in benchmark scripts (Issue #4017) (#4145)
Disable scheduled workflow runs in forks (#4159)
Remove the CircleCI job document (#4160)
Stop running reproducibility tests on CI for PR (#4162)
Stop running reproducibility tests for coverage (#4163)
Add workflow_dispatch trigger to the integration tests (#4166)
Fix CI errors when using mlflow==2.0.1 (#4171)
Add fakeredis in benchmark dependencies (#4177)
Fix asv speed benchmark (#4185)
Skip tests with minimum version for Python 3.10 and 3.11 (#4199)
Split normal tests and tests with minimum versions (#4200)
Update action/checkout@v2 -> v3 (#4206)
Update actions/cache@v2 -> v3 (#4207)
Update actions/stale@v5 -> v6 (#4208)
Pin botorch to avoid CI failure (#4228)
Add the pytest dependency for asv (#4243)
Fix mypy error about pytorch_distributed.py in Checks (integration) (#4281)
Run test_pytorch_distributed.py again (#4301)
Remove all CircleCI config (#4315)
Update minimum version of cmaes (#4321)
Add Python 3.11 to integration CI (#4327)

Other

Bump up version number to 3.1.0.dev (#3934)
Remove the news section on README (#3940)
Add issue template for code fix (#3968)
Close stale issues immediately after labeling stale (#4071)
Remove tox.ini (#4078)
Replace gitter with GitHub Discussions (#4083)
Deprecate description-checked label (#4090)
Make days-before-issue-stale 300 days (#4091)
Unnecessary space removed (#4109, thanks @gonzaload!)
Add note not to share pickle files in bug reports (#4212)
Update the description of optuna-dashboard on README (#4217)
Remove optuna.TYPE_CHECKING (#4238)
Bump up version to v3.1.0-b0 (#4262)
Remove the list of examples from examples/README (#4283)
Exclude benchmark directories from the sdist package (#4318)
Bump up version number to 3.1.0 (#4346)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

@Abelarm, @Alnusjaponica, @ConnorBaker, @Hakuyume, @HideakiImamura, @Jasha10, @amylase, @belldandyxtq, @c-bata, @contramundum53, @cross32768, @erentknn, @eukaryo, @g-votte, @gasin, @gen740, @gonzaload, @halucinor, @himkt, @hrntsm, @hvy, @jmsykes83, @jpbianchi, @jrbourbeau, @keisuke-umezawa, @knshnb, @mist714, @ncclementi, @not522, @nzw0301, @rene-rex, @reyoung, @semiexp, @shu65, @sile, @toshihikoyanase, @wattlebirdaz, @xadrianzetx, @zaburo-ch

optuna 3.1.0 v3.1.0 on Python PyPI

Highlights

New Features

CMA-ES with Margin

Distributed Optimization via NFS

A Brand-New Redis Storage

Dask.distributed Integration

Brute-force Sampler

Other Improvements

Bug Fix for TPE’s constant_liar Option

Make Scipy Dependency Optional

The New UI for Optuna Dashboard

Change Supported Python Versions

Breaking Changes

New Features

Enhancements

Bug Fixes

Installation

Documentation

Examples

Tests

Code Fixes

Continuous Integration

Other

Thanks to All the Contributors!

optuna 3.1.0
v3.1.0

on Python PyPI

Bug Fix for TPE’s `constant_liar` Option