This is the release note of v3.3.0.

Highlights

CMA-ES with Learning Rate Adaptation

A new variant of CMA-ES has been added. By setting the lr_adapt argument to True in CmaEsSampler, you can utilize it. For multimodal and/or noisy problems, adapting the learning rate can help avoid getting trapped in local optima. For more details, please refer to #4817. We want to thank @nomuramasahir0, one of the authors of LRA-CMA-ES, for his great work and the development of cmaes library.

Hypervolume History Plot for Multiobjective Optimization

In multiobjective optimization, the history of hypervolume is commonly used as an indicator of performance. Optuna now supports this feature in the visualization module. Thanks to @y0z for your great work!

Constrained Optimization Support for Visualization Functions

Plotly	matplotlib

Some samplers support constrained optimization, however, many other features cannot handle it. We are continuously enhancing support for constraints. In this release, plot_optimization_history starts to consider constraint violations. Thanks to @hrntsm for your great work!

import optuna

def objective(trial):
    x = trial.suggest_float("x", -15, 30)
    y = trial.suggest_float("y", -15, 30)
    v0 = 4 * x**2 + 4 * y**2
    trial.set_user_attr("constraint", [1000 - v0])
    return v0

def constraints_func(trial):
    return trial.user_attrs["constraint"]

sampler = optuna.samplers.TPESampler(constraints_func=constraints_func)
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=100)
fig = optuna.visualization.plot_optimization_history(study)
fig.show()

Streamlit Integration for Human-in-the-loop Optimization

Optuna Dashboard v0.11.0 provides the tight integration with Streamlit framework. By using this feature, you can create your own application for human-in-the-loop optimization. Please check out the documentation and the example for details.

Breaking Changes

• Move mxnet (optuna/optuna-integration#31)
• Remove mxnet (#4790)
• Remove ordered_dict argument from IntersectionSearchSpace (#4846)

New Features

• Add logei_candidate_func and make it default when available (#4667)
• Support JournalFileStorage and JournalRedisStorage on CLI (#4696)
• Implement hypervolume history plot for matplotlib backend (#4748, thanks @y0z!)
• Add cv_results_ to OptunaSearchCV (#4751, thanks @jckkvs!)
• Add optuna.integration.botorch.qnei_candidates_func (#4753, thanks @kstoneriv3!)
• Add hypervolume history plot for plotly backend (#4757, thanks @y0z!)
• Add FileSystemArtifactStore (#4763)
• Sort params on fetch (#4775)
• Add constraints support to _optimization_history_plot (#4793, thanks @hrntsm!)
• Bump up LightGBM version to v4.0.0 (#4810)
• Add constraints support to matplotlib._optimization_history_plot (#4816, thanks @hrntsm!)
• Introduce CMA-ES with Learning Rate Adaptation (#4817)
• Add upload_artifact api (#4823)
• Add before_trial (#4825)
• Add Boto3ArtifactStore (#4840)
• Display best objective value in contour plot for a given param pair, not the value from the most recent trial (#4848)

Enhancements

• Speed up logpdf in _truncnorm.py (#4712)
• Speed up erf (#4713)
• Speed up get_all_trials in InMemoryStorage (#4716)
• Add a warning for a progress bar not being displayed #4679 (#4728, thanks @rishabsinghh!)
• Make BruteForceSampler consider failed trials (#4747)
• Use shallow copy in _get_latest_trial (#4774)
• Speed up plot_hypervolume_history (#4776)

Bug Fixes

• Solve issue #4557 - error_score (#4642, thanks @jckkvs!)
• Fix BruteForceSampler for pruned trials (#4720)
• Fix plot_slice bug when some of the choices are numeric (#4724)
• Make LightGBMTuner reproducible (#4795)

Installation

• Bump up python version (optuna/optuna-integration#34)

Documentation

• Remove jquery-extension (#4691)
• Add FAQ on combinatorial search space (#4723)
• Fix docs (#4732)
• Add plot_rank and plot_timeline plots to visualization tutorial (#4735)
• Fix typos found in integration/sklearn.py (#4745)
• Remove study.n_objectives from document (#4796)
• Add lower version constraint for sphinx_rtd_theme (#4853)
• Artifact docs (#4855)

Examples

• Run DaskML example with Python 3.11 (optuna/optuna-examples#188)
• Show more information in terminator examples (optuna/optuna-examples#192)
• Drop support for Python 3.7 on Haiku (optuna/optuna-examples#198)
• Add LICENSE file (optuna/optuna-examples#200)

Tests

• Remove unnecessary pytestmark (optuna/optuna-integration#29)
• Add GridSampler test for failed trials (#4721)
• Follow up PR #4642 by adding a unit test to confirm OptunaSearchCV behavior (#4758)
• Fix test_log_gass_mass with SciPy 1.11.0 (#4766)
• Fix Pytorch lightning unit test (#4780)
• Remove skopt (#4792)
• Rename test directory (#4839)

Code Fixes

• Simplify the type annotations in benchmarks (#4703, thanks @caprest!)
• Unify sampling implementation in TPESampler (#4717)
• Get values after _get_observation_pairs (#4742)
• Remove unnecessary period (#4746)
• Handle deprecated argument early_stopping_rounds (#4752)
• Separate dominate function from _fast_non_dominated_sort() (#4759)
• Separate after_trial strategy (#4760)
• Remove unused attributes in TPESampler (#4769)
• Remove pkg_resources (#4770)
• Use trials as argument of _calculate_weights_below_for_multi_objective (#4773)
• Fix type annotation (#4797, thanks @taniokay!)
• Follow up separation of after trial strategy (#4803)
• Loose coupling nsgaii child generation (#4806)
• Remove _study_id parameter from Trial class (#4811, thanks @adjeiv!)
• Loose coupling nsgaii elite population selection (#4821)
• Fix checks integration (#4826)
• Remove OrderedDict (#4838, thanks @taniokay!)
• Fix typo (#4842, thanks @wouterzwerink!)
• Followup child generation strategy (#4856)
• Remove samplers._search_space.IntersectionSearchSpace (#4857)
• Add experimental decorators to artifacts functionalities (#4858)

Continuous Integration

• Output dependency tree (optuna/optuna-integration#9)
• Use OIDC (optuna/optuna-integration#33)
• Drop Python 3.7 support (optuna/optuna-integration#35)
• Enhance speed benchmark for storages (#4778)
• Drop Python 3.7 on tests-integration (#4784)
• Remove unused type:ignores (#4787)
• Restrict numpy version < 1.24 (#4788)
• Upgrade redis version (#4805)
• Add version constraints on LightGBM (#4807)
• Follow-up #4807 : Fix windows-tests and mac-tests (#4809)
• Support 3.11 integration (#4820)
• Support flake8 6.1.0 (#4847)

Other

• Bump up version number to 3.3.0dev (optuna/optuna-integration#27)
• Bump up version number to 3.3.0 (optuna/optuna-integration#36)
• Bump up version number to 3.3.0dev (#4710)
• Bump the version up to v3.3.0 (#4860)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

@Alnusjaponica, @HideakiImamura, @adjeiv, @c-bata, @caprest, @contramundum53, @cross32768, @eukaryo, @gen740, @hrntsm, @jckkvs, @knshnb, @kstoneriv3, @nomuramasahir0, @not522, @nzw0301, @rishabsinghh, @taniokay, @toshihikoyanase, @wouterzwerink, @xadrianzetx, @y0z