This is the release note of v4.9.0.

Highlights

Enhance Multi-Objective Constrained Parallel Optimization in GPSampler

@sawa3030 introduces parallelization enhancements to GPSampler, leveraging the Kriging Believer approach for constrained and multi-objective optimization (#6481). This improvement allows for more efficient exploration when multiple trials are running concurrently.

The GP surrogate is updated by assigning temporary objective function values to the running trials.

For more technical details and benchmarks, please check out our blog post: Improving Optuna’s GPSampler Parallelization by Considering Running Trials.

Deprecate Several Features

The following features are deprecated in v4.9.0 and scheduled for removal in v6.0.0.

optuna

Several arguments in TPESampler (#6635)
- prior_weight, consider_magic_clip, consider_endpoints, gamma, weights, hyperopt_parameters: These internal parameters are being deprecated to simplify the interface, as the default settings are optimal for most use cases.
- warn_independent_sampling: Deprecated because TPESampler now robustly supports both independent and joint sampling, making this warning obsolete.
- categorical_distance_func: This advanced feature will be migrated to OptunaHub in the future.
x0 and sigma0 options in CmaEsSampler (#6624)
- These options have been deprecated because they require a deep understanding of CmaEsSampler's internals to be configured effectively.
optuna.terminator module (#6668)
- This feature will be migrated to OptunaHub in the future.
RetryFailedTrialCallback (#6670)
- This class has been renamed to RetryHeartbeatStaleTrialCallback to better reflect its behavior and avoid confusion with general trial retries (#6085).
optuna.integration module
- The optuna.integration module currently acts as a shortcut to the external optuna_integration package for backward compatibility. Please import directly from the optuna_integration package going forward.

optuna-integration

PyCmaSampler: Please use Optuna's native CmaEsSampler instead.
CometCallback: This feature will be migrated to OptunaHub in the future.
MLflowCallback: This feature will be migrated to OptunaHub in the future.
TensorBoardCallback: This feature will be migrated to OptunaHub in the future.
TrackioCallback: This feature will be migrated to OptunaHub in the future.
WeightsAndBiasesCallback: This class has already been migrated to OptunaHub.

Breaking Changes

Deprecate PyCmaSampler (optuna/optuna-integration#276)
Add deprecation message for CometCallback (optuna/optuna-integration#280)
Add deprecation message for MLflowCallback (optuna/optuna-integration#281)
Add deprecation message for TensorBoardCallback (optuna/optuna-integration#282)
Add deprecation message for WeightsAndBiasesCallback (optuna/optuna-integration#283)
Add deprecation message for TrackioCallback (optuna/optuna-integration#284)
Fix importance computation for single-distributed params (#6500)
Make QMCSampler stateless (#6616)
Deprecate x0 and sigma0 options in CmaEsSampler (#6624)
Deprecate a set of TPESampler arguments (#6635)
Deprecate legacy imports from optuna.integration (#6667)
Deprecate optuna.terminator module (#6668)
Rename RetryFailedTrialCallback to RetryHeartbeatStaleTrialCallback (#6670)

Enhancements

Enhance Multi-Objective Constrained Parallel Optimization in GPSampler (#6481)
Remove prior_mu from compute_sigmas (#6574)
Fix sampling bias in BruteForceSampler (#6627, thanks @Rishabh-git10!)
Refactor BruteForceSampler (#6645)
Speed up BruteForceSampler by using any instead of count (#6647)
Fix the return type of BruteForceSampler (#6648)
Split BruteForceSampler refactoring [2/3] (#6656)
fix: collect all infeasible values in error message instead of early return (#6661, thanks @AshutoshDevpura!)

Bug Fixes

Fix categorical None handling in slice plots (#6621)
fix: handle JSONDecodeError in TPESampler._get_params to avoid race condition (#6628, thanks @AshutoshDevpura!)
Split BruteForceSampler refactoring [2/3] (#6656)

Documentation

Update LightGBM links (#6511, thanks @jameslamb!)
Improve doc for best_trial/best_trials in constrained optimization (#6522)
Update CmaEsSampler docstring regarding categorical support (#6625)
Fix RST note directive typo in QMCSampler docstring (#6631, thanks @RudrenduPaul!)
Enhance Generative Engine Optimization (GEO) of many objective and constraint handling (#6639)
Add NSGA-III to FAQ about constraint (#6641)
Update BruteForceSampler and GridSampler information in docs (#6651)
Enhance FAQ about killing trials (#6653)

Examples

Explicitly install tensorboard in the tensorboard example CI (optuna/optuna-examples#356)
Remove examples which uses deprecated integration modules (optuna/optuna-examples#357)

Code Fixes

Use broadcast_object_list instead of a custom method (optuna/optuna-integration#274)
Correct return type annotations for Axes in visualization functions (#6504, thanks @kvr06-ai!)
Fix type checking in trial folder (#6510, thanks @sateeshkumarb!)
Use TYPE_CHECKING in importance/_mean_decrease_impurity.py (#6514, thanks @saivedant169!)
Use TYPE_CHECKING in importance/_ped_anova/evaluator.py (#6515, thanks @saivedant169!)
Use TYPE_CHECKING in samplers/_cmaes.py (#6516, thanks @saivedant169!)
Use TYPE_CHECKING in samplers/_base.py (#6517, thanks @saivedant169!)
Use TYPE_CHECKING in pruners/_wilcoxon.py (#6518, thanks @saivedant169!)
Use TYPE_CHECKING in pruners/_percentile.py (#6520, thanks @saivedant169!)
Use TYPE_CHECKING in nsgaii/_constraints_evaluation.py (#6521, thanks @saivedant169!)
Use TYPE_CHECKING in nsgaii/_elite_population_selection_strategy.py (#6523, thanks @saivedant169!)
Use TYPE_CHECKING in nsgaii/_after_trial_strategy.py (#6524, thanks @saivedant169!)
Use TYPE_CHECKING in nsgaii/_child_generation_strategy.py (#6526, thanks @saivedant169!)
Move type-only imports in optuna.samplers._partial_fixed to TYPE_CHECKING (#6527, thanks @t7r0n!)
Fix TC006 cast annotation in optuna._gp.acqf (#6528, thanks @t7r0n!)
Fix TC006 cast annotation in optuna.trial._frozen (#6529, thanks @t7r0n!)
Fix TC006 cast annotation in tests.test_distributions (#6530, thanks @t7r0n!)
Fix TC006 cast annotation in optuna.study._multi_objective (#6531, thanks @t7r0n!)
Fix TC006 cast annotation in optuna.terminator.improvement.emmr (#6532, thanks @t7r0n!)
Fix TC006 cast annotation in optuna.terminator.erroreval (#6533, thanks @t7r0n!)
Fix TC006 cast annotation in optuna.study.study (#6534, thanks @t7r0n!)
Move type-only Callable import in optuna.testing.threading behind TYPE_CHECKING (#6535, thanks @t7r0n!)
Move type-only TracebackType import in optuna.testing.tempfile_pool behind TYPE_CHECKING (#6536, thanks @t7r0n!)
Move type-only stdlib imports in optuna.testing.storages behind TYPE_CHECKING (#6537, thanks @t7r0n!)
test: move type-only imports in optuna.testing.trials behind TYPE_CHECKING (#6538, thanks @t7r0n!)
test: move type-only imports in optuna.testing.pytest_samplers behind TYPE_CHECKING (#6539, thanks @t7r0n!)
test: move type-only imports in optuna.testing.pytest_storages behind TYPE_CHECKING (#6540, thanks @t7r0n!)
test: move type-only imports in optuna.testing.samplers behind TYPE_CHECKING (#6541, thanks @t7r0n!)
Quote cast type expression in tests/test_distributions.py (#6543, thanks @t7r0n!)
Move type-only datetime import in tests/trial_tests/test_trial.py (#6544, thanks @t7r0n!)
Move type-only imports to TYPE_CHECKING in samplers/_tpe/sampler.py (#6545, thanks @yasumorishima!)
Move type-only imports to TYPE_CHECKING in nsgaii/_crossover.py (#6546, thanks @yasumorishima!)
Remove dead code: unused functions, methods, and variables (#6547, thanks @duriantaco!)
Fix typing import issues in scott_parzen_estimator (#6548, thanks @rpathade!)
fix(types): use TYPE_CHECKING for imports in samplers/_cmaes.py (#6550, thanks @Aliipou!)
fix(types): use TYPE_CHECKING for BaseDistribution import in _transform.py (#6552, thanks @Aliipou!)
refactor: move BaseDistribution import to TYPE_CHECKING in search_space/group_decomposed.py (#6553, thanks @Aliipou!)
Use TYPE_CHECKING for imports in importance/_ped_anova/scott_parzen_estimator.py (#6554, thanks @Aliipou!)
fix(types): use TYPE_CHECKING for imports in samplers/_brute_force.py (#6555, thanks @Aliipou!)
Move type-only imports to TYPE_CHECKING in samplers/_nsgaiii/_elite_population_selection_strategy.py (#6557, thanks @Aliipou!)
Move type-only imports to TYPE_CHECKING in samplers/_nsgaiii/_sampler.py (#6558, thanks @Aliipou!)
Move Study import to TYPE_CHECKING in _timeline.py (#6559, thanks @rpathade!)
Use TYPE_CHECKING in optuna/samplers/_ga/_base.py (#6560, thanks @nightcityblade!)
Use TYPE_CHECKING in nsgaii/_crossovers/_base.py (#6561, thanks @saivedant169!)
Use TYPE_CHECKING in nsgaii/_crossovers/_blxalpha.py (#6562, thanks @saivedant169!)
Use TYPE_CHECKING in nsgaii/_crossovers/_uniform.py (#6563, thanks @saivedant169!)
Move imports to TYPE_CHECKING in tests/test_multi_objective (#6564, thanks @acabellom!)
Use TYPE_CHECKING in nsgaii/_sampler.py (#6565, thanks @saivedant169!)
Use TYPE_CHECKING in optuna.samplers._qmc (#6566, thanks @hnshah!)
Use TYPE_CHECKING in visualization/_rank.py (#6567, thanks @nightcityblade!)
Use TYPE_CHECKING in visualization/_parallel_coordinate.py (#6568, thanks @nightcityblade!)
Use TYPE_CHECKING in visualization/_slice.py (#6569, thanks @nightcityblade!)
Use TYPE_CHECKING in visualization/_intermediate_values.py (#6570, thanks @nightcityblade!)
Use TYPE_CHECKING in visualization/_hypervolume_history.py (#6571, thanks @nightcityblade!)
Use TYPE_CHECKING in visualization/_contour.py (#6572, thanks @saivedant169!)
Use TYPE_CHECKING in visualization/_edf.py (#6575, thanks @saivedant169!)
Use TYPE_CHECKING in visualization/_optimization_history.py (#6576, thanks @saivedant169!)
Use TYPE_CHECKING in visualization/_hypervolume_history.py (#6577, thanks @saivedant169!)
Use TYPE_CHECKING in visualization/_pareto_front.py (#6578, thanks @saivedant169!)
Use TYPE_CHECKING in visualization/_terminator_improvement.py (#6579, thanks @saivedant169!)
Use TYPE_CHECKING in visualization/_rank.py (#6580, thanks @saivedant169!)
Use TYPE_CHECKING in visualization/_slice.py (#6581, thanks @saivedant169!)
Fixed spelling/typo in error message (#6583, thanks @craigulliott!)
Use TYPE_CHECKING in visualization/_parallel_coordinate.py (#6584, thanks @saivedant169!)
Use TYPE_CHECKING in optuna/visualization/matplotlib/_utils.py (#6585, thanks @nightcityblade!)
Use TYPE_CHECKING in optuna/visualization/matplotlib/_edf.py (#6586, thanks @nightcityblade!)
Use TYPE_CHECKING in optuna/visualization/matplotlib/_contour.py (#6587, thanks @nightcityblade!)
Use TYPE_CHECKING in optuna/visualization/matplotlib/_pareto_front.py (#6588, thanks @nightcityblade!)
Use TYPE_CHECKING in optuna/visualization/matplotlib/_param_importances.py (#6589, thanks @nightcityblade!)
Fix missing blank line after TYPE_CHECKING in visualization/_intermediate_values.py (#6591, thanks @saivedant169!)
Use TYPE_CHECKING in visualization/matplotlib/_optimization_history.py (#6592, thanks @saivedant169!)
Use TYPE_CHECKING in visualization/matplotlib/_parallel_coordinate.py (#6593, thanks @saivedant169!)
Use TYPE_CHECKING in visualization/matplotlib/_rank.py (#6594, thanks @saivedant169!)
Address integration type checking in optuna.integration.__init__.py (#6597)
Remove unused type ignore comment (#6607)
fix: move FrozenTrial import into TYPE_CHECKING block in storages/_callbacks.py (#6609, thanks @satishkc7!)
fix: move type-only imports to TYPE_CHECKING in StudySummary (#6610, thanks @Zelys-DFKH!)
Enhance the error message of QMCSampler (#6614)
Enhance the warning message for TPESampler with multivariate=True (#6618)
Simplify some GP processing and update staled comments (#6642)
Enhance LSP experience by adding explicit references for optuna modules (#6671)
Remove unused variable in optuna.storages._rdb.storage.py (#6672)

Continuous Integration

Exclude Python 3.9 from fastaiv2 CI (optuna/optuna-integration#273)
Pinned every action for pypi-publish to a commit SHA (optuna/optuna-integration#285)
Restore TestPyPI repository URL for scheduled publishes (optuna/optuna-integration#286)
Pinned every action for pypi-publish to a commit SHA (#6675)
Restore TestPyPI repository URL for scheduled publishes (#6677)

Other

Bump the version up to v4.9.0.dev (optuna/optuna-integration#268)
Bump up version (optuna/optuna-integration#288)
Bump the version up to v4.9.0.dev (#6508)
Update news (#6513)
Add a PEP810 note to lazy import module (#6636)
Add some notes to GPSampler (#6637)
Remove unused intersphinx mappings (#6643)
Clarify policy on low-quality LLM-generated PRs (#6665)
Migrate to uv from pip (#6669)
Remove deprecated integration modules from README (#6673)
Modify the TODO note of GPSampler (#6680)
Remove the links to Optuna v5 roadmap and feedback survey from README (#6683)
Bump up to version number v4.9.0 (#6684)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

@Aliipou, @Alnusjaponica, @AshutoshDevpura, @Rishabh-git10, @RudrenduPaul, @Zelys-DFKH, @acabellom, @c-bata, @craigulliott, @duriantaco, @gen740, @hnshah, @jameslamb, @kAIto47802, @kvr06-ai, @nabenabe0928, @nightcityblade, @not522, @rpathade, @saivedant169, @sateeshkumarb, @satishkc7, @sawa3030, @t7r0n, @y0z, @yasumorishima

optuna 4.9.0 v4.9.0 on Python PyPI