This is the release note of v4.2.0. In conjunction with the Optuna release, OptunaHub 0.2.0 is released. Please refer to the release note of OptunaHub 0.2.0 for more details.

Highlights of this release include:

🚀gRPC Storage Proxy for Scalable Hyperparameter Optimization
🤖 SMAC3: Support for New State-of-the-art Optimization Algorithm by AutoML.org (@automl)
📁 OptunaHub Now Supports Benchmark Functions
🧑‍💻 Gaussian Process-Based Bayesian Optimization with Inequality Constraints
🧑‍💻 c-TPE: Support Constrained TPESampler

Highlights

gRPC Storage Proxy for Scalable Hyperparameter Optimization

The gRPC storage proxy is a feature designed to support large-scale distributed optimization. As shown in the diagram below, gRPC storage proxy sits between the optimization workers and the database server, proxying the calls of Optuna’s storage APIs.

In large-scale distributed optimization settings where hundreds to thousands of workers are operating, placing a gRPC storage proxy for every few tens can significantly reduce the load on the RDB server which would otherwise be a single point of failure. The gRPC storage proxy enables sharing the cache about Optuna studies and trials, which can further mitigate load. Please refer to the official documentation for further details on how to utilize gRPC storage proxy.

SMAC3: Random Forest-Based Bayesian Optimization Developed by AutoML.org

SMAC3 is a hyperparameter optimization framework developed by AutoML.org, one of the most influential AutoML research groups. The Optuna-compatible SMAC3 sampler is now available thanks to the contribution to OptunaHub by Difan Deng (@dengdifan), one of the core members of AutoML.org. We can now use the method widely used in AutoML research and real-world applications from Optuna.

# pip install optunahub smac
import optuna
import optunahub
from optuna.distributions import FloatDistribution

def objective(trial: optuna.Trial) -> float:
    x = trial.suggest_float("x", -10, 10)
    y = trial.suggest_float("y", -10, 10)
    return x**2 + y**2

smac_mod = optunahub.load_module("samplers/smac_sampler")
n_trials = 100
sampler = smac_mod.SMACSampler(
    {"x": FloatDistribution(-10, 10), "y": FloatDistribution(-10, 10)},
    n_trials=n_trials,
)
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=n_trials)

Please refer to https://hub.optuna.org/samplers/smac_sampler/ for more details.

OptunaHub Now Supports Benchmark Functions

Benchmarking the performance of optimization algorithms is an essential process indispensable to the research and development of algorithms. The newly added OptunaHub Benchmarks in the latest version v0.2.0 of optunahub is a new feature for Optuna users to conduct benchmarks conveniently.

# pip install optunahub>=4.2.0 scipy torch
import optuna
import optunahub

bbob_mod = optunahub.load_module("benchmarks/bbob")
smac_mod = optunahub.load_module("samplers/smac_sampler")
sphere2d = bbob_mod.Problem(function_id=1, dimension=2)

n_trials = 100
studies = []
for study_name, sampler in [
    ("random", optuna.samplers.RandomSampler(seed=1)),
    ("tpe", optuna.samplers.TPESampler(seed=1)),
    ("cmaes", optuna.samplers.CmaEsSampler(seed=1)),
    ("smac", smac_mod.SMACSampler(sphere2d.search_space, n_trials, seed=1)),
]:
    study = optuna.create_study(directions=sphere2d.directions,
        sampler=sampler, study_name=study_name)
    study.optimize(sphere2d, n_trials=n_trials)
    studies.append(study)

optuna.visualization.plot_optimization_history(studies).show()

In the above sample code, we compare and display the performance of the four kinds of samplers using a two-dimensional Sphere function, which is part of a group of benchmark functions widely used in the black-box optimization research community known as Blackbox Optimization Benchmarking (BBOB).

Gaussian Process-Based Bayesian Optimization with Inequality Constraints

We worked on its extension and adapted GPSampler to constrained optimization in Optuna v4.2.0 since Gaussian process-based Bayesian optimization is a very popular method in various research fields such as aircraft engineering and materials science. We show the basic usage below.

# pip install optuna>=4.2.0 scipy torch
import numpy as np
import optuna

def objective(trial: optuna.Trial) -> float:
    x = trial.suggest_float("x", 0.0, 2 * np.pi)
    y = trial.suggest_float("y", 0.0, 2 * np.pi)
    c = float(np.sin(x) * np.sin(y) + 0.95)
    trial.set_user_attr("c", c)
    return float(np.sin(x) + y)

def constraints(trial: optuna.trial.FrozenTrial) -> tuple[float]:
    return (trial.user_attrs["c"],)

sampler = optuna.samplers.GPSampler(constraints_func=constraints)
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=50)

Please try out GPSampler for constrained optimization especially when only a small number of trials are available!

c-TPE: Support Constrained TPESampler

Although Optuna has supported constrained optimization for TPESampler, which is the default Optuna sampler, since v3.0.0, its algorithm design and performance comparison have not been verified academically. OptunaHub now supports c-TPE, which is another constrained optimization method for TPESampler. Importantly, the algorithm design and its performance comparison are publicly reviewed to be accepted to IJCAI, a top-tier AI international conference. Please refer to https://hub.optuna.org/samplers/ctpe/ for details.

New Features

Enable GPSampler to support constraint functions (#5715)
Update output format options in CLI to include the value choice (#5822, thanks @iamarunbrahma!)
Add gRPC storage proxy server and client (#5852)

Enhancements

Introduce client-side cache in GrpcStorageProxy (#5872)

Bug Fixes

Fix CI (optuna/optuna-integration#185)
Fix ticks in Matplotlib contour plot (#5778, thanks @sulan!)
Adding check in cli.py to handle an empty database (#5828, thanks @willdavidson05!)
Avoid the input validation fail in Wilcoxon signed ranked test for Scipy 1.15 (#5912)
Fix the default sampler of load_study function (#5924)

Documentation

Update OptunaHub example in README (#5763)
Update distributions.rst to list deprecated distribution classes (#5764)
Remove deprecation comment for step in IntLogUniformDistribution (#5767)
Update requirements for OptunaHub in README (#5768)
Use inline code rather than italic for step (#5769)
Add notes to ask_and_tell tutorial - batch optimization recommendations (#5817, thanks @SimonPop!)
Fix the explanation of returned values of get_trial_params (#5820)
Introduce sphinx-notfound-page for better 404 page (#5898)
Follow-up #5872: Update the docstring of run_grpc_proxy_server (#5914)
Modify doc-string of gRPC-related modules (#5916)

Examples

Adapt docker recipes to Python 3.11 (optuna/optuna-examples#292)
Add version constraint for wandb (optuna/optuna-examples#293)

Tests

Add unit tests for retry_history method in RetryFailedTrialCallback (#5865, thanks @iamarunbrahma!)
Add tests for value format in CLI (#5866)
Import grpc lazy to fix the CI (#5878)

Code Fixes

Fix annotations for distributions.py (#5755, thanks @KannanShilen!)
Simplify type annotations for tests/visualization_tests/test_pareto_front.py (#5756, thanks @boringbyte!)
Fix type annotations for test_hypervolume_history.py (#5760, thanks @boringbyte!)
Simplify type annotations for tests/test_cli.py (#5765, thanks @boringbyte!)
Simplify type annotations for tests/test_distributions.py (#5773, thanks @boringbyte!)
Simplify type annotations for tests/samplers_tests/test_qmc.py (#5775, thanks @boringbyte!)
Simplify type annotations for tests/sampler_tests/tpe_tests/test_sampler.py (#5779, thanks @boringbyte!)
Simplify type annotations for tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py (#5781, thanks @boringbyte!)
Simplify type annotations for tests/samplers_tests/tpe_tests/test_parzen_estimator.py (#5782, thanks @boringbyte!)
Simplify type annotations for tests/storage_tests/journal_tests/test_journal.py (#5783, thanks @boringbyte!)
Simplify type annotations for tests/storage_tests/rdb_tests/create_db.py (#5784, thanks @boringbyte!)
Simplify type annotations for tests/storage_tests/rdb_tests/ (#5785, thanks @boringbyte!)
Simplify type annotations for tests/test_deprecated.py (#5786, thanks @boringbyte!)
Simplify type annotations for tests/test_convert_positional_args.py (#5787, thanks @boringbyte!)
Simplify type annotations for tests/importance_tests/test_init.py (#5790, thanks @boringbyte!)
Simplify type annotations for tests/storages_tests/test_storages.py (#5791, thanks @boringbyte!)
Simplify type annotations for tests/storages_tests/test_heartbeat.py (#5792, thanks @boringbyte!)
Simplify type annotations optuna/cli.py (#5793, thanks @willdavidson05!)
Simplify type annotations for tests/trial_tests/test_frozen.py (#5794, thanks @boringbyte!)
Simplify type annotations for tests/trial_tests/test_trial.py (#5795, thanks @boringbyte!)
Simplify type annotations for tests/trial_tests/test_trials.py (#5796, thanks @boringbyte!)
Refactor some funcs in NSGA-III (#5798)
Simplify type annotations optuna/_transform.py (#5799, thanks @JLX0!)
Make the assertion messages in test_trial.py readable (#5800)
Simplify type annotations for tests/pruners_tests/test_hyperband.py (#5801, thanks @boringbyte!)
Simplify type annotations for tests/pruners_tests/test_median.py (#5802, thanks @boringbyte!)
Simplify type annotations for tests/pruners_tests/test_patient.py (#5803, thanks @boringbyte!)
Use study.ask() in tests instead of create_new_trial (#5807, thanks @unKnownNG!)
Simplify type annotations for tests/pruners_tests/test_percentile.py (#5808, thanks @boringbyte!)
Simplify type annotations for tests/pruners_tests/test_successive_halving.py (#5809, thanks @boringbyte!)
Simplify type annotations for tests/study_tests/test_optimize.py (#5810, thanks @boringbyte!)
Simplify type annotations for tests/hypervolume_tests/test_hssp.py (#5812, thanks @boringbyte!)
Refactor the MOTPE split (#5813)
Simplify type annotations for optuna/_callbacks.py (#5818, thanks @boringbyte!)
Simplify type annotations for optuna/samplers/_random.py (#5819, thanks @boringbyte!)
Simplify type annotations for optuna/samplers/_gp/sampler.py (#5823, thanks @boringbyte!)
Simplify type annotations for optuna/samplers/nsgaii/_crossovers/_sbx.py (#5824, thanks @boringbyte!)
Simplify type annotations for optuna/samplers/nsgaii/_crossovers/_spx.py (#5825, thanks @boringbyte!)
Simplify type annotations for optuna/samplers/nsgaii/_crossovers/_undx.py (#5826, thanks @boringbyte!)
Simplify type annotations for optuna/samplers/nsgaii/_crossovers/_vsbx.py (#5827, thanks @boringbyte!)
Simplify type annotations for optuna/samplers/nsgaii/_crossover.py (#5831, thanks @boringbyte!)
Simplify type annotations for optuna/samplers/testing/threading.py (#5832, thanks @boringbyte!)
Simplify type annotations for optuna/pruners/_patient.py (#5833, thanks @boringbyte!)
Use study.ask() in tests/pruners_tests/test_percentile.py (#5834, thanks @fusawa-yugo!)
Simplify type annotations for optuna/search_space/group_decomposed.py (#5836, thanks @boringbyte!)
Simplify type annotations for optuna/search_space/intersection.py (#5837, thanks @boringbyte!)
Simplify type annotations for optuna/storages/journal/_base.py (#5838, thanks @boringbyte!)
Simplify type annotations for optuna/storages/journal/_redis.py (#5840, thanks @boringbyte!)
Simplify type annotations for optuna/storages/journal/_storage.py (#5841, thanks @boringbyte!)
Simplify type annotations for optuna/study/_dataframe.py (#5842, thanks @boringbyte!)
Fix logger.warn() deprecation warning in GP module (#5843, thanks @iamarunbrahma!)
Simplify type annotations for optuna/study/_optimize.py (#5844, thanks @boringbyte!)
Simplify type annotations for optuna/study/_tell.py (#5845, thanks @boringbyte!)
Fix mypy errors due to numpy 2.2.0 (#5848)
Simplify type annotations for optuna/visualization/matplotlib/_contour.py (#5851, thanks @boringbyte!)
Refactor the fix for MyPy errors due to NumPy v2.2.0 (#5853)
Simplify type annotations for optuna/visualization/matplotlib/_parallel_coordinate.py (#5854, thanks @boringbyte!)
Simplify type annotations for optuna/visualization/matplotlib/_param_importances.py (#5855, thanks @boringbyte!)
Use study.ask() in tests/pruners_tests/test_successive_halving.py (#5856, thanks @willdavidson05!)
Simplify type annotations for optuna/visualization/matplotlib/_pareto_front.py (#5857, thanks @boringbyte!)
Simplify type annotations for optuna/visualization/matplotlib/_rank.py (#5858, thanks @boringbyte!)
Simplify type annotations for optuna/visualization/matplotlib/_slice.py (#5859, thanks @boringbyte!)
Refactor plot contour (#5867)
Refactor MOTPE weighting (#5871)
Simplify type annotations for optuna/visualization/_utils.py (#5876, thanks @boringbyte!)
Simplify type annotations for optuna/visualization/_contour.py (#5877, thanks @boringbyte!)
Simplify type annotations for optuna/_gp/gp.py (#5879, thanks @boringbyte!)
Simplify type annotations for optuna/study/_tell.py (#5880, thanks @boringbyte!)
Simplify type annotations for optuna/storages/_heartbeat.py (#5882, thanks @boringbyte!)
Simplify type annotations for optuna/storages/_rdb/storage.py (#5883, thanks @boringbyte!)
Simplify type annotations for optuna/storages/_rdb/alembic/versions/v3.0.0.a.py (#5884, thanks @boringbyte!)
Simplify type annotations for optuna/storages/_rdb/alembic/versions/v3.0.0.c.py (#5885, thanks @boringbyte!)
Simplify type annotations for optuna/storages/_rdb/alembic/versions/v3.0.0.d.py (#5886, thanks @boringbyte!)
Simplify type annotations for optuna/_deprecated.py (#5887, thanks @boringbyte!)
Simplify type annotations for optuna/_experimental.py (#5888, thanks @boringbyte!)
Simplify type annotations for optuna/_imports.py (#5889, thanks @boringbyte!)
Simplify type annotations for optuna/visualization/_slice.py (#5894, thanks @boringbyte!)
Simplify type annotations for optuna/visualization/_parallel_coordinate.py (#5895, thanks @boringbyte!)
Simplify type annotations for optuna/visualization/_rank.py (#5896, thanks @boringbyte!)
Simplify type annotations for optuna/visualization/_param_importances.py (#5897, thanks @boringbyte!)
Simplify type annotations for optuna/_callbacks.py (#5899, thanks @boringbyte!)
Simplify type annotations for optuna/storages/journal/_file.py (#5900, thanks @boringbyte!)
Simplify type annotations for tests/storages_tests/test_with_server.py (#5901, thanks @boringbyte!)
Simplify type annotations for tests/test_multi_objective.py (#5902, thanks @boringbyte!)
Simplify type annotations for tests/artifacts_tests/test_gcs.py (#5903, thanks @boringbyte!)
Simplify type annotations for tests/samplers_tests/test_grid.py (#5904, thanks @boringbyte!)
Simplify type annotations for optuna/study/_dataframe.py (#5905, thanks @boringbyte!)
Simplify type annotations for tests/visualization_tests/test_optimization_history.py (#5906, thanks @boringbyte!)
Simplify type annotations for tests/visualization_tests/test_intermediate_plot.py (#5907, thanks @boringbyte!)
Simplify type annotations for multiple test files in tests/visualization_tests/ (#5908, thanks @boringbyte!)
Avoid the port number conflict in the copy study tests (#5913)
Simplify annotations in tests/study_tests/test_study.py (#5923, thanks @sawa3030!)

Continuous Integration

Fix a GitHub action workflow for publishing to PyPI (optuna/optuna-integration#181)
Fix CI by adding a version constraint (optuna/optuna-integration#186)
Rename test_cache_is_invalidated and remove assert study._thread_local.cached_all_trials is None (#5733)
Use Python 3.12 for docs build CI (#5742)
Fix a GitHub action workflow for publishing to PyPI (#5759)
Install older kaleido to fix CI errors (#5771)
Add Python 3.12 to the Docker image CI (#5789)
Move mypy related entries in setup.cfg to pyproject.toml (#5861)

Other

Bump up version number to v4.2.0.dev (optuna/optuna-integration#178)
Bump up version number to 4.2.0 (optuna/optuna-integration#191)
Add CITATION.cff (#5746)
Update news to add the autosampler article (#5748)
Bump the version up to v4.2.0.dev (#5752)
Update the news section for Optuna 4.1 release (#5758)
Update pre-commit configuration file (#5847)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

@HideakiImamura, @JLX0, @KannanShilen, @SimonPop, @boringbyte, @c-bata, @fusawa-yugo, @gen740, @himkt, @iamarunbrahma, @kAIto47802, @ktns, @mist714, @nabenabe0928, @not522, @nzw0301, @porink0424, @sawa3030, @sulan, @unKnownNG, @willdavidson05, @y0z

optuna 4.2.0 v4.2.0 on Python PyPI