What's Changed
- ROB-1704: Standardize datasource titles by @nherment in #776
- ROB-1643: document eval tags by @nherment in #633
- ROB-1741: datadog rds analysis toolset by @nherment in #653
- Add 2 evals for loki + port-forwarding support for evals that need it by @aantn in #783
- ROB-1873-validate-datadog-logs-query-working-as-expected by @moshemorad in #780
- Show better error message if eval uses invalid tags by @aantn in #774
- Dd missing test by @arikalon1 in #777
- Improve medium evals by @aantn in #784
- Fix missing workload analysis by @moshemorad in #786
- New evals for logging, runbooks, and database scenarios by @aantn in #788
- Tiny fixes on new evals logging and runbooks by @aantn in #791
- Add right information for investigate evals by @Sheeproid in #790
- More concise eval docs by @aantn in #792
- Fix mkdocs with custom domain by @aantn in #793
- Update CNAME for docs build by @aantn in #795
- Fix kubernetes events bug by @aantn in #789
- ROB-1689 call_stream support for api chat by @RoiGlinik in #759
- return errors on empty queries. Prompy update by @arikalon1 in #797
- add eval for k8s events by @aantn in #801
- Rename tempo and loki toolset in docs by @aantn in #787
- fixes bugs found during gpt-5 testing by @Sheeproid in #804
- Improve eval docs by @aantn in #803
- cap the number of tokens reserved for output by @arikalon1 in #805
- feat: adding option for custom annotations by @DMSwimTech in #802
- fix gpt5 temperature bug by upgrading litellm by @aantn in #808
- add support for REASONING_EFFORT environment variable by @aantn in #809
- force mock for datadog tests by @Sheeproid in #812
- add docs on modellist by @aantn in #813
- Update test_case.yaml by @pavangudiwada in #817
- eval improvements by @aantn in #816
- changed test 15 to be about readiness-probe and add new test for name confusion by @Sheeproid in #811
- fix judge for counter test to allow for returning the plain number by @Sheeproid in #810
- add docs on ASK_HOLMES_TEST_TYPE by @aantn in #820
- fix: replace pwd with os compatible getpass by @mainred in #814
- Improve datadog evals by @aantn in #821
- Update argocd.md by @aantn in #778
- WIP: Advanced logging capabilities by @aantn in #743
- Add eval for semi-complex kubectl queries by @aantn in #819
- Fix datadog api confuson by @nherment in #799
- allow host name in judge by @Sheeproid in #824
- ROB-1767 use correct date by @nherment in #794
- Rename fetch_pod_logsError_default_robusta-holmes_2023-06-12T00_00_00… by @moshemorad in #825
- [ROB-1799] fixed output formatting issues - better sonnet-4 support by @Avi-Robusta in #765
- Evals: remove requirement for name in test 80 by @Sheeproid in #826
- move some evals out of easy + delete bad eval by @aantn in #828
- rds test 22 better setup by @Sheeproid in #831
- Evals: speed up test 22 by triggering bad log line faster by @Sheeproid in #832
- Pin postgrest version by @moshemorad in #838
- small copy improvements by @aantn in #833
- Check dal is enabled by @moshemorad in #840
- Update robusta.md by @aantn in #843
- dd image fix by @arikalon1 in #800
- chore: Add ai safety prompt to system prompt by @nilo19 in #823
- ROB-1434: Strict tools by @nherment in #818
- Add icon_url to tool call result by @moshemorad in #842
- Fixing loading configs from env vars and Prometheus Amp support by @Avi-Robusta in #848
- Support revisionhistorylimit by @DrFaust92 in #796
- ROB-1824: do not use robusta ai api key if target model is not robusta ai by @nherment in #845
- Add community meeting announcement and banner by @pavangudiwada in #852
- changed test 89 to medium as it usually fails with gpt-4o by @Sheeproid in #853
- Add triple chain-of-causation with kafka test by @Sheeproid in #850
- Docs: add reusable markdown fences for Holmes toolset configuration by @aantn in #815
- fix: pin openai < 1.100.0 by @mainred in #859
- make test 18 faster by building an app that fails faster by @Sheeproid in #860
- Docs: update mkdocs port by @aantn in #863
- POC for running evals live in CI/CD by @aantn in #844
- bugfix: convert int to str to support negative int values by @Sheeproid in #866
- ROB-1824 fix: use env var for robusta api endpoint for on premise users by @nherment in #858
- Update llm-evaluation.yaml by @aantn in #867
- Subtasks by @arikalon1 in #851
- Rob 1942 holmes return rate limit when happen by @moshemorad in #854
- Add docs on braintrust org by @aantn in #865
- Use python 3.11 slim bookworm by @moshemorad in #870
- easier usage of OPEN_API_KEY if AZURE_BASE is not defined by @Sheeproid in #872
- Minor docs fix by @pavangudiwada in #856
- Raise holmes request memory by @moshemorad in #877
- Add ai_message for intermediate message & reasoning by @moshemorad in #879
- [ROB-1535] Added llm support to investigation by @Avi-Robusta in #857
New Contributors
- @DMSwimTech made their first contribution in #802
- @DrFaust92 made their first contribution in #796
Full Changelog: 0.12.5...0.13.0