NetApp/harvest v23.11.0

23.11.0

on GitHub

23.11.0 / 2023-11-13 Release

📌 Highlights of this major release include:

• New FlexGroup dashboard that includes FlexGroup constituents. Thanks to @Sandromuc and @ewilts for raising.

• Harvest ChangeLog plugin to detect and monitor changes related to object creation, modification, and deletion.

• We improved how Harvest calculates power. As a result, you may notice a decrease in the reported power metrics compared to previous versions. Details here. Thanks to Evan Lee for reporting!

• Added conf_path variable for specifying the search path of Harvest templates.

• 📦 Streamlined the Harvest container installation process by eliminating the need to download a tar file. Running Harvest in a container is now simpler and more convenient.

• ⭐ Several of the existing dashboards include new panels in this release:

  • Aggregate and Volume dashboard includes performance and capacity tier data. Thanks to @ewilts for raising.
  • Workload dashboard includes QoS fixed Utilization % panels. Thanks to @faguayot for raising.
  • Disk Dashboard features performance panels at the disk raid-group level. Thanks to @kinderr95 for raising.

• 🌾 Harvest includes new templates to collect:

  • Cloud target metrics. Thanks to @mamoep for raising
  • CIFS Share metrics. Thanks to @s-kuchi for raising
  • IWarp metrics are included in RestPerf
  • object_store_server metrics are included in RestPerf
  • SMB2 metrics are included in RestPerf

• 📕 Documentation additions

  • Enhanced Quickstart guide for Harvest
  • NABox logs collection guide
  • Document poller ca_cert property. Thanks to Marvin Montanus for reporting!
  • Describe how Harvest calculates power. Thanks to Evan Lee for reporting!
  • Details about hidden_fields and filter for the Rest Collector. Thanks to Johnathan Warlick for raising!

• Enhanced the Volume dashboard to include clone information.

• ⚡ Optimized the Harvest binaries, significantly reducing their size.

• The Metadata dashboard works inside container deployments.

• The FabricPool panels in the Volume dashboard now support FlexGroup volumes. Thanks to @sriniji for reporting.

• Large harvest.yml files can be refactoring into smaller ones. Thanks to @llelik and @Pengng88 for raising.

• 💡 Added help text about metrics to more Harvest dashboard panels.

Announcements

‼️ IMPORTANT Due to ONTAP bug 1585893 the Harvest team recommends using ZapiPerf instead of RestPerf when collecting performance metrics. The RestPerf collector can be used once you upgrade your cluster to a version of ONTAP with the fix. Details in 1585893.

‼️ IMPORTANT Release 23.11 disables the CIFSSession templates by default. This change was made to prevent the generation of a large number of metrics. If you require these templates, you can enable them. Please be aware that enabling them may result in a significant increase in metric collection time, Harvest memory footprint, and Prometheus used disk space. These metrics are utilized in the SMB2 dashboard.

‼️ IMPORTANT Release 23.11 has updated its power metric calculation algorithm. As a result, you may notice a decrease in the reported power metrics compared to previous versions. To collect these metrics, Rest API permissions are required. For detailed information on the power algorithm, please refer to the power algorithm documentation.

‼️ IMPORTANT NetApp moved their communities from Slack to Discord, please join us there!

💡 IMPORTANT After upgrade, don't forget to re-import your dashboards, so you get all the new enhancements and fixes. You can import them via the bin/harvest grafana import CLI, from the Grafana UI, or from the Maintenance > Reset Harvest Dashboards button in NAbox.

Known Issues

• Some AFF A250 systems do not report power metrics. See ONTAP bug 1511476 for more details.

• ONTAP does not include REST metrics for offbox_vscan_server and offbox_vscan until ONTAP 9.13.1. See ONTAP bug
  1473892 for more details.

IMPORTANT 7-mode filers that are not on the latest release of ONTAP may experience TLS connection issues with errors like tls: server selected unsupported protocol version 301 This is caused by a change in Go 1.18. The default for TLS client connections was changed to TLS 1.2 in Go 1.18. Please upgrade your 7-mode filers (recommended) or set tls_min_version: tls10 in your harvest.yml poller section. See #1007 for more details.

Thanks to all the awesome contributors

🤘 Thanks to all the people who've opened issues, asked questions on Discord, and contributed code or dashboards
this release:

@Garydep, @MrObvious, @Pengng88, @Sandromuc, @ewilts, @faguayot, @jmg011, @kinderr95, @llelik, @mamoep, @rodenj1, @s-kuchi, @shawnahall71, @slater0013, @sriniji, @statdigger, @wyahn1219, AlessandroN, Dave, Diane, Evan Lee, Francesco, Heaven7, Johnathan Warlick, Madaan, Martijn Moret, Marvin Montanus, NicoSeiberth, RBrown, TonyHsieh, Watson9121, dbakerletn, imthenightbird, roller, twodot0h, tymercer

🌱 This release includes 38 features, 26 bug fixes, 24 documentation, 5 performance, 2 refactoring, 12 miscellaneous, and 7 ci pull requests.

🚀 Features

• Change Log Detection In Harvest (#2178)
• Remove Daemon Dependency (#2195)
• Enable More Golanglint Linters (#2313)
• Gcc Is Not Required To Build Harvest (#2322)
• Ontap Permission Errors Should Be Logged As Errors (#2326)
• Harvest Should Load Templates From A Set Of Conf Directories (#2329)
• Ontap Power Calculation For Embedded Shelf (#2333)
• Enable More Golanglint Linters (#2334)
• Harvest Auto-Support Should Include Instance Count In Collector Section (#2337)
• Set Allvalue To Null When Svm Regex Is Applied (#2340)
• Add Parity For String Types Between Restperf And Zapiperf (#2342)
• Tiering Data Changes For Volume - Template Change (#2343)
• Docker Workflow Doesn't Need Tar Download (#2354)
• Enable Ports By Default In Docker Generate (#2360)
• Support Comma Separated Aggrs In Perf Metrics (#2376)
• Harvest Should Support Multiple Poller Files To Allow Refactori… (#2388)
• Adding Iwarp Restperf Template (#2390)
• Adding New Panels In Disk Dashboard (#2391)
• Harvest Should Load Templates From A Set Of Conf Directories (#2394)
• Add Api To Rest Error Log (#2401)
• Add Clone Info To Volume Dashboard (#2402)
• Cifs Share Templates (#2405)
• Support Flexgroup Constituents In Template (#2410)
• Add Flexgroup To Fabricpool Panels (#2419)
• Smb2 Restperf Counters (#2420)
• Adding Fc Rest Template For Fibre Channel Switch (#2424)
• Metric Doc Needs To Handle Templates With Same Object Names (#2426)
• Antiransomwarestate Label Should Be Exported (#2432)
• Metadata Dashboard Should Work With Containers And Remove System Resources Panel (#2433)
• Adding Restperf Object_store_server Template (#2435)
• Update Ci To Use Docker Run And Update Permissions (#2436)
• Enable More Golanglint Linters (#2439)
• Qos Fixed Utilization % Panels (#2445)
• Description Fetched From Ontap Docs Via Cli (#2454)
• Disable Cifssession Template (#2455)
• Add Labels Defined In Harvest Config To Metadata Metrics (#2456)
• Add Link_up Counter For Fcp (#2464)
• Implementing Support For Randomized Start Times In Tasks (#2465)

🐛 Bug Fixes

• Qos Policy Not Updated In Workload Counters (#2318)
• Bin Dir Prints Error When There Are No Files In It (#2324)
• Harvest Docker Generate Should Prefix Relative Paths With Dot (#2346)
• Sg Overview Dashboard Should Use Percentunit For `Top X Tenants … (#2348)
• Log Error When Config File Is Not Found During Generate Cli (#2349)
• Include Cloud_target Template For Zapi Collector (#2350)
• Sg Fabricpool Dashboard Should Use Bucket Name Variable (#2353)
• Add Quotes For Ports In Docker Compose Files (#2355)
• Remove Only Harvest Related Containers In Make Ci-Local (#2356)
• Harvest Power Algorithm Should Handle Shared Psus (#2359)
• Don't Expose End-Points With Port=False (#2361)
• Enable Root Aggregate Power (#2363)
• Read Templates From Harvest_conf When Set (#2366)
• Read Harvest.yml From Harvest_conf When Set (#2367)
• Sg Collector Throws Npe (#2384)
• Lif Template Should Use Unique Instancekeys (#2393)
• Don't Schedule A Task More Frequently Than Its Default Schedule (#2408)
• Volume Iops Per Type Legend Should Use Mean, Last, Max Instead… (#2411)
• Override In Templates Should Be From Counters (#2417)
• Make Zapi Rest Cli Consistent For Max-Records Arg (#2447)
• Exclude Dirs For Docker Generate (#2448)
• Support Multi Key For Disk Plugin Zapiperf (#2449)
• Handled Cloud_target For Fabricpool (#2467)
• Harvest.yml Defaults Should Be Applied To Child Harvest.yml (#2471)
• Update Flexgroup Text In Dashboard (#2474)
• Handled Missing Uuid In Volume For Change_log (#2478) (#2479)

📕 Documentation

• Add Workload Information To Release Notes (#2316)
• Describe How To Collect Nabox Logs (#2321)
• Add Storagegrid To Extend Docs (#2336)
• Add Details About Hidden_fields And Filter For Rest Collector (#2341)
• Describe How Harvest Calculates Power (#2362)
• Describe How To Upgrade To Nightly Image (#2370)
• Fix Nightly Image Docker Generate (#2372)
• Document Poller Ca_cert Property (#2374)
• Harvest Configuration File From Other Location With Docker Run Generate (#2377)
• Fix Link To Issue (#2382)
• Update Container Docs (#2397)
• Add Download Swagger Permission (#2398)
• Add Curl To Twisty (#2406)
• Generate Metrics Should Include Metrics Created By Builtin Plu… (#2413)
• Fix Cluster_new_status Metric (#2423)
• Add Build Docker Image Section And Fix Links (#2425)
• Fix Docker Build Path (#2429)
• Fix Typo In Ems Docs (#2431)
• Example Auto-Support Payload Should Include Rest Example (#2442)
• Quickstart Should Mention Install Section (#2446)
• Changelog Doc (#2453)
• Generate Ordered Api List For Metrics Doc (#2458)
• Fix Docker Run Steps (#2473)
• Describe Conf_path (#2480)

⚡ Performance

• Rest Client Allocs Improvements (#2381)
• Use Parse Instead Of Getmany For Gjson (#2385)
• Remove Extra Dict Wrapper (#2396)
• Avoid Byte Slice To String Conversion (#2407)

Refactoring

• Simplify Conf Path Management (#2325)
• Update Href Creation (#2399)

Miscellaneous

• Minor Release Issue Fixes (#2293)
• Release Doc Update (#2294)
• Merge 23.08.0 To Main (#2310)
• Init Bool Variable (#2319)
• Update All Dependencies (#2339)
• Update Actions/Checkout Action To V4 (#2358)
• Update Module Github.com/Tidwall/Gjson To V1.17.0 (#2378)
• Remove Drilldown From Panel Titles (#2387)
• Update All Dependencies (#2395)
• Update All Dependencies (#2414)
• Bump Golang.org/X/Net From 0.14.0 To 0.17.0 In /Integration (#2422)
• Update All Dependencies (#2459)

🔨 CI

• Create Draft Release Highlights (#2314)
• Bump Go And Dependencies (#2351)
• Reduce Artifacts Size (#2403)
• Bump Go (#2409)
• Bump Go (#2418)
• Remove Unused Volumes From Ci Machines (#2440)
• Bump Go (#2469)