It was not due to inactivity that there were no stagemonitor releases in the last few months. The reason was that we were busy implementing support for distributed tracing in stagemonitor. It has almost been a complete rewrite of the tracing functionality because at the heart of the code is now the OpenTracing API. Hence the jump in versions from 0.31.0 to 0.80.0.

So it’s time for stagemonitor users to get familiar with the new terminology. One of the former most central classes RequestTrace has beed replaced with the Span interface the OpenTracing API offers. Traces consist of multiple spans from different services which belong together.

Supporting OpenTracing was a major effort but in the end lead to a better design and cleaner code. For example the RequestMonitor class previously had too many responsibilities: Managing the life cycle of a RequestTrace, tracking metrics and also reporting RequestTraces. This functionality has been split up into SpanEventListeners which have one dedicated functionality.

Please also read the blog post about this release.

Features

• Tracing
  • Sampling
    • There are two main types of sampling: Rate based and percentage based.
    • Sampling decisions can be based on the operation type like ‚jdbc‘.
  • Adds ability to exclude Span tags by config. You can use this configuration option for example to exclude the tags call_tree_ascii or call_tree_json.
  • Adds config options to monitor @async and @scheduled annotated methods
  • Stagemonitor can now report traces to Zipkin. In order to do so, you need to include the dependency stagemonitor-tracing-zipkin. See https://github.com/stagemonitor/boot-zipkin for a example spring boot application which reports traces generated by stagemonitor to Zipkin.
  • Instances SpringRestTempate are automatically instrumented so that they propagate tracing headers downstream.
  • Adds PreExecutionInterceptorContext#shouldNotCollectCallTree and PreExecutionInterceptorContext#mustCollectCallTree to make it fully customizable wether to collect the call tree #251.
• End User Monitoring
  • The end user monitoring got a major overhaul. It is now possible with stagemonitor to monitor arbitrary HTML pages. You just have to add a javascript snippet to your webpages and your application will send data to the stagemonitor-eum-server, a spring boot application which receives beacons from your HTML pages and sends them to Elasticsearch. Stay tuned for an upcoming post at the stagemonitor mailing list for more information.
• Add GcTimeTrackingSpanEventListener #272. Thx @marcust
• Add count and time of getConnection calls #271. Thx @marcust
• Core
  • Adds ability to define dependencies between plugins. This influences the order at which plugins are initialized.
  • Adds StagemonitorPlugin#onInit(Runnable) method which lets you register callback which get executed as the plugin initializes.
• Configuration
  • The configuration has it’s own module now and it can be used outside stagemonitor as well
  • Many configuration options have been renamed for consistency. Most of the changes are backwards compatible as the configuration options can have „alias keys“.
  • The configuration can now handle the case where a config option might be null by supporting java.util.Optional via the builder method ConfigurationOptionBuilder#buildOptional. This obviously only works if you are using Java 8+. The good news is that stagemonitor itself is still Java 6 compatible.
• EhCache:
  • When initializing stagemonitor, CacheManager might not yet have been initialized. In that case, stagemonitor tries to init again on the first incoming request
• Dashboards
  • The now is a single dashboard for all request types. Previously, there was one for internal and for external requests. These dashboards offer an operation type filter you can use to only see requests of a specific type like http or jdbc. Note that there might be some changes in the future how this „type“ is named (see opentracing/specification#72).
• Spring Boot support
  • Spring Boot 1.4+ application don’t have to manually set up a EmbeddedServletContainerCustomizer anymore. All that’s left is to call Stagemonitor.init(); at the first line of the main method.

Breaking changes

Major braking changes

• stagemonitor-requestmonitor has been renamed to stagemonitor-tracing
  • As the package structure has changed from org.stagemonitor.requestmonitor to org.stagemonitor.tracing, all of your custom META-INF/services/org.stagemonitor.requestmonitor.* files have to be renamed.
• stagemonitor-web has been renamed to stagemonitor-web-servlet
  • The package structure has been renamed accordingly
• In order to report tracing information to Elasticsearch, you have to include the dependency stagemonitor-tracing-elasticsearch
• Stagemonitor now requires Elasticsearch 5.x, Kibana 5.x and Grafana 4.x
• The @MonitorRequests annotation has been renamed to @Traced
• The class RequestTrace has been completely replaced by io.opentracing.Span

Minor breaking changes

• The stagemonitor-spans* indices won't be deleted by default after 7 days
• Always samples/reports external spans. If your application makes excessive use of for example jdbc queries, you might want to deactivate or rate limit the collection of specific spans.
• The parsing of the User-Agent header is deactivated by default. If you want to enable it, set stagemonitor.requestmonitor.http.parseUserAgent = trueand add a dependency on net.sf.uadetector:uadetector-resources:2014.10. This library is no longer maintained however. If you want to have accurate User Agent information now, it however recommended to use the Elasticsearch ingest user agent plugin. See https://www.elastic.co/guide/en/elasticsearch/plugins/master/ingest-user-agent.html. Stagemonitor will support a up-to-date library in the future out of the box.
• The configuration will be read to and written from a different Elasticsearch index (stagemonitor-configuration/configuration instead of stagemonitor/configuration). This change has been made in preparation of Elasticsearch 6, as only one type per index will be allowed then.
• The Elasticsearch configuration source now overrides other sources except for the transient configuration source.
• The old Grafana 1 Graphite dashboards have been removed
• Removed configuration options stagemonitor.tracing.cpuTime and stagemonitor.tracing.collectExternalRequestTimePerRequest
• Sampling
  • Removed config keys stagemonitor.requestmonitor.collectRequestStats, stagemonitor.requestmonitor.onlyReportNRequestsPerMinuteToElasticsearch and stagemonitor.requestmonitor.external.onlyReportNExternalRequestsPerMinute. Use the new stagemonitor.tracing.sampling.* configuration options to use different sampling rates per operation type.
• Removal of deprecated methods
  • StagemonitorPlugin#initializePlugin
  • StagemonitorPlugin#getPathsOfWidgetTabPlugins
  • StagemonitorPlugin#getPathsOfWidgetMetricTabPlugins

Bug Fixes

• Fixes possible NPE in HttpClient #239. Thx @fantast
• Fix possible ConcurrentModificationExceptions when urning on the user agent parsing
• Fix index management for stagemonitor-spans index
• Only update stagemonitor-spans* index pattern if it has not been modified