Collecting Java Application Metrics

Java App
Collecting Java Application Metrics

With the release of our Orchestrated Netprobe solution, collecting useful metrics from Java applications has become much easier. No longer is it necessary to manage JMX connections between Geneos and all of your applications. Instead, the Orchestrated Netprobe uses a push model where n applications publish metrics to a central agent via the Statsd protocol. This greatly reduces the operational overhead involved when spinning up new applications.

Statsd operates in a client/server fashion, where the client (the application) sends metrics to a server that maintains the current value of each metric. Every ten seconds, the server reports each metric to the visualization layer, which in our case is the Geneos Active Console. The benefit of this architecture is that the client does as little work as possible with a minimal resource footprint.

We provide a Java library that includes a Statsd client and a JVM metrics collector which can be used to instrument your applications.

Metric Types

The following types of metrics can be collected:

  • Counter: Non-negative integer value that can be incremented and decremented.
  • Gauge: Positive or negative floating point value that can be arbitrarily adjusted.
  • Timer: Elapsed time in milliseconds. The recorded values are reported as a histogram.
  • Set: Bucket of unique string values. The set is reported as a counter whose value is the number of unique elements in the set.
  • Event: User-defined event. Unlike other metrics which are reported at ten-second intervals, events are immediately reported upstream by the server.
  • Attribute: Static information that doesn't change during runtime.

Identifying Metric Sources

In order to identify its source, a Statsd metric is tagged with one or more "dimensions". A dimension is simply a string that has meaning within the context of your environment, and a set of dimensions comprise the unique identifier for a metric's source. The dimensions drive the hierarchical organization of data when displayed in Geneos Active Console.

Before instrumenting an application, you must choose a dimension scheme that makes sense for your environment. In many cases the hostname and app name may be enough, while in larger environments it may be necessary to use some combination of hostname, datacenter name, city etc.

We do provide some out-of-the-box schemes that can be employed with minimal configuration. They are driven by environment variables that the Statsd client looks for automatically. Kubernetes and Pivotal Cloud Foundry are supported (more details), but we'll use the simplest scheme of just HOSTNAME and APP_NAME. The client will translate these variables into dimensions of the same name (in lowercase).

Using the Client

There are two ways to collect metrics:

  • Instrument Your Application: Add the client as an application dependency, create an instance of the client and instrument your code where needed. This method is the only way to collect custom metrics.
  • Java Agent: Initialize the client automatically by adding a -javaagent flag to your Java startup command. This method supports the collection of JVM metrics only. Since no code modifications are required, it's a great way to start collecting useful metrics with zero development time and minimal operational changes. Just set some environment variables, change the startup command and you're done.

Instrumenting Your Application For Custom Metrics

  1. Include the Statsd client library in your application:
    1. To add as a Maven dependency, first define the ITRS Maven repository in your pom.xml or ~/.m2/settings.xml:

      <repositories> <repository> <id>itrsgroup-collection</id> <url>https://itrsgroup.bintray.com/collection</url> <snapshots> <enabled>false</enabled> </snapshots> </repository> </repositories>

      Then add the dependency in the POM:

      <dependencies> <dependency> <groupId>com.itrsgroup.collection</groupId> <artifactId>statsd-client</artifactId> <version>1.2.0</version> </dependency> </dependencies>

    2. Or manually download statsd-client-VERSION.jar from ITRS Resources and add it to the classpath.
  2. Build an instance of the Statsd client:

    StatsdClientBuilder builder = new StatsdClientBuilder();

    For many environments the default builder options will suffice. The resulting client publishes to localhost:8125 via UDP and auto-detects the dimensions based on certain environment variables:

    StatsdClient statsd = builder.build();

    However, if you need to make customizations:

    // Use TCP instead of UDP builder.useTcp(); // Send to a different host and port builder.server(InetAddress.getByName("192.168.1.1")).port(9000); // Define a custom dimension that will be added to all metrics builder.defaultDimension("region", "Americas"); // If opting not to use any of the built-in dimension schemes, they can be disabled builder.disableEnvironmentalDimensions(); // Define an error callback builder.errorCallback(e -> logger.error("Statsd error", e)); // Finally create the client StatsdClient statsd = builder.build();

    A few notes about the client:
    • It is thread safe — a single instance can be used throughout the application.
    • All methods are non-blocking with very little overhead and minimal garbage creation. Delivery to the server is handled in a different thread.
    • Its methods do not throw — by default, exceptions are silently caught and ignored. A callback can be provided if you wish to log these errors.
  3. Instrument your application:

    // Increment a counter void login() { if (failed) { statsd.increment("failed_logins"); } } // Set a gauge void cacheObject(Object obj) { cache.put(obj); statsd.gauge("cache_size", cache.getSizeInMB(), Unit.MEGABYTES); } // Register periodic gauge sampling (call this only once per gauge) void startup() { statsd.gauge("messages_in_buffer", buffer::getSize); } // Raise a custom event statsd.event("MyCustomEvent", "Something has just occurred", Severity.WARN); // Count unique values in a set void login(String username) { if (success) { statsd.unique("unique_logins", username); } } // Record a timer void receiveMessage(Message m) { long t1 = m.timestamp(); long t2 = System.currentTimeMillis(); statsd.timer("message_latency", t2-t1); } // Report a static attribute void startup() { statsd.attribute("product_version", "1.2"); }

  4. Collecting JVM Metrics:

    All that's required is to provide the client created in step 2:

    // Create the collector new JvmMetricsBuilder().client(statsd).build();

    The collector will poll for memory, GC, threading and runtime metrics every 10 seconds and report them via the Statsd client. Refer to the docs for a list of all metrics collected.

Using the Java Agent

To quickly collect JVM metrics from existing applications without writing any code, run the Statsd client and JVM metrics collector as a Java Agent. Simply download statsd-client-VERSION.jar from ITRS Resources and add the -javaagent flag to your startup command:

java -javaagent:statsd-client-VERSION.jar -jar app.jar

By default, the agent will publish to localhost:8125 via UDP. See the docs for more configuration details.

Dimension Environment Variables

Whether you instrumented your code or opted to use the Java Agent, one last important step remains before starting your application. Environment variables must be defined in order for the client to detect and tag each metric with dimensions.

If using an orchestration platform like Kubernetes or Pivotal Cloud Foundry, the variables can be defined in the application's deployment manifest.

In a non-orchestrated environment, define the variables in your startup script:

#!/bin/sh export APP_NAME=JavaApp export HOSTNAME=`hostname` java -jar app.jar

Viewing the Metrics

After starting your application, metrics will start to appear in Active Console after about 10 seconds. Here are some screenshots showing data from the examples above:

Custom Metrics:

Custom metrics

Heap Memory Usage:

Heaps

 

GC Stats:

GC

 

Thread Stats:

Thread Stats

Related posts