Can AI replace all our existing troubleshooting tools and carry out difficult tasks such as memory leak detection in one easy move?

Sadly, the answer right now is no.  But what if traditional tools and AI could work together, cutting our diagnostic time drastically? Deterministic AI does exactly this, giving us the best of both worlds: accuracy and speedy problem resolution.

In this article, we’ll use GCeasy Deterministic AI to solve a hard-to-find memory leak quickly and efficiently.

What Can Deterministic AI Tell You That Traditional AI Misses?

First, let’s define what we mean by deterministic problem-solving. This approach applies a fixed set of rules and algorithms to a set of data and produces a reproducible result. AI, on the other hand, uses probability to come up with the most likely solution. AI won’t necessarily always come up with identical answers, even when given the same set of data.

AI, although it’s becoming more and more advanced every day, still has a few problems when it comes to troubleshooting production problems:

  • AI has been known to “hallucinate”, or, in other words, present a solution that it views as probable as if it were an actual fact. This can be a huge time-waster when trying to get a production system back up to speed.
  • Unless you’re an experienced troubleshooter, you may not know the right questions to ask.
  • High-quality AI platforms can become expensive when dealing with very large data sets.
  • For performance tuning, we need precise accuracy rather than probability.

GCeasy’s deterministic AI, on the other hand, calculates precise metrics first before presenting them to an LLM for interactive exploration. It processes large amounts of raw data into a small subset of relevant facts in JSON format. This is illustrated in the diagram below.

Fig: Deterministic AI Process

This approach takes the time-consuming analysis away from potentially expensive services and makes sure conclusions are based on real and precisely calculated facts. It also gives us the option to switch to viewing accurate graphs and charts, where we can instantly spot trends.

Fig: GCeasy Option to View Graphs and Charts

See this article for more information on Deterministic AI.

A Real-World Memory Leak Detection Walkthrough

To illustrate all this in action, let’s take a hard-to-solve memory leak similar to a real-world scenario and work through it with GCeasy Deterministic AI. Ordinarily, we can spot a memory leak fairly easily by using GCeasy’s heap usage graph. The diagram below shows a healthy GC pattern contrasted to a memory leak pattern. The graph shows heap usage over time, with full GC events marked by red triangles.

Fig: Healthy GC Pattern vs Memory Leak Pattern

In the healthy pattern, although memory usage rises during normal operations, the GC is always able to bring it back to a similar level. When there is a memory leak, although the GC clears memory in every cycle, the bottom line keeps increasing over time, indicating that there are objects that aren’t being released to the GC.

So why do we need AI? Not all cases are this clear. In our example, the memory leak is intermittent. 

Some background and history: It’s an online sales application, and initially it sometimes lost performance before crashing with an OutOfMemoryError. The administrators increased the heap size using the -Xmx parameter, and it stopped crashing. However, it still intermittently had poor response times, and dropped sessions due to timeouts.

Eventually, after much frustration and many system outages, it was diagnosed as having an intermittent memory leak. The leak regularly cleared itself when day-end procedures were run. 

The problem was very difficult to isolate because it didn’t show the classic leak pattern, and GCs were able, with difficulty, to keep the heap within limits once -Xmx was set high. As the day progressed, the GC had to work harder and harder to clear enough memory for normal operations, and at times, full GC events ran almost back to back. Almost all the application’s resources were being used for garbage collection by the end of the day, resulting in application stalls.

In subsequent sections, we’ll write a small simulator program to mimic this pattern, and we’ll see how deterministic AI could have pinpointed the problem quickly and easily.

Building a Java Application to Simulate a Memory Leak

The sample application needs to simulate the normal running of the application, with some short-term objects that will be collected in minor GC events, and some longer-term objects that will only be collected in full GC events. Over and above these, it has a memory leak that slowly increases heap usage over time, in spite of full GC events. At defined intervals, a process simulating the day-end run clears the leaking object, which is a TreeMap that grows indefinitely.

The constructor starts threads to:

  • Create and release longer-term variables;
  • Run the clear-down at fixed intervals.

It then loops indefinitely, adding entries to the TreeMap. It also creates a short-lived variable on each cycle, which fills up the Young Generation.

import java.util.TreeMap;
public class BuggyProg18 {
// This program simulates an intermittent memory leak
// caused by a growing tree map. The tree map is cleared 
// at intervals selected on the command line in minutes.
// To mimic real-life scenarios, it also creates extra 
// variables, which are released normally. It has both 
// short-lived and longer-lived additional variables
// ====================================================
        TreeMap map=new TreeMap(); // The growing map
// Default timings
        static int pause = 1;     // Pause time between cycles
        static int clearPause=45; // Pause time between cleardowns
//
        public static void main(String[] args){
// If arguments are supplied, they override the defaults
           if(args.length>0)
              try {
                pause=Integer.parseInt(args[0]);
                }
              catch(Exception e) {}
           if(args.length>1)
              try {
                clearPause=Integer.parseInt(args[1]);
                }
              catch(Exception e) {}
             
// Create an object from this class
// ================================	  
           BuggyProg18 bug = new BuggyProg18();
        }

The constructor first initiates the background threads: one for the cleardown process, and a set of threads to create medium-term objects. It then loops indefinitely, adding a fairly long String to the tree map. Within the loop, it also creates a short-lived object to ensure the young generation fills up.

// Constructor
// ===========
        public BuggyProg18() { 
// Start threads to create clutter
           startClutter();
// Start a thread in the background that clears the map
           startCleardownThread();
	   long counter = 0;
// Loop that has no valid termination condition
// Simulates a memory leak
// ============================================           
           while (true) {
// Display a counter every 1000 records		
	        if (counter % 1000 == 0) {			    
		      System.out.println
                    ("Inserted " + counter + " Records to list");	
		  }
             try{Thread.sleep(pause*100);} catch (Exception e) {}		
// This map will grow indefinitely
// ============================================
             String s = new String("ABCDEFGHIJK");
             StringBuffer b = new StringBuffer(s);
             for(int i=0;i<60;i++)
                b.append(s);
             map.put(counter,b.toString());
	       ++counter;
// Create short-lived clutter variable to simulate normal
// program working
             byte[] temp = new byte[16384];   
	   }
	}
// End of constructor
// ==================

Next, we have a method that runs a thread in the background to clear the hash map at defined intervals.

// This method starts a separate thread running in the background.
// At intervals specified by the clearPause variable, it clears the 
// tree map, simulating a day end procedure
private void startCleardownThread() {
    Thread backgroundThread = new Thread(() -> {
        while (true) {
          try {
            Thread.sleep(clearPause * 60 * 1000); 
            } 
            catch (InterruptedException e) {
              Thread.currentThread().interrupt();
              break;
              }
          System.out.println("Clearing");
          map.clear();
       }
     }, "backgroundThread");
   backgroundThread.setDaemon(true); 
   backgroundThread.start();
   }
// End of cleardown thread method

Finally, we have the section that runs several threads, each of which creates a large object and holds onto it for 50 seconds before releasing it. This makes sure full GC cycles run regularly, since these objects fill up the old generation.

// This method starts a large number of threads at timed intervals 
// that will be used to create long-lived variables to fill up
// the Old Generation and force full GC events. 
// Again, this simulates normal program working
  private void startClutter() {
    Thread starterThread = new Thread(() -> {
        for (int i=0;i<20;i++) {
// Start a thread in the background that creates some clutter
                     startClutterThread();
                     try{Thread.sleep(1000);} catch (Exception e) {}	
                     }	
     
     }, "starterThread");
   starterThread.setDaemon(true);
   starterThread.start();
  }
// Method defines threads that create longer-living variables.
// Creates a variable, sleeps for 50 seconds, then releases it.
// -------------------------------------------------------------------
private void startClutterThread() {
    Thread clutterThread = new Thread(() -> {
       while (true) {
          try {
            byte[] clutter = new byte[500000];
            Thread.sleep( 50 * 1000); 
            clutter=null;
            } 
          catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            break;
        }
       }
     }, "clutterThread");
   clutterThread.setDaemon(true);
   clutterThread.start();
   }
}

How GCeasy Deterministic AI Accelerated Memory Leak Detection

Intermittent problems like this are often the hardest to troubleshoot, and much time can be wasted. Let’s load the GC logs into GCeasy’s Deterministic AI and see how it helps detect memory leaks more efficiently.

The GCeasy dashboard now gives a choice between the classic version and the new Deterministic AI option, as shown in the image below. The Deterministic AI tab allows us to ask a question and upload a log by pressing the ‘+’ button. 

Here is the list of questions that the deterministic AI answered: 

1. Determining Whether the Heap is Underconfigured

Fig: Upload to GCeasy Deterministic AI

 We uploaded the log and, since the application wasn’t performing as it should, asked “Is this heap under-configured?”

The response included the following:

Fig: GCeasy Response

This tells us that under-configuration is not the problem, and therefore increasing -Xmx won’t solve it. The answer is followed by a summary of key findings and recommendations. We then have a choice of asking a new question or viewing the report.

2. Does the Application Have a Memory Leak?

We next asked whether the application had a memory leak.

Fig: LLM Response

We now know that the log is not showing a classic memory leak pattern, so we need to delve further. 

3. Are All Full GC Events Fully Effective?

We asked whether full GC events were consistently bringing the heap back to the same level.

Fig: Analysis of Whether GC Events Consistently Clear to a Similar Level

We see that some GC events are effective, while others are not. 

4. Is There Any Pattern to the Memory Issues?

Next, we asked if there was any pattern to this finding.

Fig: Looking for Patterns

The LLM recommends taking heap dumps after ineffective GC events. 

5. Heap Dump Comparison Recommendations

We then asked the LLM to recommend the best times to take heap dumps to compare the state of memory between effective and non-effective GC events.

Fig: Heap Dump Timing Recommendations

Following the LLM’s Troubleshooting Recommendations

We then re-ran the program to obtain the heap dumps. Following the LLM instructions, we took heap dumps:

  • When the application had been running for approximately 60 minutes after an effective GC cycle;
  • When the application had been running for approximately 57 minutes after an ineffective GC. 

We then fed them into a heap dump analyzer tool. We chose HeapHero for the analysis. 

The heap dump taken after an ineffective GC cycle showed the following in the Largest Objects report:

Fig: Largest Object Report: High Usage

We see that more than 88% of the heap is occupied by the TreeMap object.

The dump taken after an effective GC event shows the following:

Fig: Largest Object Report: Low Usage

The TreeMap, which on the previous dump was using 88.54% of the heap, is not showing as a large object. If we search the report for the TreeMap, we see it’s retained memory is negligible.

Fig: TreeMap After Effective Heap Dump

This tells us there is an intermittent leak caused by the TreeMap object not being released for GC in a timely manner. We can then follow normal heap dump analysis procedures to trace what is keeping the object alive and fix the problem.

Conclusion

For difficult memory leak detection, deterministic tools such as GCeasy used together with LLMs speed up the diagnostic process, saving time, costs, and frustration. 

Deterministic AI gives us the advantage of AI without its disadvantages, since the LLM is working with accurate, precisely calculated metrics. Step into the future and try it out!