一次由groovy引起的fullGC问题排查("Groovy脚本引发Full GC问题深度排查与解决")

一、背景介绍

在一次项目部署过程中，我们遇到了一个由Groovy脚本引起的Full GC问题。Full GC（全线程堆垃圾回收）是Java虚拟机（JVM）对整个堆空间进行垃圾回收的一种操作，它会让应用程序的响应时间变长，甚至出现卡顿。下面将详细介绍我们是怎样一步步排查并解决这个问题的。

二、问题现象

问题出现时，我们观察到以下现象：

• 应用程序响应时间变长，用户体验受到影响。
• JVM堆内存使用率持续上升，最终触发Full GC。
• Full GC后，堆内存使用率有所下降，但随后又迅速上升。

三、排查过程

1. 收集日志信息

首先，我们收集了JVM的GC日志、堆栈信息以及应用程序的日志。以下是GC日志的一个示例片段：

2023-04-01T12:34:56.789+0800: 109.789: [Full GC (Allocation Failure) 109.789: [CMS: 9769K->8703K(1048576K), 0.0128940 secs] 109.802: [CMS PermGen: 917K->917K(12288K)] 109.802: [CMS CMSPermGen: 0K->0K(12288K)] 109.802: [CMS CMSOldGen: 9769K->8703K(1048576K)] 109.802: [CMS CMSNewGen: 1024K->1024K(1024K)] 109.802: [CMS CMSInitialMark: 0.0010526 secs] 109.802: [CMS CMSConcurrentMark: 0.0035356 secs] 109.806: [CMS CMSConcurrentPreclean: 0.0000404 secs] 109.806: [CMS CMSConcurrentAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSFinalRemark: 0.0000172 secs] 109.806: [CMS CMSFinalRemark: 0.0000172 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.0000116 secs] 109.806: [CMS CMSAbortablePreclean: 0.000
本文由IT视界版权所有,禁止未经同意的情况下转发