Parfait does not work with Tomcat, Cassandra

[mac133k]

Hi Guys,

I have Parfait 1.1.1 built, PCP 5.3.1 installed on my CentOS 7 box. I placed parfait-agent-jar-with-dependencies.jar in /usr/lib64/parfait.jar. It works with the ACME examples, but I tried to use it with Tomcat by adding -javaagent:/usr/lib64/parfait.jar to JAVA_OPTS in /etc/tomcat/tomcat.conf and no metrics appeared under mmv after Tomcat was started. I tried to generate some debug logs from Tomcat, but even with the logging level set to FINEST I only get this:

INFO: Command line argument: -javaagent:/usr/lib64/parfait.jar

I also tried to plug it into Apache's Cassandra. It also did not work, but generated more logs:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-cassandra-3.11.10/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib64/parfait.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
INFO  [main] 2021-07-05 11:51:18,109 ParfaitAgent.java:146 - Starting Parfait agent [cassandra]
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
	at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkState(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
	at io.pcp.parfait.AgentMonitoringView.checkCompositeDataItem(AgentMonitoringView.java:186)
	at io.pcp.parfait.AgentMonitoringView.createMonitorable(AgentMonitoringView.java:104)
	at io.pcp.parfait.AgentMonitoringView.register(AgentMonitoringView.java:92)
	at io.pcp.parfait.ParfaitAgent.startView(ParfaitAgent.java:119)
	at io.pcp.parfait.ParfaitAgent.startLocal(ParfaitAgent.java:128)
	at io.pcp.parfait.ParfaitAgent.premain(ParfaitAgent.java:148)
	... 6 more
FATAL ERROR in native method: processing of -javaagent failed

Is there something else required for this to work? If anyone has suggestions please share. I can do some more tests and checks if needed.

Regards,
MS

[natoscott]

Sorry, forgot to follow up on this earlier. I discussed this with @tallpsmith awhile ago on the slack #parfait channel - here's what was said (note questions at end):

nathans:  @psmith when ya get a minute, could you take a quick look at issue #83?
I reckon what's maybe happening there is tomcat/cassandra are creating some classes in a funky state (NoSuchMethodError?) ... such that the class reflection process freaks out?
maybe looking at that stacktrace will trigger something in your java/lizard brain about what that funkiness might be?  if so, lemme know & i could have a crack at tweaking the code to avoid the problem

psmith:  that looks like a difference between the compile time version of Google Guava that Parfait agent was built against, and the runtime version of Guava that the JVM is referencing.
I had thought the Java Agent method causes a different classloader to load the agent jar (which is an Uber jar of the Parfait dependencies isn’t it?).
If that were true, the Parfait agent jar should have its own static dependency versions, independent of whatever is running by the main Java (e.g. Tomcat).  Possible it doesn’t work like that - I haven’t been in that space for years sorry.
It would be interesting to ask:
What JDK version they’re using
What version of Tomcat (and then find out whether that version has Guava shipped with it

[mac133k]

Hi @natoscott ,

I install Parfait and Tomcat with Yum. Parfait was installed with the following dependencies:

  apache-commons-io.noarch 1:2.4-12.el7                apache-commons-lang.noarch 0:2.6-15.el7        apache-commons-logging.noarch 0:1.1.2-7.el7     
  avalon-framework.noarch 0:4.3-10.el7                 avalon-logkit.noarch 0:2.1-14.el7              bea-stax-api.noarch 0:1.2.0-9.el7               
  cal10n.noarch 0:0.7.7-4.el7                          geronimo-jms.noarch 0:1.1.1-19.el7             guava.noarch 0:13.0-6.el7                       
  jackson.noarch 0:1.9.4-7.el7                         javamail.noarch 0:1.4.6-8.el7                  javapackages-tools.noarch 0:3.4.1-11.el7        
  javassist.noarch 0:3.16.1-10.el7                     jdk1.8.x86_64 2000:1.8.0_202-fcs               joda-convert.noarch 0:1.3-5.el7                 
  joda-time.noarch 0:2.2-3.tzdata2013c.el7             jsr-311.noarch 0:1.1.1-6.el7                   log4j.noarch 0:1.2.17-17.el7_4                  
  objectweb-asm.noarch 0:3.3.1-9.el7                   python-javapackages.noarch 0:3.4.1-11.el7      python-lxml.x86_64 0:3.2.1-4.el7                
  si-units.noarch 0:0.6.5-1.el7                        slf4j.noarch 0:1.7.4-4.el7_4                   stax2-api.noarch 0:3.1.1-10.el7                 
  tomcat-servlet-3.0-api.noarch 0:7.0.76-16.el7_9      unit-api.noarch 0:1.0-3.el7                    uom-lib.noarch 0:1.0.1-5.el7                    
  uom-se.noarch 0:1.0.4-3.el7                          uom-systems.noarch 0:0.7-1.el7                 xalan-j2.noarch 0:2.7.1-23.el7                  
  xerces-j2.noarch 0:2.11.0-17.el7_0                   xml-commons-apis.noarch 0:1.4.01-16.el7        xml-commons-resolver.noarch 0:1.2-15.el7 

Tomcat 7 was installed with the following dependencies:

  apache-commons-collections.noarch 0:3.2.1-22.el7_2    apache-commons-daemon.x86_64 0:1.0.13-7.el7    apache-commons-dbcp.noarch 0:1.4-17.el7       
  apache-commons-pool.noarch 0:1.6-9.el7                ecj.x86_64 1:4.5.2-3.el7                       geronimo-jta.noarch 0:1.1.1-17.el7            
  jakarta-taglibs-standard.noarch 0:1.1.2-14.el7_1      tomcat-el-2.2-api.noarch 0:7.0.76-16.el7_9     tomcat-jsp-2.2-api.noarch 0:7.0.76-16.el7_9   
  tomcat-lib.noarch 0:7.0.76-16.el7_9  

So Guava v. 13.0-6.el7 was installed with Parfait from the standard CentOS repository and so did Tomcat, so I would expect Tomcat to depend on it too. However I just realised that JDK 1.8 was from my company's in-house repo and was a custom built. I will repeat the tests on my Fedora host over the weekend and see if there is a difference.

[tallpsmith]

@natoscott I'm not sure why installing the Parfait agent would actually bring in all those individual dependencies above, because the idea was that parfait-agent included them statically (to avoid this dependency hell). Are we accidentally declaring dependencies in the Parfait Agent RPM unnecessarily?

[natoscott]

@tallpsmith there are (were) multiple RPM packages - providing both parfait-agent and the API jars, and javadoc, across three separate RPMs. I expect its the separate Parfait APIs that have all those deps.

It's a moot point now though because of the Fedora Java changes of recent times - the parfait rpms were a casualty of mass removal of java packages we depended on. So nowadays the typical way Fedora users install parfait(-agent) is using a local build from github, there are no longer rpms unfortunately.