reactor / BlockHound

Java agent to detect blocking calls from non-blocking threads.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

How to integrate with Akka?

Poorva17 opened this issue · comments

Hi @bsideup ,

I am trying to integrate blockhound for akka actors. I am running example code from your blog - https://bsideup.github.io/posts/blockhound_with_akka/

When I run this code I get following blocking calls reported -

	at esw.actor.ActorClass.lambda$main$2(ActorClass.java:32)
	at reactor.blockhound.BlockHound$Builder.lambda$install$8(BlockHound.java:382)
	at reactor.blockhound.BlockHoundRuntime.checkBlocking(BlockHoundRuntime.java:89)
	at java.base/jdk.internal.misc.Unsafe.park(Unsafe.java)
	at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
java.lang.Exception: java.io.FileInputStream#readBytes
	at esw.actor.ActorClass.lambda$main$2(ActorClass.java:32)
	at reactor.blockhound.BlockHound$Builder.lambda$install$8(BlockHound.java:382)
	at reactor.blockhound.BlockHoundRuntime.checkBlocking(BlockHoundRuntime.java:89)
	at java.base/java.io.FileInputStream.readBytes(FileInputStream.java)
	at java.base/java.io.FileInputStream.read(FileInputStream.java:279)
	at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
	at java.base/java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.base/java.io.BufferedReader.read(BufferedReader.java:182)
	at java.base/java.io.StreamTokenizer.read(StreamTokenizer.java:500)
	at java.base/java.io.StreamTokenizer.nextToken(StreamTokenizer.java:544)
	at java.base/sun.security.provider.PolicyParser.read(PolicyParser.java:181)
	at java.base/sun.security.provider.PolicyFile.init(PolicyFile.java:514)
	at java.base/sun.security.provider.PolicyFile$3.run(PolicyFile.java:358)
	at java.base/sun.security.provider.PolicyFile$3.run(PolicyFile.java:355)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/sun.security.provider.PolicyFile.initPolicyFile(PolicyFile.java:355)
	at java.base/sun.security.provider.PolicyFile.init(PolicyFile.java:345)
	at java.base/sun.security.provider.PolicyFile.<init>(PolicyFile.java:298)
	at java.base/java.security.Policy.loadPolicyProvider(Policy.java:207)
	at java.base/java.security.Policy.getPolicyNoCheck(Policy.java:178)
	at java.base/java.security.ProtectionDomain.implies(ProtectionDomain.java:321)
	at java.base/java.security.ProtectionDomain.impliesWithAltFilePerm(ProtectionDomain.java:353)
	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:450)
	at java.base/java.security.AccessController.checkPermission(AccessController.java:895)
	at ch.qos.logback.core.util.Loader$1.run(Loader.java:48)
	at ch.qos.logback.core.util.Loader$1.run(Loader.java:45)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at ch.qos.logback.core.util.Loader.<clinit>(Loader.java:45)
	at ch.qos.logback.classic.util.ContextInitializer.findURLOfDefaultConfigurationFile(ContextInitializer.java:119)
	at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:148)
	at org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:84)
	at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
	at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
	at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
	at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
	at akka.actor.typed.internal.adapter.ActorSystemAdapter.<init>(ActorSystemAdapter.scala:90)
	at akka.actor.typed.internal.adapter.ActorSystemAdapter$AdapterExtension.<init>(ActorSystemAdapter.scala:126)
	at akka.actor.typed.internal.adapter.ActorSystemAdapter$AdapterExtension$.createExtension(ActorSystemAdapter.scala:133)
	at akka.actor.typed.internal.adapter.ActorSystemAdapter$AdapterExtension$.createExtension(ActorSystemAdapter.scala:129)
	at akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:1159)
	at akka.actor.ExtensionId.apply(Extension.scala:78)
	at akka.actor.ExtensionId.apply$(Extension.scala:77)
	at akka.actor.typed.internal.adapter.ActorSystemAdapter$AdapterExtension$.apply(ActorSystemAdapter.scala:129)
	at akka.actor.typed.internal.adapter.ActorSystemAdapter$.apply(ActorSystemAdapter.scala:122)
	at akka.actor.typed.internal.adapter.ActorContextAdapter.<init>(ActorContextAdapter.scala:59)
	at akka.actor.typed.internal.adapter.ActorAdapter.ctx(ActorAdapter.scala:66)
	at akka.actor.typed.internal.adapter.ActorAdapter.preStart(ActorAdapter.scala:244)
	at akka.actor.Actor.aroundPreStart(Actor.scala:545)
	at akka.actor.Actor.aroundPreStart$(Actor.scala:545)
	at akka.actor.typed.internal.adapter.ActorAdapter.aroundPreStart(ActorAdapter.scala:56)
	at akka.actor.ActorCell.create(ActorCell.scala:637)
	at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:509)
	at akka.actor.ActorCell.systemInvoke(ActorCell.scala:531)
	at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:294)
	at akka.dispatch.Mailbox.run(Mailbox.scala:229)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:242)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)

This stack trace shows so many blocking calls which are internal to akka implementation. I am only interested in detecting blocking calls in my actor code.

As per akka 2.6 documentation internal calls of akka use internal dispatcher not default dispatcher. So I modified nonBlockingThreadPredicate to monitor only default dispatcher thread -> thread.getName().contains("greeter-akka.actor.default-dispatcher"). Still I am getting all internal blocking calls reported. I have attached my code below.

package esw.actor;

import akka.actor.typed.ActorSystem;
import akka.actor.typed.Behavior;
import akka.actor.typed.javadsl.Behaviors;
import reactor.blockhound.BlockHound;

public class ActorClass {

    static final Behavior<String> behavior = Behaviors.receive((ctx, who) -> {
        if ("Blocking call".equals(who)) {
           Thread.sleep(2000);
        }

        System.out.println("Hello, " + who);

        return Behaviors.same();
    });

    public static void main(String[] args) throws Exception {
        BlockHound.install(builder -> {
            builder.nonBlockingThreadPredicate(p -> {
                return p.or(thread -> thread.getName().contains("greeter-akka.actor.default-dispatcher"));
            });
            builder.disallowBlockingCallsInside(
                    "akka.dispatch.MonitorableThreadFactory.AkkaForkJoinWorkerThread",
                    "execute"
            );
            builder.allowBlockingCallsInside("java.io.PrintStream", "println");
            builder.allowBlockingCallsInside("jdk.internal.misc.Unsafe", "park");
            builder.blockingMethodCallback(m -> {
                new Exception(m.toString()).printStackTrace();
            });
        });
        System.out.println("starting");
        var greeter = ActorSystem.create(behavior, "greeter");

        greeter.tell("Akka");
        greeter.tell("Java");
        greeter.tell("Blocking call");

        Thread.currentThread().join();
    }
}
  1. Is there any way to bypass akka internal blocking calls and only focus on my actor code?
  2. What does this snippet is for builder.disallowBlockingCallsInside( "akka.dispatch.MonitorableThreadFactory.AkkaForkJoinWorkerThread", "execute" );

Is there any way to bypass akka internal blocking calls and only focus on my actor code?

Yes, you can whitelist them, see:
https://github.com/reactor/BlockHound/blob/1.0.2.RELEASE/docs/customization.md#dis-allowing-blocking-calls-inside-methods

What does this snippet is for builder.disallowBlockingCallsInside( "akka.dispatch.MonitorableThreadFactory.AkkaForkJoinWorkerThread", "execute" );

This code says "report blocking calls that were executed from AkkaForkJoinWorkerThread#execute or down the stack (unless marked with allowBlockingCallsInside).

When the article was written, it was mandatory to mark some entry point, otherwise nothing would be reported. Starting with BlockHound's release version, there is no need to do so (I guess I need to update the article).

It is still useful to disallow certain methods, you will find some info about it in the tips:
https://github.com/reactor/BlockHound/blob/1.0.2.RELEASE/docs/tips.md#how-to-select-what-to-whitelist

I have whitelisted in above code snippet.

builder.allowBlockingCallsInside("jdk.internal.misc.Unsafe", "park");

Which did not solve my problem

May be I have chosen wrong method to whitelist. I would like to understand that from stack trace of blocking call which method I should whitelist

Whitelisting Unsafe#park is not a good idea, since it is a blocking call that should be reported.
See the linked tips for recommendations on how to whitelist things.

Also, in your case, it is FileInputStream#readBytes triggered from StaticLoggerBinder#<clinit>. It is a good idea to whitelist StaticLoggerBinder#<clinit>.

Whitelisting Unsafe#park is not a good idea, since it is a blocking call that should be reported.
See the linked tips for recommendations on how to whitelist things.

@bsideup i am also working on an Akka integration and the hard part is integrating the akka dispatcher which is a ForkJoinPool.
In some circumstances, it invokes:

As you said, i wouldn't like to whiltelist Unsafe#park but how?
Do you think i should whitelist the methods ForkJoinPool#runWorker and ForkJoinPool#poll? If yes, the risk would be to white list other execution branches.

closing inactive question