quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.

Home Page:https://quarkus.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Keycloak authorization: "Invalid identity" (userId=null)

vicentini opened this issue · comments

Describe the bug

Trying the security-keycloak-authorization-quickstart, there is a problem when running in a containerized environment (Kubernetes or even single node docker installation). The retreived access token is not working and I see the following log outputs when running both Keycloak and quickstart application running in containers.

Keycloak log:

WARN  [org.keycloak.events] (default task-6) type=PERMISSION_TOKEN_ERROR, realmId=quarkus, clientId=backend-service, userId=null, ipAddress=10.12.1.3, error=invalid_token, reason='HTTP 500 Internal Server Error', auth_method=oauth_credentials, audience=backend-service, grant_type=urn:ietf:params:oauth:grant-type:uma-ticket, permission=df1b74a9-3f10-499d-a581-368de48e512b, client_auth_method=client-secret

Authorization quickstart example log:

Authorization failed: java.lang.RuntimeException: org.keycloak.authorization.client.util.HttpResponseException: Unexpected response from server: 400 / Bad Request / Response from server: {"error":"unauthorized_client","error_description":"Invalid identity"}
...

The userId=null is probably the culprit, but why is it null in the first place?

Running only Keycloak in a container and the REST application on the host machine with the quarkus-app.jar file, the example is working as expected (even with the same access token retrieved from Keycloak).

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

Build quickstart example container image with additional dependency quarkus-container-image-docker and property quarkus.container-image.build=true.

On a host machine with docker installation (+quarkus network)

  1. Start Keycloak in a container with
    docker container run -d --name keycloak -e KEYCLOAK_USER=admin -e KEYCLOAK_PASSWORD=admin -e DB_VENDOR=H2 --network quarkus -p 8080:8080 jboss/keycloak:15.0.2 -b=0.0.0.0
    Import keycloak realm from quickstart example
  2. Start quickstart example container with
    docker container run -d -e QUARKUS_OIDC_AUTH_SERVER_URL="http://keycloak:8080/auth/realms/quarkus" -e QUARKUS_OIDC_TOKEN_ISSUER="http://localhost:8080/auth/realms/quarkus" --network quarkus -p 8081:8080 quarkus/security-keycloak-authorization-quickstart:1.0.0-SNAPSHOT
  3. Get user access token
curl -L -X POST 'http://localhost:8080/auth/realms/quarkus/protocol/openid-connect/token' \
-H 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'username=alice' \
--data-urlencode 'password=alice' \
--data-urlencode 'grant_type=password' \
--data-urlencode 'client_id=backend-service' \
--data-urlencode 'client_secret=secret'
  1. Request user endpoint
    curl -L -X GET 'http://localhost:8081/api/users/me' -H 'Authorization: Bearer <retrieved-access-token>'

When running the quickstart example jar file directly on the host (QUARKUS_HOST_PORT=8081, accordingly) with the same Keycloak container instance, the authorization is working.

Output of uname -a or ver

No response

Output of java -version

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

We observed the problem with one of our own REST services when trying to enable authorization. After some investigation I was able to reproduce the same behavior with the keycloak authorization quickstart example version 2.2.1.Final. I tested with images of Keycloak 14.0.0 and 15.0.2 with the same result.

Hi @vicentini

in

WARN  [org.keycloak.events] (default task-6) type=PERMISSION_TOKEN_ERROR, realmId=quarkus, clientId=backend-service, userId=null, ipAddress=10.12.1.3, error=invalid_token, reason='HTTP 500 Internal Server Error', auth_method=oauth_credentials, audience=backend-service, grant_type=urn:ietf:params:oauth:grant-type:uma-ticket, permission=df1b74a9-3f10-499d-a581-368de48e512b, client_auth_method=client-secret

InternalServerError is happening in Keycloak itself and I'm not aware of the protocol details (what keycloak authorization sends to Keycloak), my guess useId is not related to the problem.
@pedroigor Hi Pedro - can you look into it please ?
Are you aware where this warning may be originating from ?

@vicentini I run your steps with some changes:

  • podman run -d --name keycloak -e KEYCLOAK_USER=admin -e KEYCLOAK_PASSWORD=admin -e DB_VENDOR=H2 -p 8080:8080 quay.io/keycloak/keycloak -b=0.0.0.0
  • podman run -d -e QUARKUS_OIDC_AUTH_SERVER_URL="http://192.168.0.17:8080/auth/realms/quarkus" -e QUARKUS_OIDC_TOKEN_ISSUER="http://192.168.0.17:8080/auth/realms/quarkus" -p 8081:8080 localhost/quarkus/security-keycloak-authorization-jvm

The cUrl command is the same.

In addition to using podman instead of docker (which I think should not change the result at all) I have:

  • I'm not using docker network
  • Changed QUARKUS_OIDC_TOKEN_ISSUER and QUARKUS_OIDC_AUTH_SERVER_URL to not use localhost but a consistent server host (to make sure the issuer is always the same)
  • Using latest Keycloak server image

I could not reproduce the issue by following your steps. Could you please check if any of the highlights above help? Or perhaps I'm still missing something ?

Thanks @pedroigor for having a look at it.

I see what key information is missing or not clear from my original post.

The problem starts, when the service can't access the "public" domain or ip address of the Keycloak server. In a setup where Keycloak and other services are deployed in the same cluster, the services should access Keycloak with the internal ip address and not the public domain/ip.
In most cases we can't access the public address from the inside or if possible at all, it doesn't make sense to route all traffic around and back again.

Therefore I used the additional docker network in my example, where the service directly connects to Keycloak on the internal network (with the corresponding auth-server-url setting) and not from the outside.
With the setup you describe the auth-server-url and the token-issuer have the same (public) url which is indeed working.

@vicentini

The problem starts, when the service can't access the "public" domain or ip address of the Keycloak server.

The exception trace in the issue description is reported at the KC level and you also copied a log message reported by quarkus-keycloak-authorization level after 400 is returned from Keycloak - which implies it is not a connection problem

Sorry, poor choice of words.

When using the internal Keycloak url in a cluster as auth-server-url it's still no connection problem, but I get the log above with the PERMISSION_TOKEN_ERROR.
When using the "public" url as auth-server-url instead, it's working correctly.

@vicentini np, thanks for the clarification.
Is Quarkus endpoint itself configured to require the authentication and/or authorization or you only depend on the realm authorization config directly in Keycloak ? If it is the case, can you try to configure it locally in Quarkus ? I'm not sure it will help though

I used the security-keycloak-authorization-quickstart example without any changes with the provided realm configuration.

In the example all authorization configuration is specified in the 'backend-service' client on the Keycloak server.
The application.properties file is as follows:

# Configuration file
quarkus.oidc.auth-server-url=https://localhost:8543/auth/realms/quarkus
quarkus.oidc.client-id=backend-service
quarkus.oidc.credentials.secret=secret
quarkus.oidc.tls.verification=none
quarkus.http.cors=true

# Enable Policy Enforcement
quarkus.keycloak.policy-enforcer.enable=true
quarkus.keycloak.policy-enforcer.lazy-load-paths=false

# Disables policy enforcement for a path
quarkus.keycloak.policy-enforcer.paths.1.path=/api/public
quarkus.keycloak.policy-enforcer.paths.1.enforcement-mode=DISABLED

So far I didn't have time to test your suggestion to configure everything in quarkus.

After debugging the application (based on the exception below), I wasn't able to see any differences in the parameters passed to KeycloakAdapterPolicyEnforcer.requestAuthorizationToken method other than different frontend/backend urls.

Authorization failed: java.lang.RuntimeException: org.keycloak.authorization.client.util.HttpResponseException: Unexpected response from server: 400 / Bad Request / Response from server: {"error":"unauthorized_client","error_description":"Invalid identity"}
        at org.keycloak.authorization.client.util.Throwables.handleAndWrapHttpResponseException(Throwables.java:99)
        at org.keycloak.authorization.client.util.Throwables.handleWrapException(Throwables.java:42)
        at org.keycloak.authorization.client.util.Throwables.retryAndWrapExceptionIfNecessary(Throwables.java:65)
        at org.keycloak.authorization.client.resource.AuthorizationResource.authorize(AuthorizationResource.java:98)
        at org.keycloak.adapters.authorization.KeycloakAdapterPolicyEnforcer.requestAuthorizationToken(KeycloakAdapterPolicyEnforcer.java:167)
        at org.keycloak.adapters.authorization.KeycloakAdapterPolicyEnforcer.isAuthorized(KeycloakAdapterPolicyEnforcer.java:66)
        at org.keycloak.adapters.authorization.AbstractPolicyEnforcer.authorize(AbstractPolicyEnforcer.java:121)
        at io.quarkus.keycloak.pep.runtime.KeycloakPolicyEnforcerAuthorizer.apply(KeycloakPolicyEnforcerAuthorizer.java:64)
        at io.quarkus.keycloak.pep.runtime.KeycloakPolicyEnforcerAuthorizer.apply(KeycloakPolicyEnforcerAuthorizer.java:28)
        at io.quarkus.vertx.http.runtime.security.HttpAuthorizer$1$1$1.run(HttpAuthorizer.java:72)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$13.runWith(VertxCoreRecorder.java:543)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.keycloak.authorization.client.util.HttpResponseException: Unexpected response from server: 400 / Bad Request / Response from server: {"error":"unauthorized_client","error_description":"Invalid identity"}
        at org.keycloak.authorization.client.util.HttpMethod.execute(HttpMethod.java:95)
        at org.keycloak.authorization.client.util.HttpMethodResponse$2.execute(HttpMethodResponse.java:50)
        at org.keycloak.authorization.client.resource.AuthorizationResource$1.call(AuthorizationResource.java:92)
        at org.keycloak.authorization.client.resource.AuthorizationResource$1.call(AuthorizationResource.java:74)
        at org.keycloak.authorization.client.resource.AuthorizationResource.authorize(AuthorizationResource.java:96)
        ... 13 more

Therefore it's definitely the keycloak server not accepting the given access token, but for me it's not clear why.
Maybe there's some configuration missing in the example or the different urls for frontend/backend access are a problem?

@vicentini unauthorized_client may indicate the the client and its secret have not been recognized.
Might be worth trying to launch Keycloak with KEYCLOAK_FRONTEND_URL system property set to the externally accessible base URL...

@sberyozkin @vicentini Perhaps Sergey is right and the error is related to the issuer mismatch. We lack a better message but you can try to force the frontend URL and see if it helps.

That setting should for Keycloak to issue tokens using the same issuer despite the URL you are using to reach out to the server. By default, the server calculates the issuer based on the host from the request.

@sberyozkin @pedroigor Defining the frontend url is unfortunately not working in this case, because the authorization_endpoint url from the OIDC discovery endpoint is then forced to the frontend url (which can't be accessed from the internal application).

As far as I understand and tested it, it's also no solution to disable OIDC discovery and manually define quarkus.oidc.authorization-path, because the path is only relative to the frontend url and therefore not an option.

Hi @vicentini @pedroigor We've had several issues before with the internal vs external set ups but they were all affecting quarkus-oidc, the way it verifies the tokens, issuers etc, and we have a few way of dealing with such cases.
The important point is that quarkus-keycloak-authorization would not even get the opportunity to call to Keycloak if quarkus-oidc had failed to verify the incoming bearer token.

https://github.com/quarkusio/quarkus/blob/main/extensions/keycloak-authorization/runtime/src/main/java/io/quarkus/keycloak/pep/runtime/KeycloakPolicyEnforcerAuthorizer.java#L44

By the time it is called SecurityIdentity is already created - the flow there is as follows:

  • if it is anonymous (meaning Quarkus itself sees it as an access to the public resource) - check with Keycloak if the path is protected in its realm configuration - if yes - deny
  • next we get AccessTokenCredential
  • If it is null - it means other mechanism but OIDC has created the identity, example, the basic one which can happen when more than mechanism is combined
  • If AccessTokenCredential is not null then it means quarkus-oidc has already verified the token and now quarkus-keycloak-authorization gets its chance to call to Keycloak to do the authorization - so Quarkus itself has already verified the access token at this point of time.
  • 400 is returned from Keycloak in this case - so internal vs external has no impact on connecting to Keycloak.
  • the source of 400 is an internal KC error - at this point it is hard to tell what it can be but @vicentini, since you are already debugging, please try to intercept the request at the Keycloak level and identify the reason behind that 400/internal error - it may point in the end back to Quarkus :-), but you'll have some concrete information about the cause.

My guess is that it might be Keycloak itself which is affected by the internal vs external case when it verifies the token Quarkus posts to it

@pedroigor I had a look at Keycloak debug output and it's like you @sberyozkin stated in the last post. Keycloak doesn't accept the token due to a different issuer.

To sum the situation up a little:
Keycloak and a backend service are deployed in the same cluster. The service has to reach Keycloak via its internal url and is therefore started with quarkus.oidc.auth-server-url = backend-url, and quarkus.oidc.token.issuer = frontend-url.

For Keycloak we now have two options, though for both of them the authorization is failing.

  1. Frontend URL is defined in Keycloak
    As described above, the service is not able to reach the enforced frontend base url of the authorization_endpoint and can't access the authorization resources.

  2. Frontend URL is not defined in Keycloak
    Keycloak doesn't accept the token during authorization due to a mismatch of the issuer. I'll post the debug output below.

As far as I can tell, there's currently no possibility to use Keycloak authorization when deployed on the same cluster using internal urls. I believe it's a conceptual issue which would have to be resolved first.

06:28:08,251 DEBUG [org.keycloak.authentication.AuthenticationProcessor] (default task-4) AUTHENTICATE CLIENT
06:28:08,251 DEBUG [org.keycloak.authentication.ClientAuthenticationFlow] (default task-4) client authenticator: client-secret
06:28:08,251 DEBUG [org.keycloak.authentication.ClientAuthenticationFlow] (default task-4) client authenticator SUCCESS: client-secret
06:28:08,251 DEBUG [org.keycloak.authentication.ClientAuthenticationFlow] (default task-4) Client backend-service authenticated by client-secret
06:28:08,251 DEBUG [org.keycloak.authentication.AuthenticationProcessor] (default task-4) AUTHENTICATE ONLY
06:28:08,251 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-4) processFlow: direct grant
06:28:08,252 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-4) check execution: 'direct-grant-validate-username', requirement: 'REQUIRED'
06:28:08,252 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-4) authenticator: direct-grant-validate-username
06:28:08,252 DEBUG [org.keycloak.authentication.AuthenticationSelectionResolver] (default task-4) Going through the flow 'direct grant' for adding executions
06:28:08,252 DEBUG [org.keycloak.authentication.AuthenticationSelectionResolver] (default task-4) Selections when trying execution 'direct-grant-validate-username' : [ authSelection - direct-grant-validate-username]
06:28:08,252 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-4) invoke authenticator.authenticate: direct-grant-validate-username
06:28:08,255 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-4) authenticator SUCCESS: direct-grant-validate-username
06:28:08,255 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-4) check execution: 'direct-grant-validate-password', requirement: 'REQUIRED'
06:28:08,255 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-4) authenticator: direct-grant-validate-password
06:28:08,257 DEBUG [org.keycloak.authentication.AuthenticationSelectionResolver] (default task-4) Selections when trying execution 'direct-grant-validate-password' : [ authSelection - direct-grant-validate-password]
06:28:08,257 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-4) invoke authenticator.authenticate: direct-grant-validate-password
06:28:08,280 DEBUG [org.keycloak.authentication.DefaultAuthenticationFlow] (default task-4) authenticator SUCCESS: direct-grant-validate-password
06:28:08,282 DEBUG [org.keycloak.services.managers.AuthenticationSessionManager] (default task-4) Removing authSession 'd07788c9-c3e2-4c37-a6f4-50df1c7fcf48'. Expire restart cookie: true
06:28:08,288 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (default task-4) JtaTransactionWrapper  commit
06:28:08,289 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (default task-4) JtaTransactionWrapper end
06:28:08,289 DEBUG [org.keycloak.events] (default task-4) type=LOGIN, realmId=quarkus, clientId=backend-service, userId=eb4123a3-b722-4798-9af5-8957f823657a, ipAddress=10.12.1.1, auth_method=openid-connect, token_id=c03ead56-d66f-438c-aa9b-dbc2bfed3371, grant_type=pas
sword, refresh_token_type=Refresh, scope='email profile', refresh_token_id=ec3fc7fa-c14f-40ca-9830-d16b6cab19f3, client_auth_method=client-secret, username=alice, authSessionParentId=d07788c9-c3e2-4c37-a6f4-50df1c7fcf48, authSessionTabId=1xTyYLD2Opc
06:28:10,427 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-2) new JtaTransactionWrapper
06:28:10,427 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-2) was existing? false
06:28:10,428 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-2) JtaTransactionWrapper  commit
06:28:10,428 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (Timer-2) JtaTransactionWrapper end
06:28:15,182 DEBUG [org.keycloak.authentication.AuthenticationProcessor] (default task-4) AUTHENTICATE CLIENT
06:28:15,182 DEBUG [org.keycloak.authentication.ClientAuthenticationFlow] (default task-4) client authenticator: client-secret
06:28:15,182 DEBUG [org.keycloak.authentication.ClientAuthenticationFlow] (default task-4) client authenticator SUCCESS: client-secret
06:28:15,182 DEBUG [org.keycloak.authentication.ClientAuthenticationFlow] (default task-4) Client backend-service authenticated by client-secret
06:28:15,187 DEBUG [org.keycloak.services.managers.AuthenticationManager] (default task-4) Failed to verify identity token: Invalid token issuer. Expected 'http://keycloak:8080/auth/realms/quarkus', but was 'http://localhost:8080/auth/realms/quarkus'
06:28:15,187 DEBUG [org.keycloak.authorization.authorization.AuthorizationTokenService] (default task-4) PERMISSION_TOKEN_ERROR: org.keycloak.services.ErrorResponseException: HTTP 500 Internal Server Error
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.authorization.common.KeycloakIdentity.<init>(KeycloakIdentity.java:71)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.authorization.common.KeycloakIdentity.<init>(KeycloakIdentity.java:66)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.authorization.authorization.AuthorizationTokenService.lambda$static$0(AuthorizationTokenService.java:132)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.authorization.authorization.AuthorizationTokenService.createEvaluationContext(AuthorizationTokenService.java:467)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.authorization.authorization.AuthorizationTokenService.authorize(AuthorizationTokenService.java:206)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.protocol.oidc.endpoints.TokenEndpoint.permissionGrant(TokenEndpoint.java:970)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.protocol.oidc.endpoints.TokenEndpoint.processGrantRequest(TokenEndpoint.java:208)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:138)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:546)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:435)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:396)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:398)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:365)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:150)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:110)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:141)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:104)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:440)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:138)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:215)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:245)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:61)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
        at javax.servlet.api@2.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
        at org.keycloak.keycloak-wildfly-extensions@15.0.2//org.keycloak.provider.wildfly.WildFlyRequestFilter.lambda$doFilter$0(WildFlyRequestFilter.java:41)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.services.filters.AbstractRequestFilter.filter(AbstractRequestFilter.java:43)
        at org.keycloak.keycloak-wildfly-extensions@15.0.2//org.keycloak.provider.wildfly.WildFlyRequestFilter.doFilter(WildFlyRequestFilter.java:39)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
        at io.undertow.core@2.2.5.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
        at io.undertow.core@2.2.5.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.core@2.2.5.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
        at io.undertow.core@2.2.5.Final//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
        at io.undertow.core@2.2.5.Final//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
        at io.undertow.core@2.2.5.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
        at io.undertow.core@2.2.5.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
        at io.undertow.core@2.2.5.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
        at io.undertow.core@2.2.5.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)
        at io.undertow.core@2.2.5.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
        at io.undertow.core@2.2.5.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:841)
        at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
        at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
        at org.jboss.xnio@3.8.4.Final//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
        at java.base/java.lang.Thread.run(Thread.java:829)

06:28:15,188 DEBUG [org.keycloak.authorization.authorization.AuthorizationTokenService] (default task-4) Error while evaluating permissions: org.keycloak.services.CorsErrorResponseException: HTTP 500 Internal Server Error
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.authorization.authorization.AuthorizationTokenService.lambda$static$0(AuthorizationTokenService.java:135)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.authorization.authorization.AuthorizationTokenService.createEvaluationContext(AuthorizationTokenService.java:467)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.authorization.authorization.AuthorizationTokenService.authorize(AuthorizationTokenService.java:206)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.protocol.oidc.endpoints.TokenEndpoint.permissionGrant(TokenEndpoint.java:970)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.protocol.oidc.endpoints.TokenEndpoint.processGrantRequest(TokenEndpoint.java:208)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:138)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:546)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:435)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:396)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:398)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:365)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:150)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:110)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:141)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:104)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:440)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:138)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:215)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:245)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:61)
        at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
        at javax.servlet.api@2.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
        at org.keycloak.keycloak-wildfly-extensions@15.0.2//org.keycloak.provider.wildfly.WildFlyRequestFilter.lambda$doFilter$0(WildFlyRequestFilter.java:41)
        at org.keycloak.keycloak-services@15.0.2//org.keycloak.services.filters.AbstractRequestFilter.filter(AbstractRequestFilter.java:43)
        at org.keycloak.keycloak-wildfly-extensions@15.0.2//org.keycloak.provider.wildfly.WildFlyRequestFilter.doFilter(WildFlyRequestFilter.java:39)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
        at io.undertow.core@2.2.5.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
        at io.undertow.core@2.2.5.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.core@2.2.5.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
        at io.undertow.core@2.2.5.Final//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
        at io.undertow.core@2.2.5.Final//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
        at io.undertow.core@2.2.5.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
        at io.undertow.core@2.2.5.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
        at io.undertow.core@2.2.5.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
        at io.undertow.core@2.2.5.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
        at org.wildfly.extension.undertow@23.0.2.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)
        at io.undertow.servlet@2.2.5.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)
        at io.undertow.core@2.2.5.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
        at io.undertow.core@2.2.5.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:841)
        at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
        at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
        at org.jboss.xnio@3.8.4.Final//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
        at java.base/java.lang.Thread.run(Thread.java:829)

06:28:15,189 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (default task-4) JtaTransactionWrapper  commit
06:28:15,189 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (default task-4) JtaTransactionWrapper end
06:28:15,189 WARN  [org.keycloak.events] (default task-4) type=PERMISSION_TOKEN_ERROR, realmId=quarkus, clientId=backend-service, userId=null, ipAddress=10.12.1.3, error=invalid_token, reason='HTTP 500 Internal Server Error', auth_method=oauth_credentials, audience=backe
nd-service, grant_type=urn:ietf:params:oauth:grant-type:uma-ticket, permission=df1b74a9-3f10-499d-a581-368de48e512b, client_auth_method=client-secret

@vicentini Thanks for this investigation - can you please open a Keycloak issue and CC there to Pedro @pedroigor ? And link to it here for us to be aware of it. thanks

Thanks @sberyozkin I created a Keycloak issue but couldn't find @pedroigor to cc.

@vicentini Thanks, that is fine, Pedro is CC-ed from this issue so he is aware :-)

@pedroigor Hi Pedro - the same problem can be easily reproduced without Kubernetes, simply start Keycloak server as described in the guide (both HTTP 8180 and HTTPS 8543 ports are available) - then use a curl command to request a token for admin:admin as described in the Testing section - but replace https://localhost:8543 with http://localhost:8180/, and forward the token to the endpoint - you'll get 401 - because the endpoint is configured to work over HTTPS - so it is fixed by adding quarkus.oidc.token.issuer=any, resend the token and you'll see:

16:26:48,991 WARN  [org.keycloak.events] (default task-3) type=PERMISSION_TOKEN_ERROR, realmId=quarkus, clientId=backend-service, userId=null, ipAddress=172.17.0.1, error=invalid_token, reason='HTTP 500 Internal Server Error', auth_method=oauth_credentials, audience=backend-service, grant_type=urn:ietf:params:oauth:grant-type:uma-ticket, permission=7124e2f1-e6dc-44b4-87ab-24b010090b97, client_auth_method=client-secret

@pedroigor Hi Pedro - the same problem can be easily reproduced without the docker, simply start Keycloak server as described in the guide (both HTTP 8180 and HTTPS 8543 ports are available) - then use a curl command to request a token for admin:admin as described in the Testing section - but replace https://localhost:8543 with http://localhost:8180/, and forward the token to the endpoint - you'll get 401 - because the endpoint is configured to work over HTTPS - so it is fixed by adding quarkus.oidc.token.issuer=any, resend the token and you'll see:

16:26:48,991 WARN  [org.keycloak.events] (default task-3) type=PERMISSION_TOKEN_ERROR, realmId=quarkus, clientId=backend-service, userId=null, ipAddress=172.17.0.1, error=invalid_token, reason='HTTP 500 Internal Server Error', auth_method=oauth_credentials, audience=backend-service, grant_type=urn:ietf:params:oauth:grant-type:uma-ticket, permission=7124e2f1-e6dc-44b4-87ab-24b010090b97, client_auth_method=client-secret

I had the same problem and solved it by entering the Realm settings -> General -> Frontend URL
https://<your-container-name>:<port>
for example
https://keycloak:8443

Reason: I issued my tokens with Postman using localhost so the token issuer was localhost. But internally the backend referenced keycloak by container name keycloak, so I got following error

DEBUG [org.keycloak.services.managers.AuthenticationManager] (executor-thread-13) 
Failed to verify identity token: Invalid token issuer. Expected 
'https://keycloak:8443/realms/quarkus', but was 'https://localhost:8443/realms/quarkus'

After changing the frontend url, the token request's base url gets overriden and that fixed my problem.

We are now running into this issue for all of our containerized workloads.

Furthermore, production workloads have a non-customer-facing DNS name for Keycloak (e.g. id.prd-1.workload-name.edge.example.com) and a customer-facing DNS name (e.g. id.example.com. Whereas, cluster-internal micro-services use http://keycloakx-http/auth.

Now this doesn't work when the JWT is issued from, for example, id.example.com. This scenario used to succeed.

Section 4.1.1 of RFC 7519 indicates the iss claim can be a StringOrURI value. Can the behavior be overridden to replace the iss claims with a consistent string regardless of the URL from which the JWT was issued?

@dcnis nailed it with his answer. here is how my compose file looks like:

version: '3.8'
services:

  homelab:
    image: __REGISTRY__/__IMAGE_NAME__:__IMAGE_TAG__
    restart: unless-stopped
    environment:
      QUARKUS_OIDC_AUTH_SERVER_URL: http://auth:8080/realms/__REALM__
      QUARKUS_OIDC_CREDENTIALS_SECRET: __SECRET_FROM_REALM_CLIENT__
      QUARKUS_DATASOURCE_JDBC_URL: jdbc:postgresql://db:5432/db?schema=public
      QUARKUS_OIDC_TOKEN_ISSUER: http://localhost:9080/realms/__REALM__
    ports:
      - 8080:8080
    depends_on:
      - db
      - auth

  db:
    image: postgres:16.1-alpine3.19
    restart: unless-stopped
    ports:
      - 5432:5432
    environment:
      POSTGRES_USER: admin
      POSTGRES_PASSWORD: admin
      POSTGRES_DB: db
    volumes:
      - dbdata:/var/lib/postgresql/data

  auth:
    image: __REGISTRY__/__CUSTOM_KEYCLOAK_IMAGE__:__IMAGE_TAG__
    restart: unless-stopped
    command: start-dev
    ports:
      - 9080:8080
    environment:
      KC_PROXY: none
      KC_HEALTH_ENABLED: false
      KC_METRICS_ENABLED: false
      KC_HOSTNAME: localhost
      KC_HOSTNAME_ADMIN: localhost
      KEYCLOAK_ADMIN: admin
      KEYCLOAK_ADMIN_PASSWORD: admin
      KEYCLOAK_FRONTEND_URL: http://localhost:9080/

  adminer:
    image: adminer:4.8.1
    restart: always
    ports:
      - 8090:8080

volumes:
  dbdata:

with that, I can issue an access token for a user of the realm using the client and secret, then use the token to access protected resources using the policy enforcer.