Concurrent requests with different services, using consul, returns the same service and then 404

Question

Concurrent requests with different services, using consul, returns the same service and then 404

minnocenti901 opened this issue 16 days ago · comments

Max commented 16 days ago

Expected Behavior

Ocelot should request from Consul the correct service for every route

Actual Behavior

If concurrent requests, ocelot uses the same servicename for every concurrent request

Steps to Reproduce the Problem

On consul I have 2 services registered:
ProjectsService with url http://localhost:5013
CustomersService with url http://localhost:5004

Ocelot is running on http://localhost:5280

Global configuration:

  "GlobalConfiguration": {
    "RequestIdKey": "ApiGateway",
    "ServiceDiscoveryProvider": {
      "Type": "Consul",
      "Port": "8502",
      "Host": "192.168.0.204"
    }
  }

Routes configuration:

{
  "UpstreamHttpMethod": ["GET","POST","PUT","DELETE"],
  "UpstreamPathTemplate": "/projects/{all}",
  "DownstreamPathTemplate": "/{all}",
  "ServiceName": "ProjectsService"
},
{
  "UpstreamHttpMethod": ["GET","POST","PUT","DELETE"],
  "UpstreamPathTemplate": "/customers/{all}",
  "DownstreamPathTemplate": "/{all}",
  "ServiceName": "CustomersService"
}

Then, at startup, I make 3 different requests at the same time:

GET http://localhost:5280/projects/api/projects
GET http://localhost:5280/projects/api/industrialSectors
GET http://localhost:5280/customers/api/customers

This is the log in console with LogLevel Debug

dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'ocelot pipeline started'
dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'ocelot pipeline started'
dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'ocelot pipeline started'
dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'Upstream URL path is '/projects/api/projects'.'
dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'Upstream URL path is '/customers/api/customers'.'
dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'Upstream URL path is '/projects/api/industrialSectors'.'
dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'downstream templates are /{all}'
dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'downstream templates are /{all}'
dbug: Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'downstream templates are /{all}'
info: Ocelot.RateLimiting.Middleware.RateLimitingMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'EndpointRateLimiting is not enabled for /{all}'
info: Ocelot.RateLimiting.Middleware.RateLimitingMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'EndpointRateLimiting is not enabled for /{all}'
info: Ocelot.RateLimiting.Middleware.RateLimitingMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'EndpointRateLimiting is not enabled for /{all}'
info: Microsoft.AspNetCore.Authentication.AuthenticationMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: '/customers/api/customers is an authenticated route. AuthenticationMiddleware checking if client is authenticated'
info: Microsoft.AspNetCore.Authentication.AuthenticationMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: '/projects/api/industrialSectors is an authenticated route. AuthenticationMiddleware checking if client is authenticated'
info: Microsoft.AspNetCore.Authentication.AuthenticationMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: '/projects/api/projects is an authenticated route. AuthenticationMiddleware checking if client is authenticated'
info: Microsoft.AspNetCore.Authentication.AuthenticationMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'Client has been authenticated for /customers/api/customers'
info: Microsoft.AspNetCore.Authentication.AuthenticationMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'Client has been authenticated for /projects/api/industrialSectors'
info: Microsoft.AspNetCore.Authentication.AuthenticationMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'Client has been authenticated for /projects/api/projects'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'route is authenticated scopes must be checked'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'route is authenticated scopes must be checked'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'route is authenticated scopes must be checked'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'user scopes is authorized calling next authorization checks'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'user scopes is authorized calling next authorization checks'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'user scopes is authorized calling next authorization checks'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: '/{all} route does not require user to be authorized'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: '/{all} route does not require user to be authorized'
info: Ocelot.Authorization.Middleware.AuthorizationMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: '/{all} route does not require user to be authorized'
info: Ocelot.ServiceDiscovery.ServiceDiscoveryProviderFactory[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'The UseServiceDiscovery mode of the route '^(?i)/projects(?:|/.*)$' is enabled.'
info: Ocelot.ServiceDiscovery.ServiceDiscoveryProviderFactory[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'The UseServiceDiscovery mode of the route '^(?i)/projects(?:|/.*)$' is enabled.'
info: Ocelot.ServiceDiscovery.ServiceDiscoveryProviderFactory[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'The UseServiceDiscovery mode of the route '^(?i)/customers(?:|/.*)$' is enabled.'
info: Ocelot.ServiceDiscovery.ServiceDiscoveryProviderFactory[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'Getting service discovery provider of Type 'Consul'...'
info: Ocelot.ServiceDiscovery.ServiceDiscoveryProviderFactory[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'Getting service discovery provider of Type 'Consul'...'
info: Ocelot.ServiceDiscovery.ServiceDiscoveryProviderFactory[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'Getting service discovery provider of Type 'Consul'...'
dbug: Ocelot.Provider.Consul.Consul[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'Consul Provider: Found total 1 service entries for 'ProjectsService' service.'
dbug: Ocelot.Provider.Consul.Consul[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'Consul Provider: Found total 1 service entries for 'ProjectsService' service.'
dbug: Ocelot.Provider.Consul.Consul[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'Consul Provider: Found total 1 service entries for 'ProjectsService' service.'
dbug: Ocelot.Provider.Consul.Consul[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'Consul Provider: Found total 1 catalog nodes.'
dbug: Ocelot.Provider.Consul.Consul[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'Consul Provider: Found total 1 catalog nodes.'
dbug: Ocelot.Provider.Consul.Consul[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'Consul Provider: Found total 1 catalog nodes.'
dbug: Ocelot.DownstreamUrlCreator.Middleware.DownstreamUrlCreatorMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'Downstream url is http://localhost:5013/api/projects'
dbug: Ocelot.DownstreamUrlCreator.Middleware.DownstreamUrlCreatorMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'Downstream url is http://localhost:5013/api/industrialSectors'
dbug: Ocelot.DownstreamUrlCreator.Middleware.DownstreamUrlCreatorMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'Downstream url is http://localhost:5013/api/customers'
warn: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: '404 (Not Found) status code of request URI: http://localhost:5013/api/customers.'
dbug: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'setting http response message'
dbug: Ocelot.Responder.Middleware.ResponderMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'no pipeline errors, setting and returning completed response'
dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
      requestId: 0HN50O7EO7FED:00000001, previousRequestId: No PreviousRequestId, message: 'ocelot pipeline finished'
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[9]
      Connection id "0HN50O7EO7FED" completed keep alive response.
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5280/customers/api/customers - 404 0 - 1404.2153ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[16]
      Request reached the end of the middleware pipeline without being handled by application code. Request path: GET http://localhost:5280/customers/api/customers, Response status code: 404
info: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: '200 (OK) status code of request URI: http://localhost:5013/api/projects.'
info: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: '200 (OK) status code of request URI: http://localhost:5013/api/industrialSectors.'
dbug: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'setting http response message'
dbug: Ocelot.Requester.Middleware.HttpRequesterMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'setting http response message'
dbug: Ocelot.Responder.Middleware.ResponderMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'no pipeline errors, setting and returning completed response'
dbug: Ocelot.Responder.Middleware.ResponderMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'no pipeline errors, setting and returning completed response'
dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
      requestId: 0HN50O7EO7FEC:00000001, previousRequestId: No PreviousRequestId, message: 'ocelot pipeline finished'
dbug: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware[0]
      requestId: 0HN50O7EO7FEE:00000001, previousRequestId: No PreviousRequestId, message: 'ocelot pipeline finished'
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[9]
      Connection id "0HN50O7EO7FEC" completed keep alive response.
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[9]
      Connection id "0HN50O7EO7FEE" completed keep alive response.
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5280/projects/api/projects - 200 2743 application/json;+charset=utf-8 3878.4309ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5280/projects/api/industrialSectors - 200 339 application/json;+charset=utf-8 3879.4389ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[16]
      Request reached the end of the middleware pipeline without being handled by application code. Request path: GET http://localhost:5280/projects/api/projects, Response status code: 200
info: Microsoft.AspNetCore.Hosting.Diagnostics[16]
      Request reached the end of the middleware pipeline without being handled by application code. Request path: GET http://localhost:5280/projects/api/industrialSectors, Response status code: 200

I'm expecting that for the route /customers it uses CustomersService instead of ProjectsService.

Specifications

Version: Ocelot 23.3.3, Ocelot.Provider.Consul 23.3.3, Consul 1.12.0
Platform: .NET8

Raman Maksimchuk · Answer 1 · Wed Jul 10 2024 20:42:25 GMT+0800 (China Standard Time)

Hello @minnocenti901! What is your full name?
I suspect there is an issue with your project setup, Consul configuration, or networking. Let's work together to identify the root cause.

Ocelot is running on http://localhost:5280

Global configuration:

  "GlobalConfiguration": {
    "RequestIdKey": "ApiGateway",
    "ServiceDiscoveryProvider": {
      "Type": "Consul",
      "Port": "8502",
      "Host": "192.168.0.204"
    }
  }

The BaseUrl property is missing and is required. As outlined in our Getting Started documentation, the BaseUrl must be defined in the Configuration section of the JSON file, as shown below:

  "GlobalConfiguration": {
    "BaseUrl": "http://localhost:5280", // this is crucial!
    "RequestIdKey": "ApiGateway",
    "ServiceDiscoveryProvider": {
      "Type": "Consul",
      "Port": "8502",
      "Host": "192.168.0.204"
    }
  }

Please could you provide the complete ocelot.json file for review?

Raman Maksimchuk · Answer 2 · Wed Jul 10 2024 21:00:12 GMT+0800 (China Standard Time)

Expected Behavior

Ocelot should request from Consul the correct service for every route

Steps to Reproduce the Problem

On consul I have 2 services registered:
ProjectsService with url http://localhost:5013
CustomersService with url http://localhost:5004
Ocelot is running on http://localhost:5280

🆗 Could you show us complete Consul configuration please, service mappings, nodes etc.?

Regarding networking... I see you use learning solution at local machine in Docker or without, right?

Raman Maksimchuk · Answer 3 · Wed Jul 10 2024 21:04:39 GMT+0800 (China Standard Time)

Specifications

Version: Ocelot 23.3.3, Ocelot.Provider.Consul 23.3.3, Consul 1.12.0

Platform: .NET8

🆗 Please revert the version to 23.2.2 and conduct the same tests.

Max · Answer 4 · Wed Jul 10 2024 21:06:00 GMT+0800 (China Standard Time)

Adding BaseUrl didn't solve the issue.
This is the complete ocelot.json

{
  "Routes": [
    {
      "DownstreamPathTemplate": "/{all}",
      "DownstreamScheme": "http",
      "ServiceName": "CustomersService",
      "UpstreamPathTemplate": "/customers/{all}",
      "UpstreamHttpMethod": [
        "GET",
        "POST",
        "PUT",
        "DELETE"
      ],
      "ReRouteIsCaseSensitive": false,
      "AuthenticationOptions": {
        "AuthenticationProviderKey": "Bearer"
      },
      "LoadBalancerOptions": {
        "Type": "LeastConnection"
      }
    },
    {
      "DownstreamPathTemplate": "/{all}",
      "DownstreamScheme": "http",
      "ServiceName": "ProjectsService",
      "UpstreamPathTemplate": "/projects/{all}",
      "UpstreamHttpMethod": [
        "GET",
        "POST",
        "PUT",
        "DELETE"
      ],
      "ReRouteIsCaseSensitive": false,
      "AuthenticationOptions": {
        "AuthenticationProviderKey": "Bearer"
      },
      "LoadBalancerOptions": {
        "Type": "LeastConnection"
      }
    }
  ],
  "GlobalConfiguration": {
    "BaseUrl": "http://localhost:5280",
    "RequestIdKey": "ApiGateway",
    "ServiceDiscoveryProvider": {
      "Type": "Consul",
      "Port": "8502",
      "Host": "192.168.0.204"
    }
  }
}

I'm running it inside visual studio 2022 professional on windows 11. The issue is also present if running it in a docker container on Ubuntu Server 22.04

I forgot to mention that I have upgraded from Ocelot 22.0.1 on .NET6 to latest version 23.3.3 on .NET8. Before the upgrade everything was working perfectly.

Consul is running on docker compose with this .yml:

  consul:
    restart: always
    image: consul:1.12.0
    volumes:
      - ./data/consul:/consul/data
    command: agent -server -ui -node=server-1 -client=0.0.0.0 -bootstrap-expect=1
    ports:
      - "8502:8500"
    logging:
      driver: "json-file"
      options:
          max-size: "50m"

Max · Answer 5 · Wed Jul 10 2024 21:08:08 GMT+0800 (China Standard Time)

Reverting to 23.2.2 fixed the issue

Max · Answer 6 · Wed Jul 10 2024 21:08:37 GMT+0800 (China Standard Time)

Hello @minnocenti901! What is your full name?

Hello! It's Massimiliano Innocenti

Raman Maksimchuk · Answer 7 · Wed Jul 10 2024 21:16:07 GMT+0800 (China Standard Time)

Actual Behavior

If concurrent requests, ocelot uses the same servicename for every concurrent request

Then, at startup, I make 3 different requests at the same time:

GET http://localhost:5280/projects/api/projects

GET http://localhost:5280/projects/api/industrialSectors

GET http://localhost:5280/customers/api/customers

logs...

I'm expecting that for the route /customers it uses CustomersService instead of ProjectsService.

It seems we may have encountered a bug following a significant refactoring of the Consul provider by PR #2067 and commit 34cb3eb

Reverting to 23.2.2 fixed the issue

@minnocenti901 Excellent! Continue using version 23.2.2 until we have resolved or understood the problem.

@ggnaegi, could you please join us and collaborate on this issue? Have you deployed version 23.3.x to Production?

Max · Answer 8 · Wed Jul 10 2024 21:18:11 GMT+0800 (China Standard Time)

@raman-m Thank you for your time!

Raman Maksimchuk · Answer 9 · Wed Jul 10 2024 21:26:07 GMT+0800 (China Standard Time)

@minnocenti901 commented

Hello! It's Massimiliano Innocenti

LoL Massimiliano Innocenti on LinkedIn
Which one could be you? 😄

Max · Answer 10 · Wed Jul 10 2024 21:27:34 GMT+0800 (China Standard Time)

I prefer not to answer :)

Raman Maksimchuk · Answer 11 · Wed Jul 10 2024 21:28:30 GMT+0800 (China Standard Time)

@minnocenti901, are you a C# developer?
If so, would you be willing to collaborate on problem-solving: code researching, debugging, design review and probably open a PR?

Max · Answer 12 · Wed Jul 10 2024 21:30:18 GMT+0800 (China Standard Time)

@minnocenti901, are you a C# developer?

Yes

If so, would you be willing to collaborate on problem-solving: code researching, debugging, design review and probably open a PR?

No, thank you. I don't have time :\ Im sorry

Raman Maksimchuk · Answer 13 · Wed Jul 10 2024 21:32:03 GMT+0800 (China Standard Time)

No, thank you. I don't have time :\ Im sorry

How did you detect the issue? Are you using Ocelot for certain projects?
Regardless, I appreciate you reporting the bug!

Max · Answer 14 · Wed Jul 10 2024 21:40:15 GMT+0800 (China Standard Time)

I use it as main api gateway for a microservices architecture. I need to upgrade from NET6 to NET8 due to upcoming end of service and with the update I had to recheck that everything still works. Then comes the issue

Raman Maksimchuk · Answer 15 · Wed Jul 10 2024 21:43:28 GMT+0800 (China Standard Time)

Understood. For the time being, please proceed with version 23.2.2 for your solution upgrade.

Raman Maksimchuk · Answer 16 · Fri Jul 19 2024 19:14:36 GMT+0800 (China Standard Time)

@minnocenti901 commented Jul 10, 2024, 1:08 PM:

Reverting to 23.2.2 fixed the issue

Good. Did you change anything in the Consul services registrations, or was there nothing except a version increase? It seems the issue was introduced by PR #2067. You are correct; the issue might be related to concurrency. Max, could you please share the JSON from the Consul client? I need the complete list of services registered in Consul.

Max · Answer 17 · Fri Jul 19 2024 19:17:00 GMT+0800 (China Standard Time)

I just increase version of Ocelot. How do I get the JSON you need?