Using Azure service discovery in K8s containers

[Daniel R. Dagfinrud]

Hello!

I have been attempting to configure the log analytics exporter to use Azure service discovery and detect all workspaces within multiple subscriptions in my Azure tenant.

I am struggling a bit to define this in my containers running in Kubernetes. Do you have any configuration examples of how to run the exporter like this?

I'm getting the following error message when attempting to reach the /probe/subscription endpoint locally:

ERROR: parameter "subscription" is missing

Which makes sense. I have tried defining the subscription list both as environment variables and arguments in the container, but it still fails.

Using the query tester when running the exporter with the binary on my local system - everything works fine. So basically what I can't seem to figure out is how to define the subscription parameter for the Kubernetes container.

Edit one:

Trying to run the container without subscriptions specified in the container itself and defining a prometheus job that scrapes the container with the subscription list gives me the following error in the container log:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x93a1d2]

goroutine 156 [running]:
github.com/webdevops/azure-loganalytics-exporter/loganalytics.(*LogAnalyticsProber).sendQueryToSingleWorkspace(_, _, {_, _}, {{0xc00020b868, {0xc000290780, 0x3, 0x3}, 0x0, {{0x0, ...}, ...}, ...}, ...}, ...)
	/go/src/github.com/webdevops/azure-loganalytics-exporter/loganalytics/prober.go:415 +0x312
github.com/webdevops/azure-loganalytics-exporter/loganalytics.(*LogAnalyticsProber).executeQueries.func1.2()
	/go/src/github.com/webdevops/azure-loganalytics-exporter/loganalytics/prober.go:282 +0xfe
created by github.com/webdevops/azure-loganalytics-exporter/loganalytics.(*LogAnalyticsProber).executeQueries.func1
	/go/src/github.com/webdevops/azure-loganalytics-exporter/loganalytics/prober.go:279 +0x638
	

The container is killed on every scrape.

Edit two:

I realized I forgot to set the queryMode to multi. After configuring the queryMode as multi to query all the discovered workspaces - but I'm getting pretty much the same error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x939807]

goroutine 83 [running]:
github.com/webdevops/azure-loganalytics-exporter/loganalytics.(*LogAnalyticsProber).sendQueryToMultipleWorkspace(_, _, {_, _, _}, {{0x0, {0xc000310000, 0x3, 0x3}, 0x0, ...}, ...}, ...)
	/go/src/github.com/webdevops/azure-loganalytics-exporter/loganalytics/prober.go:359 +0x387
github.com/webdevops/azure-loganalytics-exporter/loganalytics.(*LogAnalyticsProber).executeQueries.func1.1()
	/go/src/github.com/webdevops/azure-loganalytics-exporter/loganalytics/prober.go:264 +0xfe
created by github.com/webdevops/azure-loganalytics-exporter/loganalytics.(*LogAnalyticsProber).executeQueries.func1
	/go/src/github.com/webdevops/azure-loganalytics-exporter/loganalytics/prober.go:261 +0x245

The container/pod is still getting killed.

[Daniel R. Dagfinrud]

@mblaschke Do you have any idea why this occurs?

[Markus Blaschke]

can you try container tag :main (please pull bevor using)?

i've migrated workspace query execution to azure-sdk-for-go but it's not yet tested. i'm currently migrating everything.

[Daniel R. Dagfinrud]

can you try container tag :main (please pull bevor using)?

i've migrated workspace query execution to azure-sdk-for-go but it's not yet tested. i'm currently migrating everything.

Using :main as the image tag seems to resolve the error with the container crashing - thank you 🚀

[Markus Blaschke]

keep in mind that :main is just the main branch, will release a proper version soon.

[Daniel R. Dagfinrud]

keep in mind that :main is just the main branch, will release a proper version soon.

Yeah, I will stay updated and update it when it's released. Thanks for the heads up though 🙏