Error watching metadata: invalid character '<' looking for beginning of value - Windows server

Question

Error watching metadata: invalid character '<' looking for beginning of value - Windows server

deepak-rxt opened this issue 5 months ago · comments

Getting the below error in all our Windows servers. Any specific version to install to fix the issue?

`2024/01/30 16:13:53 GCEGuestAgent: Error watching metadata: invalid character '<' looking for beginning of value

2024/01/30 16:13:53 GCEGuestAgent: Metadata event watcher failed, ignoring: invalid character '<' looking for beginning of value

2024/01/30 16:13:53 GCEGuestAgent: Metadata event watcher failed, ignoring: invalid character 'M' looking for beginning of value

2024/01/30 16:13:53 GCEGuestAgent: Metadata event watcher failed, ignoring: invalid character 'M' looking for beginning of value`'

There is already a closed issue #320

Chaitanya Kulkarni · Answer 1 · Tue Feb 27 2024 07:06:48 GMT+0800 (China Standard Time)

Hi @deepak-rxt, what Guest Agent and Windows image version you're using?

deepak-rxt · Answer 2 · Tue Feb 27 2024 20:40:32 GMT+0800 (China Standard Time)

Hey @ChaitanyaKulkarni28 ,

Edition Windows Server 2022 Datacenter
Version 21H2
OS build 20348.2113

GCE Agent version 20231004.02

Chaitanya Kulkarni · Answer 3 · Fri Mar 01 2024 03:01:52 GMT+0800 (China Standard Time)

Is the error you're seeing consistent? If not, Guest Agent would retry connecting to MDS.

Also, is there any change in behavior if you reboot your system.

deepak-rxt · Answer 4 · Mon Mar 04 2024 22:24:49 GMT+0800 (China Standard Time)

Nope, errors are not consistent. haven't tried rebooting.

Chaitanya Kulkarni · Answer 5 · Tue Mar 12 2024 08:28:00 GMT+0800 (China Standard Time)

@deepak-rxt are you seeing these errors only on instance startup or at random times?

outbackStack · Answer 6 · Mon Apr 08 2024 17:35:16 GMT+0800 (China Standard Time)

same issues with Debian Server came out of no where.

severity: "ERROR"
sourceLocation: {
  file: "metadata.go"
  function: "github.com/GoogleCloudPlatform/guest-agent/google_guest_agent/events/metadata.(*Watcher).Run"
  line: "68"
}
jsonPayload: {
  localTimestamp: "2024-04-08T19:17:25.3397+10:00"
  message: "Error watching metadata: invalid character '<' looking for beginning of value"
  omitempty: null
}

outbackStack · Answer 7 · Mon Apr 08 2024 17:36:17 GMT+0800 (China Standard Time)

{
  "jsonPayload": {
    "omitempty": null,
    "message": "Metadata event watcher failed, ignoring: invalid character 'M' looking for beginning of value",
    "localTimestamp": "2024-04-08T19:17:25.8373+10:00"
  },
  "sourceLocation": {
    "file": "main.go",
    "line": "225",
    "function": "main.runAgent.func1"
  },
  "receiveTimestamp": "2024-04-08T09:17:26.414441172Z"
}

Chaitanya Kulkarni · Answer 8 · Tue Apr 09 2024 02:48:40 GMT+0800 (China Standard Time)

This issue is fixed in Guest Agent and should be available soon. I will update it here once the new version is released.

Regardless of these error logs, it should not impact any functioning of the Guest Agent at all as agent will retry all transient errors. This is just the mishandling of the error status returned by Metadata server to Guest Agent.

Mark Schewe · Answer 9 · Fri Apr 26 2024 14:42:54 GMT+0800 (China Standard Time)

@ChaitanyaKulkarni28 Is there any news on when the update will be published? Last week (7 days), we received around 28K log messages from this, of which 26 were error level.