TimeSinceEpoch is not correct

Question

TimeSinceEpoch is not correct

rickywei opened this issue a year ago · comments

It seems that the time should multiply time.Millisecond (1e6) rather than time.Second (1e9)

The corresponding code in cdproto are https://github.com/chromedp/cdproto/blob/8501c74e8228308e0dbd4aedb1f24ea1c103dd6e/network/types.go#LL1027
and
https://github.com/chromedp/cdproto/blob/8501c74e8228308e0dbd4aedb1f24ea1c103dd6e/cdp/types.go#L742

The following code snapshot is what I tried

package main

import (
	"fmt"
	"time"
)

func main() {
	// a response time from cdp log
	t := 1.685446691349539e+12

	// correct time 2023-05-30 19:38:11.349539072 +0800 CST
	fmt.Println(time.Unix(0, int64(t*float64(time.Millisecond))))

	// wrong time 1677-09-21 08:12:43.145224192 +0800 CST
	fmt.Println(time.Unix(0, int64(t*float64(time.Second))))
}

Kenneth Shaw · Answer 1 · Wed May 31 2023 04:25:52 GMT+0800 (China Standard Time)

@rickywei Thanks for bringing this to my attention. I'll look at this. I don't recall specifically why I implemented this as I did at the time, except to mention that the behavior has at time been incorrect on Chromium's code base, and documented incorrectly.

In the interim, I'd suggest just multiplying the time value if you need the different value at the moment.

Kenneth Shaw · Answer 2 · Fri Jun 02 2023 06:40:59 GMT+0800 (China Standard Time)

@rickywei I've changed this, but I'm not yet going to update this on the chromedp package. In the interim, you can update to the latest cdproto package. I will need to do some testing against the many other places TimeSinceEpoch is used to determine if this is sound/correct, or it's only with the network.Response type where this differs.

Zeke Lu · Answer 3 · Fri Jun 02 2023 07:18:19 GMT+0800 (China Standard Time)

Good morning Ken!

I checked the Network.requestWillBeSent and Network.responseReceived events, only response.responseTime in Network.responseReceived contains the number of milliseconds (which is conflicted with the document).

Network.requestWillBeSent
- wallTime
Network.responseReceived
- response.responseTime
- response.securityDetails.validFrom
- response.securityDetails.validTo

Network.requestWillBeSent (truncated)

{
  "requestId": "4393E3633F10D1113C22E7A368A550C4",
  "loaderId": "4393E3633F10D1113C22E7A368A550C4",
  "documentURL": "https://bing.com/",
  "request": {
    "url": "https://bing.com/",
    "method": "GET"
  },
  "timestamp": 13720.870149,
  "wallTime": 1685659975.297646
}

Network.responseReceived (truncated):

{
  "type": "Document",
  "response": {
    "url": "https://cn.bing.com/",
    "status": 200,
    "statusText": "",
    "responseTime": 1685659975757.947,
    "securityDetails": {
      "protocol": "TLS 1.2",
      "keyExchange": "ECDHE_RSA",
      "keyExchangeGroup": "P-384",
      "cipher": "AES_256_GCM",
      "certificateId": 0,
      "subjectName": "www.bing.com",
      "issuer": "Microsoft RSA TLS CA 02",
      "validFrom": 1676519265,
      "validTo": 1692157665
    }
  }
}

Kenneth Shaw · Answer 4 · Fri Jun 02 2023 07:31:35 GMT+0800 (China Standard Time)

@ZekeLu much appreciated. I haven't had time to go through all the places where there are timestamps in the cdproto code to check everything. Obviously before tagging anything, this should be done, and confirmed. My greatest concern is with the input.* events, as those are likely being used by actual complex programs.

Zeke Lu · Answer 5 · Fri Jun 02 2023 07:34:56 GMT+0800 (China Standard Time)

I will try to go through all of them today.

Kenneth Shaw · Answer 6 · Fri Jun 02 2023 07:38:17 GMT+0800 (China Standard Time)

@ZekeLu you're amazing man. I appreciate the help!

RickyWei · Answer 7 · Sat Jun 03 2023 12:43:16 GMT+0800 (China Standard Time)

@ZekeLu @kenshaw Thanks for all of your effort. I really appreciate it. 👍👍

Zeke Lu · Answer 8 · Sat Jun 03 2023 16:56:17 GMT+0800 (China Standard Time)

Below is the full list of the places where Network.TimeSinceEpoch or Input.TimeSinceEpoch are used. Among them, only Response.responseTime does not carry the correct value (in respect of the CDP doc).

Please note that I'm not familiar with Chromium, and the listed Chromium Implementation could be wrong. But I think it's enough for finding out whether the implementation is consistent with the CDP doc.

Network.TimeSinceEpoch

CDP doc: Network.TimeSinceEpoch
cdproto package: cdp.TimeSinceEpoch

Seq	CDP doc	Chromium Implementation	cdproto package	Chromium
1	Network Response responseTime	`response.ResponseTime().ToJsTimeIgnoringNull()`	network Response ResponseTime	BuildObjectForResourceResponse
2	Network SecurityDetails validFrom validTo	`ssl_info.cert->valid_start().ToDoubleT()` `ssl_info.cert->valid_expiry().ToDoubleT()`	network SecurityDetails ValidFrom ValidTo	BuildSecurityDetails
3	Network requestWillBeSent wallTime	`base::Time::Now().ToDoubleT()`	network EventRequestWillBeSent WallTime	InspectorNetworkAgent:: WillSendRequestInternal
4	Network webSocketWillSendHandshakeRequest wallTime	`base::Time::Now().ToDoubleT()`	network EventWebSocketWillSendHandshakeRequest WallTime	InspectorNetworkAgent:: WillSendWebSocketHandshakeRequest
5	Page OriginTrialToken expiryTime	`blink_trial_token.expiry_time().ToDoubleT()`	cdp OriginTrialToken ExpiryTime	CreateOriginTrialToken
6	Page FrameResource lastModified	`last_modified.value().ToDoubleT()`	page FrameResource LastModified	InspectorPageAgent:: BuildObjectForResourceTree
7	Page ScreencastFrameMetadata timestamp	`base::Time::Now().ToDoubleT()`	page ScreencastFrameMetadata Timestamp	BuildScreencastFrameMetadata
8	Security CertificateSecurityState validFrom validTo	`state.certificate->valid_start().ToDoubleT()` `state.certificate->valid_expiry().ToDoubleT()`	security CertificateSecurityState ValidFrom ValidTo	CreateCertificateSecurityState
9	Storage interestGroupAccessed accessTime	`access_time.ToDoubleT()`	storage EventInterestGroupAccessed AccessTime	StorageHandler:: OnInterestGroupAccessed
10	Storage sharedStorageAccessed accessTime	`access_time.ToDoubleT()`	storage EventSharedStorageAccessed AccessTime	StorageHandler:: NotifySharedStorageAccessed
11	Storage InterestGroupDetails expirationTime	`group.expiry.ToDoubleT()`	storage InterestGroupDetails ExpirationTime	SendGetInterestGroup
418	Storage SharedStorageMetadata creationTime	`metadata.creation_time.ToDoubleT()`	storage SharedStorageMetadata CreationTime	SendSharedStorageMetadata
420	Storage StorageBucketInfo expiration	`bucket.expiration.ToDoubleT()`	storage BucketInfo Expiration	BuildBucketInfo
14	BackgroundService BackgroundServiceEvent Timestamp	`timestamp.ToJsTime() / 1'000`	backgroundservice Event Timestamp	ToBackgroundServiceEvent
15	Network ReportingApiReport timestamp	`(report.queued - base::TimeTicks::UnixEpoch()).InSecondsF())`	network ReportingAPIReport Timestamp	NetworkHandler:: BuildProtocolReport
16	PerformanceTimeline LargestContentfulPaint renderTime loadTime	`ToProtocolTime(timeOrigin, lcp.renderTime())` `ToProtocolTime(timeOrigin, lcp.loadTime())`	performancetimeline LargestContentfulPaint RenderTime LoadTime	BuildEventDetails
17	PerformanceTimeline LayoutShift lastInputTime	`ToProtocolTime(timeOrigin, ls.lastInputTime())`	performancetimeline LayoutShift LastInputTime	BuildEventDetails
18	PerformanceTimeline TimelineEvent time	`ToProtocolTime(timeOrigin, entry->startTime())`	performancetimeline TimelineEvent Time	BuildProtocolEvent
19	Emulation setVirtualTimePolicy initialVirtualTime	`base::Time::FromDoubleT(initial_virtual_time.fromJust())`	emulation SetVirtualTimePolicyParams InitialVirtualTime	InspectorEmulationAgent:: setVirtualTimePolicy
20	Network setCookie expires	`MakeCookieFromProtocolValues`	network SetCookieParams Expires	NetworkHandler:: SetCookie
21	Network CookieParam expires	`MakeCookieFromProtocolValues`	network CookieParam Expires	NetworkHandler:: SetCookies

ToProtocolTime:

double ToProtocolTime(DOMHighResTimeStamp timeOrigin,
                      DOMHighResTimeStamp time) {
  return time ? ConvertDOMHighResTimeStampToSeconds(timeOrigin + time) : 0.0;
}

MakeCookieFromProtocolValues:

  if (expires >= 0) {
    expiration_date =
        expires ? base::Time::FromDoubleT(expires) : base::Time::UnixEpoch();
  }

Input.TimeSinceEpoch

CDP doc: Input.TimeSinceEpoch
cdproto package: input.TimeSinceEpoch

Seq	CDP doc	Chromium Implementation	cdproto package	Chromium
1	Input dispatchKeyEvent timestamp	`GetEventTimeTicks(timestamp)`	input DispatchKeyEventParams Timestamp	InputHandler:: DispatchKeyEvent
2	Input dispatchMouseEvent timestamp	`GetEventTimeTicks(timestamp)`	input DispatchMouseEventParams Timestamp	CreateWebMouseEvent
3	Input dispatchTouchEvent timestamp	`GetEventTimeTicks(timestamp)`	input DispatchTouchEventParams Timestamp	CreateWebTouchEvents
4	Input emulateTouchFromMouseEvent timestamp	`GetEventTimeTicks(timestamp)`	input EmulateTouchFromMouseEventParams Timestamp	InputHandler:: EmulateTouchFromMouseEvent

GetEventTimeTicks

base::TimeTicks GetEventTimeTicks(const Maybe<double>& timestamp) {
  // Convert timestamp, in seconds since unix epoch, to an event timestamp
  // which is time ticks since platform start time.
  return timestamp.isJust() ? base::Seconds(timestamp.fromJust()) +
                                  base::TimeTicks::UnixEpoch()
                            : base::TimeTicks::Now();
}

Zeke Lu · Answer 9 · Sat Jun 03 2023 17:58:16 GMT+0800 (China Standard Time)

We can add some code to correct responseTime for the cdproto users. But since this issue is not noticed for a long time, I think it's OK to leave it as what it is now and just wait for the upstream (Chromium) to fix it. BTW, I have just reported the issue to the Chromium team: https://bugs.chromium.org/p/chromium/issues/detail?id=1451082.

Kenneth Shaw · Answer 10 · Sun Jun 04 2023 05:55:39 GMT+0800 (China Standard Time)

@ZekeLu in your opinion should we revert/change back the Millisecond resolution on pdlgen/cdproto-gen?

Kenneth Shaw · Answer 11 · Sun Jun 04 2023 05:58:03 GMT+0800 (China Standard Time)

@ZekeLu There are other possibilities as well: we can manually mark certain fields in the generator's fixup package, and add an additional field to the TimeSinceEpoch that tells the resolution. Alternately, I can write some code that tries to "guess" what the time is supposed to be, based on how large/small the value is.

Zeke Lu · Answer 12 · Sun Jun 04 2023 11:20:38 GMT+0800 (China Standard Time)

Hi Ken. Yes I think we should revert the Millisecond resolution on pdlgen because the initial implementation is correct. And only responseTime in the Response type is wrong.

Alternately, I can write some code that tries to "guess" what the time is supposed to be, based on how large/small the value is

This is what came up to my mind too.

I think it's better to limit the change to the responseTime field only. But I can not find a easy fix to pdlgen to do this.

Kenneth Shaw · Answer 13 · Mon Jun 05 2023 08:19:23 GMT+0800 (China Standard Time)

I just pushed a commit to cdproto that uses a different type just for the responseTime field (cdp.TimeSinceEpochMilli). It has the same interface as cdp.TimeSinceEpoch, but uses millisecond resolution. Should the Chromium code change, this will need to be change accordingly.

@ZekeLu if you can check this out, assuming all the various time times are correct, we should tag a new version of chromedp with the updated cdproto dependency.

Zeke Lu · Answer 14 · Mon Jun 05 2023 15:21:24 GMT+0800 (China Standard Time)

Thank you @kenshaw ! I have sent a PR to upgrade cdproto. And I have modified an existing test to check the responseTime so that we will be noticed once chromium fixed the issue.

If you're OK with the PR, I will tag a new release as 0.9.2.

Zeke Lu · Answer 15 · Sat Aug 05 2023 11:35:11 GMT+0800 (China Standard Time)

https://bugs.chromium.org/p/chromium/issues/detail?id=1451082 is closed as "WontFix". So it's good to go with cdp.TimeSinceEpochMilli. I'm going to tag a new release.