Potential bug with `protocols=http`
guanzo opened this issue · comments
I'm load testing with getting intermittent Invalid status (504) for bafybeihsxbobenr27qs27jk26nuhzvey3zseukoeasifcaop6vgbpqpa7e. failed to fetch CID: CAR root CID mismatch; all retrievals failed; all retrievals failed; context canceled
due to this line:
lassie/pkg/verifiedcar/verifiedcar.go
Line 81 in e97d544
2nd case is occuring (
cbr.Roots[0] != cfg.Root)
repro'd issue https://github.com/guanzo/lassie-debug. could you take a look? something's weird with protocols=http.
I added this log to the verifiedcar.go file linked above
if cbr.Roots[0] != cfg.Root {
fmt.Println("cbr.Roots[0] != cfg.Root", cbr.Roots[0], cfg.Root)
}grepped output
cbr.Roots[0] != cfg.Root bafybeie7u3mw7nrfbfuxejk472zyuvulhw3z4af4arvwp4jjurf2neoc64 bafybeibwfabpmmlrqo42dsc5hfyye4n2gr74dmh4u5hicaf7oydt2ede4m
cbr.Roots[0] != cfg.Root bafybeie7u3mw7nrfbfuxejk472zyuvulhw3z4af4arvwp4jjurf2neoc64 bafybeifrzjza3axbt6byiqfsacv6xim6md2z46gdexofoocpqmxxcqs5z4
cbr.Roots[0] != cfg.Root bafybeie7u3mw7nrfbfuxejk472zyuvulhw3z4af4arvwp4jjurf2neoc64 bafybeidm6hq6f3ojbzabwyhansos5suzqit4fgwjfaaq3kfc4wzqjaty5u
the first cid ...c64 is repeated. the 2nd cids are all in the repo's cids.txt. every cid in cids.txt is being requested once.
cbr.Roots[0] comes from the http response body. to me it looks like the response is somehow being reused for multiple retrievals.
here's the full grepped output, you can see the cbr.Roots[0] cid is repeated quite often
guanzo:~/workspace/lassie$ GOLOG_LOG_LEVEL=debug lassie daemon --vv -p 8989 2>&1 | grep cbr.Roots
cbr.Roots[0] != cfg.Root bafybeie7u3mw7nrfbfuxejk472zyuvulhw3z4af4arvwp4jjurf2neoc64 bafybeibwfabpmmlrqo42dsc5hfyye4n2gr74dmh4u5hicaf7oydt2ede4m
cbr.Roots[0] != cfg.Root bafybeie7u3mw7nrfbfuxejk472zyuvulhw3z4af4arvwp4jjurf2neoc64 bafybeifrzjza3axbt6byiqfsacv6xim6md2z46gdexofoocpqmxxcqs5z4
cbr.Roots[0] != cfg.Root bafybeie7u3mw7nrfbfuxejk472zyuvulhw3z4af4arvwp4jjurf2neoc64 bafybeidm6hq6f3ojbzabwyhansos5suzqit4fgwjfaaq3kfc4wzqjaty5u
cbr.Roots[0] != cfg.Root bafybeie7u3mw7nrfbfuxejk472zyuvulhw3z4af4arvwp4jjurf2neoc64 bafybeib446vggauz2zwgznbcjwj7z5zilzq3vkvf4jvvjzc4hltlp2synq
cbr.Roots[0] != cfg.Root bafybeie7u3mw7nrfbfuxejk472zyuvulhw3z4af4arvwp4jjurf2neoc64 bafybeigtnqi6ccrz2whizdli6whkpaltdktvyikyzixmz7lhbdh6rcswha
cbr.Roots[0] != cfg.Root bafybeie7u3mw7nrfbfuxejk472zyuvulhw3z4af4arvwp4jjurf2neoc64 bafybeihsxbobenr27qs27jk26nuhzvey3zseukoeasifcaop6vgbpqpa7e
cbr.Roots[0] != cfg.Root bafybeie7u3mw7nrfbfuxejk472zyuvulhw3z4af4arvwp4jjurf2neoc64 bafybeidlocdncawavdfuewulttfurdln25jkt6rwulxc3wfgg6srprqioi
cbr.Roots[0] != cfg.Root bafybeie7u3mw7nrfbfuxejk472zyuvulhw3z4af4arvwp4jjurf2neoc64 bafybeicllscw6qppw6takq4bkcghym3cv6lsayrxbccl7cubc7th6k25pm
cbr.Roots[0] != cfg.Root bafybeie7u3mw7nrfbfuxejk472zyuvulhw3z4af4arvwp4jjurf2neoc64 bafybeidnxyixviba7c4l72kkbzjff73uualkmjuprmkiaiwhugcxlfuatu
cbr.Roots[0] != cfg.Root bafybeidks4faxpnpjgnxc5fezp3bjxw2jxw646qtdhuxuut77gkw5zujxa bafybeigl47zt6f67o6c2a674w4erfdrp3fwcbylyzvz7mjxyglwq2v4o7a
cbr.Roots[0] != cfg.Root bafybeidks4faxpnpjgnxc5fezp3bjxw2jxw646qtdhuxuut77gkw5zujxa bafybeienc34go4u6xqkjn35pqwoseiu5utpidftjl3ptiqunhk6b2tgerq
cbr.Roots[0] != cfg.Root bafybeidks4faxpnpjgnxc5fezp3bjxw2jxw646qtdhuxuut77gkw5zujxa bafybeifcn2aaqynucg7glgfi7lfqbs3n2cixchbfpijjxwucdswodllb7y
cbr.Roots[0] != cfg.Root bafybeidks4faxpnpjgnxc5fezp3bjxw2jxw646qtdhuxuut77gkw5zujxa bafybeiav3rg5olg3konw5q4w2g744ziqwri4duqxb67zbz3r5etgunz7tu
cbr.Roots[0] != cfg.Root bafybeidks4faxpnpjgnxc5fezp3bjxw2jxw646qtdhuxuut77gkw5zujxa bafybeibwcurvhjkfwpdfjx34kg2fuf3gl2nbk4entzkkrm64sh2op7sqre
cbr.Roots[0] != cfg.Root bafybeidks4faxpnpjgnxc5fezp3bjxw2jxw646qtdhuxuut77gkw5zujxa bafybeieaucve5xay72qbeu3sk72uctrniey54vcmqp2nssnarj63ctjidy
cbr.Roots[0] != cfg.Root bafybeidks4faxpnpjgnxc5fezp3bjxw2jxw646qtdhuxuut77gkw5zujxa bafybeidqjldidi2odcrwsyvu23punqe4qracb5dtrcxwp6tzsfccrld634
cbr.Roots[0] != cfg.Root bafybeidks4faxpnpjgnxc5fezp3bjxw2jxw646qtdhuxuut77gkw5zujxa bafybeigwvh4fappcfyufieondevozob4bq5rfdx6nt4dl3b66xw2ne2osm
cbr.Roots[0] != cfg.Root bafybeiguhn6pbxidptd5mw22watgriz2z7l2ne36quynigdnlhy2dmbsjy bafybeif3mh6cxbn5te5aljc43kxcgrt5h6pdsynnwjtnsrroxwpvozkwka
cbr.Roots[0] != cfg.Root bafybeiguhn6pbxidptd5mw22watgriz2z7l2ne36quynigdnlhy2dmbsjy bafybeia4ltfgvrzzop43jtra4id5xb72efnqecy74ce2hocpofeugrjko4
cbr.Roots[0] != cfg.Root bafybeihvl3rokgty3ybm272s47thuvkf5ad2tgx2womis7aemibg4z7kw4 bafybeifqpo2qob2yjh2ljnbwetue4qq6s4dzq2lmsn2qaaaezlz7nbizp4
cbr.Roots[0] != cfg.Root bafybeiguhn6pbxidptd5mw22watgriz2z7l2ne36quynigdnlhy2dmbsjy bafybeiga6hfzvyyktul5tjacdv3tgtoe5nravx7iwntgu7ahtyz5mjfese
cbr.Roots[0] != cfg.Root bafybeiguhn6pbxidptd5mw22watgriz2z7l2ne36quynigdnlhy2dmbsjy bafybeierbifxqifi46rv2dvs2vl2sti5hry4j3cqme2spuzhdekgcl4xpm
cbr.Roots[0] != cfg.Root bafybeiguhn6pbxidptd5mw22watgriz2z7l2ne36quynigdnlhy2dmbsjy bafybeihvl3rokgty3ybm272s47thuvkf5ad2tgx2womis7aemibg4z7kw4
cbr.Roots[0] != cfg.Root bafybeiguhn6pbxidptd5mw22watgriz2z7l2ne36quynigdnlhy2dmbsjy bafybeidr76cd5ltzvyir4sttgig26fzhrcmky7u7fyptqbn5l3s2hkeoke
cbr.Roots[0] != cfg.Root bafybeif2rsbvckfzzikrbajbt3acl73fgxlerar773nldhh5y5idpd6kqe bafybeidjkmeqfp2j6tjszt4oc6aeqdc7gkk7wagisbct6rybjn5icsndfy
cbr.Roots[0] != cfg.Root bafybeif2rsbvckfzzikrbajbt3acl73fgxlerar773nldhh5y5idpd6kqe bafybeifn2omhk3sf4244njgus37aj5yamge5hmjjkzgxodczu2nddvedvy
cbr.Roots[0] != cfg.Root bafybeif2rsbvckfzzikrbajbt3acl73fgxlerar773nldhh5y5idpd6kqe bafybeidvv7kirpl2eb5m7c6kfdovkxrfpvxgut4j6qzfzfyuztqbih2aui
cbr.Roots[0] != cfg.Root bafybeif2rsbvckfzzikrbajbt3acl73fgxlerar773nldhh5y5idpd6kqe bafybeihtlyhfqfu435d6exfxula3rpgttc2gsswqaovqva4fg4x3okewre
cbr.Roots[0] != cfg.Root bafybeif2rsbvckfzzikrbajbt3acl73fgxlerar773nldhh5y5idpd6kqe bafybeihlxwcye24w6mnmdnaajdylnxd3qkkmoqdkdeybc7qeyvfimatffq
cbr.Roots[0] != cfg.Root bafybeif2rsbvckfzzikrbajbt3acl73fgxlerar773nldhh5y5idpd6kqe bafybeifr4nbutklhvjt4rnmeidabdjbyx4ippkyuj4iez6ujqo3z2bucsa
cbr.Roots[0] != cfg.Root bafybeifx7izucm5je5jowwldhxqgsqda3thvoeqcbnyjso2drdvublfgz4 bafybeic47gn5ksa6yng44gbbhlb44gv4d4fjxc25di3zwzw6npfjpgby4q
cbr.Roots[0] != cfg.Root bafybeifx7izucm5je5jowwldhxqgsqda3thvoeqcbnyjso2drdvublfgz4 bafybeid4u5m265vh3bvmpzcvw27hy3xkia6e6beow7knrtzozarekrhlhi
cbr.Roots[0] != cfg.Root bafybeifx7izucm5je5jowwldhxqgsqda3thvoeqcbnyjso2drdvublfgz4 bafybeigwr7bohgzrvormf2dgr5im4cbnghfnjjppmz64afg46re5cqif7y
cbr.Roots[0] != cfg.Root bafybeifx7izucm5je5jowwldhxqgsqda3thvoeqcbnyjso2drdvublfgz4 bafybeib55kdwcipjpu2spaweu56c72u4hwqpnlal2jsnhfq4q3pk2lczka
cbr.Roots[0] != cfg.Root bafybeifx7izucm5je5jowwldhxqgsqda3thvoeqcbnyjso2drdvublfgz4 bafybeibkih7kmrfkxb6ixrzkqqvgfzgy2nljy3o2s3v7gcqywu53i6mpue
cbr.Roots[0] != cfg.Root bafybeid6rvorembmczbay5pmb37v26dbvpdqbyl5bwjstbjzbt5druggx4 bafybeice77ke4kyilropn3hihxx2ilouw3zhbzsof7ytbzhg3gmf55yeki
cbr.Roots[0] != cfg.Root bafybeid6rvorembmczbay5pmb37v26dbvpdqbyl5bwjstbjzbt5druggx4 bafybeie3gqjdwjpxqg7432gasildghjx265clffysgyczztrtb7j2l4n6y
cbr.Roots[0] != cfg.Root bafybeid6rvorembmczbay5pmb37v26dbvpdqbyl5bwjstbjzbt5druggx4 bafybeign6ush3c36czmv75wpb3uw5qpgldvimydhbsqaopyv7eq2ihu7dq
cbr.Roots[0] != cfg.Root bafybeid6rvorembmczbay5pmb37v26dbvpdqbyl5bwjstbjzbt5druggx4 bafybeiek46gcke2npmbkn3aazzmgsxuldjpep6vdjtfnmf6vbswyddsn54
cbr.Roots[0] != cfg.Root bafybeicdutmu2sfpz5yzi7yhvpbi3p3vveleqttq3tv5vqdkk5s3dn4qua bafybeifiimxpho5uf7yg7jyxkjv32c33qfg2rstite3qsduaiyawd36ta4
cbr.Roots[0] != cfg.Root bafybeicdutmu2sfpz5yzi7yhvpbi3p3vveleqttq3tv5vqdkk5s3dn4qua bafybeicfxpkx3m2zsvguovuhgq3qz5hwhpgtvgyhdqefi457k2ruu6fmie
cbr.Roots[0] != cfg.Root bafybeicdutmu2sfpz5yzi7yhvpbi3p3vveleqttq3tv5vqdkk5s3dn4qua bafybeig6xam7xa2g7xmkcfnql3xiqedzg3apzpxwwtgxux5c2brzje6ffe