in-toto verify failing using ED25519
KyalSmith opened this issue · comments
I followed the steps in the in-toto/demo as described up to "Verify final product"
and I got the following error when running the command:
in-toto verify --layout root.layout --layout-keys alice.pub
inspection failed: step 'clone' requires '1' link metadata file(s), found '0'
The only way I have diverged from the in-toto/demo is that I generated the bob and carl private and public keys using the following commands:
openssl genpkey -algorithm ed25519 -out bob
openssl pkey -in bob -pubout -out bob.pub
openssl genpkey -algorithm ed25519 -out carl
openssl pkey -in bob -pubout -out carl.pub
I then used the following command to generate the json files required by the create_layout.py:
in-toto key layout bob.pub
in-toto key layout carl.pub
The output of which, I saved as bob_json.pub and carl_json.pub which I then modified the create_layout.py accordingly to reference these files.
I have included the results stored in the final_product directory as a tar.gz file.
final_product.tar.gz
Hi @KyalSmith, are you using in-toto-golang?
That is correct.
Would you prefer me to create the issue there?
https://github.com/in-toto/in-toto-golang
And did you leave the rest of the script as is? You used in-toto-python to create the links, right?
Can you share the versions of the in-toto tools you used? Both the Python and Go implementations have had some recent-ish updates which could be causing these sorts of issues.
I have included the create_layout.py file that I have modified in the final_product.tar.gz.
The only lines I modified are line 12 and 13 regarding the file references and the method "import_ed25519_publickey_from_file".
By the in-toto-python file used to create the links, are you referring to:
"in-toto-record" and "in-toto-run"
Unfortunately Not, I used the golang equivalents:
"in-toto record" and "in-toto run"
The in-toto golang code I got from:
https://github.com/in-toto/in-toto-golang
The branch being "master"
Which I compiled using "make build" and simply copied the in-toto binary to my /usr/bin/.
My current python version is : 3.10.10
and golang is: 1.20.3
Apologies I am using the "main" branch from:
https://github.com/in-toto/demo
Interesting. I think the issue is because of the stdout recorded in byproducts. There's a trailing newline causing an issue when deserializing from JSON. jq is complaining about this as is the in-toto python implementation when I try to verify that file's signature directly (in-toto-sign --verify -f clone.0a88087a.link -k bob_json.pub -t ed25519). The signature on the update-version link verifies fine but the one on package again runs into this issue, and it again has stdout recorded. The backtrace indicates a specific col in the metadata causing the issue which matches the \n character in the string representation of the link.
I'm speculating that the Go implementation is running into a similar issue with the JSON decoding, just that the specific error got suppressed before it could make it back to you. I'm also speculating that we haven't run into this before because this is a relatively recent DSSE specific change in the stack--this character was likely getting escaped when used with cjson. At this point, I'm going to cc @lukpueh in as well for his expertise to confirm this.
Separately, @KyalSmith, could you try generating a set of metadata by removing --use-dsse?
Apologies, its a bit late on this side. Will do it as soon as I get a change tomorrow.
I removed the --use-dse and when running the:
in-toto verify --layout root.layout --layout-keys alice.pub
I got the following error:
inspection failed: artifact verification failed for Inspection 'untar', materials [update-version.0a88087a.link clone.0a88087a.link package.733aeb2b.link] disallowed by rule [DISALLOW *]
I have included the contents of the final_product directory below:
final_product2.tar.gz
Great, that's a known deviation between in-toto-python and in-toto-golang luckily.
When an inspection is executed, link metadata is generated that records all the artifacts in the PWD. The python implementation ignores some by default (https://github.com/in-toto/in-toto/blob/develop/in_toto/settings.py#L33) while the go implementation does not. So if you clean up that directory and verify using in-toto-python, it will pass. Alternatively, modifying the layout to allow the link metadata will also do the trick (ALLOW *.link). We've got some plans to test and squash these deviations between implementations.
As for the DSSE specific issue, I suspect this goes across in-toto implementations so I'm not sure where to open the ticket. @lukpueh wdyt?
Should I modify the create_layout.py to use the following library:
from in_toto.models.metadata import Metablock
instead of:
from in_toto.models.metadata import Envelope ?
The layout can continue to be DSSE, the only issue seems to be when \n is included in the payload. By modify the layout, I meant you can add rules to the inspection to ALLOW *.link before DISALLOW * so that the link metadata for clone, update-version, package etc don't cause an issue.
You wish me to modify the layout file to include the rules ALLOW *.link.
Where would I add that in the following layout file:
{"payload": "eyJfdHlwZSI6ICJsYXlvdXQiLCAiZXhwaXJlcyI6ICIyMDIzLTA2LTIyVDIxOjI0OjIwWiIsICJpbnNwZWN0IjogW3siX3R5cGUiOiAiaW5zcGVjdGlvbiIsICJleHBlY3RlZF9tYXRlcmlhbHMiOiBbWyJNQVRDSCIsICJkZW1vLXByb2plY3QudGFyLmd6IiwgIldJVEgiLCAiUFJPRFVDVFMiLCAiRlJPTSIsICJwYWNrYWdlIl0sIFsiQUxMT1ciLCAiLmtlZXAiXSwgWyJBTExPVyIsICJhbGljZS5wdWIiXSwgWyJBTExPVyIsICJyb290LmxheW91dCJdLCBbIkRJU0FMTE9XIiwgIioiXV0sICJleHBlY3RlZF9wcm9kdWN0cyI6IFtbIk1BVENIIiwgImRlbW8tcHJvamVjdC9mb28ucHkiLCAiV0lUSCIsICJQUk9EVUNUUyIsICJGUk9NIiwgInVwZGF0ZS12ZXJzaW9uIl0sIFsiQUxMT1ciLCAiZGVtby1wcm9qZWN0Ly5naXQvKiJdLCBbIkFMTE9XIiwgImRlbW8tcHJvamVjdC50YXIuZ3oiXSwgWyJBTExPVyIsICIua2VlcCJdLCBbIkFMTE9XIiwgImFsaWNlLnB1YiJdLCBbIkFMTE9XIiwgInJvb3QubGF5b3V0Il0sIFsiRElTQUxMT1ciLCAiKiJdXSwgIm5hbWUiOiAidW50YXIiLCAicnVuIjogWyJ0YXIiLCAieHpmIiwgImRlbW8tcHJvamVjdC50YXIuZ3oiXX1dLCAia2V5cyI6IHsiMGE4ODA4N2ExZjA4NDU1MjQwZTczZTAyMzc2MjA0NWM4MzcwNmQyOGZkMzZhNGU0ZjM1ZmJiMWU5ZWI3ZGU5YyI6IHsia2V5aWQiOiAiMGE4ODA4N2ExZjA4NDU1MjQwZTczZTAyMzc2MjA0NWM4MzcwNmQyOGZkMzZhNGU0ZjM1ZmJiMWU5ZWI3ZGU5YyIsICJrZXlpZF9oYXNoX2FsZ29yaXRobXMiOiBbInNoYTI1NiIsICJzaGE1MTIiXSwgImtleXR5cGUiOiAiZWQyNTUxOSIsICJrZXl2YWwiOiB7InB1YmxpYyI6ICI3OTNmYjRjMzkxMmZiZGJhYTJlMTJlZDE5YWVjNWUyOGMzNGRjNmEwYTJiOWU4ZWUzNjIyOWJmYjM2MmIzMmNmIn0sICJzY2hlbWUiOiAiZWQyNTUxOSJ9LCAiNzMzYWViMmI0NGVhM2ZmY2I4ZTViMjdiNzQ3M2Y2YWFhYjZmYWUxOWExZDJjYmVlYmU5YTllYTExYmE4NTIyYyI6IHsia2V5aWQiOiAiNzMzYWViMmI0NGVhM2ZmY2I4ZTViMjdiNzQ3M2Y2YWFhYjZmYWUxOWExZDJjYmVlYmU5YTllYTExYmE4NTIyYyIsICJrZXlpZF9oYXNoX2FsZ29yaXRobXMiOiBbInNoYTI1NiIsICJzaGE1MTIiXSwgImtleXR5cGUiOiAiZWQyNTUxOSIsICJrZXl2YWwiOiB7InB1YmxpYyI6ICI2NjY3ZTY3OGVhNjFlYzQzYjc2OTk2NGZjZjEyNDgxMzFlNjg2MWRkNjBmMDc5OGE0ZTZiNTFhMzJlYjRhNDZiIn0sICJzY2hlbWUiOiAiZWQyNTUxOSJ9fSwgInJlYWRtZSI6ICIiLCAic3RlcHMiOiBbeyJfdHlwZSI6ICJzdGVwIiwgImV4cGVjdGVkX2NvbW1hbmQiOiBbImdpdCIsICJjbG9uZSIsICJodHRwczovL2dpdGh1Yi5jb20vaW4tdG90by9kZW1vLXByb2plY3QuZ2l0Il0sICJleHBlY3RlZF9tYXRlcmlhbHMiOiBbXSwgImV4cGVjdGVkX3Byb2R1Y3RzIjogW1siQ1JFQVRFIiwgImRlbW8tcHJvamVjdC9mb28ucHkiXSwgWyJESVNBTExPVyIsICIqIl1dLCAibmFtZSI6ICJjbG9uZSIsICJwdWJrZXlzIjogWyIwYTg4MDg3YTFmMDg0NTUyNDBlNzNlMDIzNzYyMDQ1YzgzNzA2ZDI4ZmQzNmE0ZTRmMzVmYmIxZTllYjdkZTljIl0sICJ0aHJlc2hvbGQiOiAxfSwgeyJfdHlwZSI6ICJzdGVwIiwgImV4cGVjdGVkX2NvbW1hbmQiOiBbXSwgImV4cGVjdGVkX21hdGVyaWFscyI6IFtbIk1BVENIIiwgImRlbW8tcHJvamVjdC8qIiwgIldJVEgiLCAiUFJPRFVDVFMiLCAiRlJPTSIsICJjbG9uZSJdLCBbIkRJU0FMTE9XIiwgIioiXV0sICJleHBlY3RlZF9wcm9kdWN0cyI6IFtbIk1PRElGWSIsICJkZW1vLXByb2plY3QvZm9vLnB5Il0sIFsiRElTQUxMT1ciLCAiKiJdXSwgIm5hbWUiOiAidXBkYXRlLXZlcnNpb24iLCAicHVia2V5cyI6IFsiMGE4ODA4N2ExZjA4NDU1MjQwZTczZTAyMzc2MjA0NWM4MzcwNmQyOGZkMzZhNGU0ZjM1ZmJiMWU5ZWI3ZGU5YyJdLCAidGhyZXNob2xkIjogMX0sIHsiX3R5cGUiOiAic3RlcCIsICJleHBlY3RlZF9jb21tYW5kIjogWyJ0YXIiLCAiLS1leGNsdWRlIiwgIi5naXQiLCAiLXpjdmYiLCAiZGVtby1wcm9qZWN0LnRhci5neiIsICJkZW1vLXByb2plY3QiXSwgImV4cGVjdGVkX21hdGVyaWFscyI6IFtbIk1BVENIIiwgImRlbW8tcHJvamVjdC8qIiwgIldJVEgiLCAiUFJPRFVDVFMiLCAiRlJPTSIsICJ1cGRhdGUtdmVyc2lvbiJdLCBbIkRJU0FMTE9XIiwgIioiXV0sICJleHBlY3RlZF9wcm9kdWN0cyI6IFtbIkNSRUFURSIsICJkZW1vLXByb2plY3QudGFyLmd6Il0sIFsiRElTQUxMT1ciLCAiKiJdXSwgIm5hbWUiOiAicGFja2FnZSIsICJwdWJrZXlzIjogWyI3MzNhZWIyYjQ0ZWEzZmZjYjhlNWIyN2I3NDczZjZhYWFiNmZhZTE5YTFkMmNiZWViZTlhOWVhMTFiYTg1MjJjIl0sICJ0aHJlc2hvbGQiOiAxfV19", "payloadType": "application/vnd.in-toto+json", "signatures": [{"keyid": "556caebdc0877eed53d419b60eddb1e57fa773e4e31d70698b588f3e9cc48b35", "sig": "VFM9IbiyJm8GPQMy7KA6cCtTW5ffOOf/oI4dUdZyl9jIu75xdfNxIPDhpXp80RqbsAA+7TFtczy9BJbwR6UgVJFxe+8klfCUvk55sJaSpA3CEICvb+vaRUpXJaugqXwudOrJSH0m7USuHl5GDV/HOUKbS26TdqRepAemx8L0M0UdY29c+Wiw4Pv0vxEV5AjQ7G+qz9RJdfwsJ+cDkBKHlJA40Lld76IB7KX9jRa4wdq8X2BCfvMyWbg8z0r3g41Dkmotpxvga9ywDKZHDm9C6FQoVojjIYox3NgwkC/5PHPoIbA5h7cXBm+B6d8kvXzMduS4n/r5TGpUB1ZFgrjoEFZom4/n4IcwHrTeLh5ISeoIXBiz42QlARB1TVykrDN8kOPB32yg/H0OqIa8rdivUFgupYHX+5I04MJ34OySDdzsFODrHUhWilOJbjI/iyhsiES1sIhnVIAMEkCMDbfvt66mx5NFHv1XaTUpbQxYp4A6TX7Rdc1a6zXIE6RKDSey"}]}
This was generated by the current create_layout.py file.
No, no, the change belongs in https://github.com/in-toto/demo/blob/main/owner_alice/create_layout.py#L70 and https://github.com/in-toto/demo/blob/main/owner_alice/create_layout.py#L79 so that the additional rules are also part of the signed payload. As you can see, there are already other ALLOW rules for the inspection.
I see, thank will try now and see.
Okay so without using the --use-dse argument and modifying the create_layout.py as you specified now returns and exit status of 0.
Thank You.
I'm assuming that DSSE stands for Dynamic Searchable Symmetric Encryption?
I'm wondering what functionality is missing when I do not use the flag.
The Help on in-toto states:
--use-dsse Create metadata using DSSE instead of the legacy signature wrapper.
Could not using this pose a serious security vulnerability?
DSSE is a new signature envelope: https://github.com/secure-systems-lab/dsse
It addresses some concerns that are detailed here: https://github.com/secure-systems-lab/dsse/blob/master/background.md
While there's a bug here, it shouldn't block long term adoption / use of DSSE in in-toto.