Job listing breaks after using x509v2 states on salt 3006.1
tazaki opened this issue · comments
Describe the bug
I am testing the new x509v2 saltstates here https://docs.saltproject.io/en/master/ref/states/all/salt.states.x509_v2.html .
It seems like as soon as I issue any certs using states like x509.pem_managed or x509.certificate_managed saltgui only shows "(error)" for anything listing jobs. Inside the api log on debug I get:
2023-07-13 01:55:32,312 [salt.loaded.int.netapi.rest_cherrypy.app:917 ][DEBUG ][46961] Could not serialize the return data from Salt.
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/netapi/rest_cherrypy/app.py", line 913, in hypermedia_handler
response = out(ret)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/json.py", line 137, in dumps
return json_module.dumps(obj, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/json/__init__.py", line 238, in dumps
**kw).encode(obj)
File "/opt/saltstack/salt/lib/python3.10/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/opt/saltstack/salt/lib/python3.10/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/opt/saltstack/salt/lib/python3.10/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable
To Reproduce
- Install saltgui as stated in the readme.
- Follow the example https://docs.saltproject.io/en/master/ref/states/all/salt.states.x509_v2.html
- Once you apply x509.pem_managed or x509.certificate_managed states the jobs list will break in saltgui.
Expected behaviour
The jobs listing not breaking.
Screenshots
Additional context
I'll do more testing this week to try and narrow this down...maybe its just a bug in the new x509v2 code? Figured I'd open it here since its where I noticed the issue, as the states run and generate certs, but seem to break the job cache lists?
Thanks again for this great ui! Please let me know if there is anything I can do or research to help!
I'm sure this error is in salt-api itself.
It would also appear if you use the salt-api with another application (e.g. in a shell script using curl/wget)
salt-api uses a REST interface that has a JSON encoding style.
but internally, saltstack uses Python code which has the ability to use data that cannot be converted to JSON
when that data ends up in a response that triggers the error you see.
We have seen this type of error before in the /stats api while also using setting collect_stats=true.
in that specific case, the return data had a bytes-string as the value for some field.
I think that one was solved btw
can you supply me with a minimal states-file that triggers the error?
can you see which api-call triggers the error? should be visible in the debug-view of the browser in section "network".
Please let me know if there is anything I can do or research to help!
I'm trying to replay this scenario.
So I added all the files from the example.
But the effect does not occur here.
Do you have a (preferably simple) scenario in which it does?
TypeError: Object of type bytes is not JSON serializable
(I was in a hurry this morning...)
that was the same error as I tried to refer to.
Thanks again for this great ui!
THX! passing it also to the original creators/maintainers: @oliverdunk and @maerteijn
I am currently trying to narrow this down to where it starts failing...
Ok found it. Here is the state that is breaking it x509.certificate_managed. Here's my test file:
# -*- coding: utf-8 -*-
# vim: ft=sls
{% from "clusterca/map.jinja" import clusterca with context %}
{%- set clusterca_map = salt['mine.get']('roles:clusterca','minion_ips', tgt_type='pillar') %}
{%- set clusterca_minion_ids = clusterca_map.keys()|list %}
{%- set clusterca_minion_id = clusterca_minion_ids[0] %}
clusterca certs clusterca dir:
file.directory:
- name: {{ clusterca.minion.dir }}
clusterca certs host key:
x509.private_key_managed:
- name: {{ clusterca.minion.dir }}/host.key
- keysize: {{ clusterca.minion.keysize }}
# Build sans list
{%- set sans = [] %}
# Handle dns sans
# Add fqdn
{%- do sans.append('DNS:' + grains['fqdn']) %}
# Add hostname if its different from fqdn
{%- if 'DNS:' + grains['host'] not in sans %}
{%- do sans.append('DNS:' + grains['host']) %}
{%- endif %}
# Add minion_id if it differs from fqdn or hostname
{%- if 'DNS:' + grains['id'] not in sans %}
{%- do sans.append('DNS:' + grains['id']) %}
{%- endif %}
# Handle ip sans
{%- for ip in salt['grains.get']('ipv4', []) %}
{%- do sans.append('IP:' + ip) %}
{%- endfor %}
# States up to here will show status in jobs. As soon as this state runs I get the broken jobs in the ui
clusterca system crt:
x509.certificate_managed:
- name: {{ clusterca.minion.dir }}/host.crt
- ca_server: {{ clusterca_minion_id }}
- signing_policy: server
- private_key: {{ clusterca.minion.dir }}/host.key
- CN: {{ grains['fqdn'] }}
- days_remaining: 30
- subjectAltName: "{{ sans|join(',') }}"
- require:
- file: clusterca certs clusterca dir
- x509: clusterca certs host key
Here is the output of running the state from salt:
[root@saltmaster ~]# sudo salt '*' state.sls clusterca.issue_server_certs
saltmaster:
----------
ID: clusterca certs clusterca dir
Function: file.directory
Name: /opt/clusterca
Result: True
Comment: The directory /opt/clusterca is in the correct state
Started: 20:07:42.988866
Duration: 5.676 ms
Changes:
----------
ID: clusterca certs host key
Function: x509.private_key_managed
Name: /opt/clusterca/host.key
Result: True
Comment: The private key has been created
Started: 20:07:43.050882
Duration: 775.727 ms
Changes:
----------
created:
/opt/clusterca/host.key
----------
ID: clusterca certs host key
Function: file.managed
Name: /opt/clusterca/host.key
Result: True
Comment: File /opt/clusterca/host.key updated
Started: 20:07:43.813562
Duration: 12.7 ms
Changes:
----------
diff:
New file
mode:
0400
----------
ID: clusterca system crt
Function: x509.certificate_managed
Name: /opt/clusterca/host.crt
Result: True
Comment: The certificate has been created
Started: 20:07:43.827296
Duration: 744.215 ms
Changes:
----------
created:
/opt/clusterca/host.crt
----------
ID: clusterca system crt
Function: file.managed
Name: /opt/clusterca/host.crt
Result: True
Comment: File /opt/clusterca/host.crt updated
Started: 20:07:44.558555
Duration: 12.587 ms
Changes:
----------
diff:
New file
Summary for saltmaster
------------
Succeeded: 5 (changed=4)
Failed: 0
------------
Total states run: 5
Total run time: 1.551 s
minion1:
----------
ID: clusterca certs clusterca dir
Function: file.directory
Name: /opt/clusterca
Result: True
Comment: The directory /opt/clusterca is in the correct state
Started: 20:07:42.970988
Duration: 5.669 ms
Changes:
----------
ID: clusterca certs host key
Function: x509.private_key_managed
Name: /opt/clusterca/host.key
Result: True
Comment: The private key has been created
Started: 20:07:43.031800
Duration: 899.863 ms
Changes:
----------
created:
/opt/clusterca/host.key
----------
ID: clusterca certs host key
Function: file.managed
Name: /opt/clusterca/host.key
Result: True
Comment: File /opt/clusterca/host.key updated
Started: 20:07:43.918897
Duration: 12.373 ms
Changes:
----------
diff:
New file
mode:
0400
----------
ID: clusterca system crt
Function: x509.certificate_managed
Name: /opt/clusterca/host.crt
Result: True
Comment: The certificate has been created
Started: 20:07:43.932322
Duration: 820.969 ms
Changes:
----------
created:
/opt/clusterca/host.crt
----------
ID: clusterca system crt
Function: file.managed
Name: /opt/clusterca/host.crt
Result: True
Comment: File /opt/clusterca/host.crt updated
Started: 20:07:44.740729
Duration: 12.224 ms
Changes:
----------
diff:
New file
Summary for minion1
------------
Succeeded: 5 (changed=4)
Failed: 0
------------
Total states run: 5
Total run time: 1.751 s
And me trying to get some some pictures of the gui:
What other files/places can I show to help debug? Thanks again!
Edit - I don't think that gui screen shot shows anything you wanted...looking around firefix dev tools to see if i can find any errors...
FYI...
in the previous occasion that I mentioned, I placed a workaround in file .../netapi/rest_cherrypy/app.py, just before the line response = out(ret)
there, I tested for the URL (/status in that case, probably just / in this case)
and then testing whether specific fields are byte-strings: if type(something) == type(b"")
when true replace it: something = something.decode('ascii', 'replace')
but first just intercept the response and see which fields are affected...
(firefox user here too)
you'll probably find nothing in the browser...
the api call crashed on the server-side
the response code '500' is likely the only thing you see
just click on the '500' line to see the details and look in the 'response' tab
Is this what you meant about "finding nothing in the browser?"
I am currently looking around in the file you mentioned above which for me is
/opt/saltstack/salt/lib/python3.10/site-packages/salt/netapi/rest_cherrypy/app.py
no you need to select the "500" line.
the line with "events?token=" is just the event-stream (see page "Events") that one works fine, it was just interrupted
/opt/saltstack/salt/lib/python3.10/site-packages/salt/netapi/rest_cherrypy/app.py
that is the full filename for me too
Ack sorry. Here's the right one?
HTTP code 500 indicated the the server crashed (here for this request only)
typically that means a python exception was raised while working on the request
but an error the backend-code itself is not that fatal, by failure in the conversion to json is
since that happens only in salt-api (and not in salt commands) the dev-team does not always spot it
Thanks for being patient with me! What are the next steps here? Should I make a ticket for salt-api or?
I can trim down a salt state to trigger this easily if needed. I realized my state was based around having pillar and such already configured.
What are the next steps here? Should I make a ticket for salt-api or?
indeed!
see https://github.com/saltstack/salt/issues
but you need to provide a really simple reproducer with the ticket. something that does not need other sls includefiles or directories like /etc/clusterca, output for one minion only, etc.
I would appreciate it when you post the url of that ticket here.
btw, I will close this ticket after that since there is nothing I can change in SaltGUI that helps against this
I can trim down a salt state to trigger this easily if needed. I realized my state was based around having pillar and such already configured.
that would help me too
since I could not reproduce the effect even with the info from #532 (comment)
Is this the related issue you were referring to? saltstack/salt#59620
I also notice in trace log_level I can see a byte string for the 'public_key' is in bytes in the salt-api logs:
2023-07-14 00:01:26,294 [salt.utils.event :32 ][TRACE ][4236] get_event() received = {'data': {'fun': 'runner.jobs.list_jobs', 'jid': '20230714000125247475', 'user': 'saltuser1', 'fun_args': [], '_stamp': '2023-07-14T00:01:26.285480', 'return': {'20230713235509404634': {'Function': 'saltutil.sync_all', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:09.404634'}, '20230713235510710927': {'Function': 'saltutil.refresh_pillar', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:10.710927'}, '20230713235511310291': {'Function': 'mine.update', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:11.310291'}, '20230713235512168953': {'Function': 'state.sls', 'Arguments': ['mine_hosts', {'__kwarg__': True, 'queue': False, 'concurrent': False}], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:12.168953'}, '20230713235521421624': {'Function': 'grains.items', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'saltuser1', 'StartTime': '2023, Jul 13 23:55:21.421624'}, '20230713235522759582': {'Function': 'saltutil.running', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:22.759582'}, '20230713235523474794': {'Function': 'test.version', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:23.474794'}, '20230713235525637575': {'Function': 'saltutil.running', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:25.637575'}, '20230713235528704888': {'Function': 'saltutil.running', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:28.704888'}, '20230713235550263847': {'Function': 'saltutil.running', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:50.263847'}, '20230713235554012710': {'Function': 'saltutil.running', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:54.012710'}, '20230713235604699929': {'Function': 'grains.items', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'saltuser1', 'StartTime': '2023, Jul 13 23:56:04.699929'}, '20230713235605656092': {'Function': 'test.version', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:56:05.656092'}, '20230713235606169578': {'Function': 'saltutil.running', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:56:06.169578'}, '20230714000028210876': {'Function': 'statels.sls', 'Arguments': ['trigger'], 'Target': 'saltmaster', 'Target-type': 'glob', 'User': 'sudo_root', 'StartTime': '2023, Jul 14 00:00:28.210876'}, '20230714000033355193': {'Function': 'state.sls', 'Arguments': ['trigger'], 'Target': 'saltmaster', 'Target-type': 'glob', 'User': 'sudo_root', 'StartTime': '2023, Jul 14 00:00:33.355193'}, '20230714000035334537': {'Function': 'x509.sign_remote_certificate', 'Arguments': ['server', {'digest': 'sha256', 'signing_private_key': None, 'signing_private_key_passphrase': None, 'signing_cert': None, 'public_key': b'0\x82\x01"0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x01\x05\x00\x03\x82\x01\x0f\x000\x82\x01\n\x02\x82\x01\x01\x00\xc2/\xec\xbf\xe7G\xa0\x8e\xb5m\xda+g\xe9\xd3H\xf4\x89x\xe6nC\x82\x0c(Z\xa0\x92\xd2\x8a\xab\xd8f\xa2\xee\x19C\xdf\x9f\xa8\x99uKw\x18\xfa3!>\x01A\x7fJ*\x9f\x1b\xe9X\xff\x88*D\xd3M\x97\x1d\xado\n+\x0c\xbe\x13\xa8h\xec\xca\xd1"\xbcXF\xb7>\x95\x83I\x1a\x85OoP\x84\xfe\xf9|\xcd\xdd{\xed\xcc\xb3;v\x0fJ\xf1\x93\xe7t\xc0\xf5\xe2\x89\xcfd\x87OCJl+\x15\x9aiVL\xdb.\x06\xa4:\xca!\x12\xde\xb9=\xc1\xba\x08\xf5Gy\x08\x1b\xd6C\xc6\x90\xa8!\x07>\xdf8\x96YQ\xc1\x16\xca]\x9e\x02\xe3\x0b\x16\xa5\xf9:\x93\xaa\xe3\x11X\xfc\x1c\x1f\x92\x06\xf5i5o\x0b@\xd9n)\xd7\x08Y\x1f\x0c\xa3\x9e\x9bLDVH\xcck\x89\xe2\xe0r\xcd5\x8e\x1c\xce\xf8\xb3\xa9\x85\xd6\xb6\xc2\xa6\xa1\x8d+\xabg4;i\xd8\xec\xf27\x81W_\xec\xef\xd9\x11\xc5_~$T\xa5\x84\n5F\xc1\xe40D\xe9\x07\x02\x03\x01\x00\x01', 'csr': None, 'subject': None, 'serial_number': None, 'not_before': None, 'not_after': None, 'days_valid': 365, 'CN': 'saltmaster.vagrant.lan'}, False], 'Target': 'saltmaster', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 14 00:00:35.334537'}, '20230714000038524520': {'Function': 'saltutil.running', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 14 00:00:38.524520'}, '20230714000125993037': {'Function': 'saltutil.running', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 14 00:01:25.993037'}}, 'success': True}, 'tag': 'salt/run/20230714000125247475/ret'}
2023-07-14 00:01:26,294 [salt.transport.ipc:372 ][DEBUG ][4236] Closing IPCMessageSubscriber instance
2023-07-14 00:01:26,295 [salt.loaded.int.netapi.rest_cherrypy.app:917 ][DEBUG ][4236] Could not serialize the return data from Salt.
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/netapi/rest_cherrypy/app.py", line 913, in hypermedia_handler
response = out(ret)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/json.py", line 137, in dumps
return json_module.dumps(obj, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/json/__init__.py", line 238, in dumps
**kw).encode(obj)
File "/opt/saltstack/salt/lib/python3.10/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/opt/saltstack/salt/lib/python3.10/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/opt/saltstack/salt/lib/python3.10/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable
And the state/pillar I am able to trigger this with:
/srv/pillar/x509.sls
x509_signing_policies:
server:
- minions: '*'
- signing_private_key: /etc/pki/clusterca/clusterca.key
- signing_cert: /etc/pki/clusterca/clusterca.crt
- basicConstraints: "critical CA:false"
- keyUsage: "critical keyEncipherment,digitalSignature"
- subjectKeyIdentifier: hash
- authorityKeyIdentifier: keyid,issuer:always
- days_valid: 1095
- copypath: /etc/pki/clusterca/issued_certs/
/srv/salt/trigger.sls
clusterca ca dir:
file.directory:
- name: /etc/pki/clusterca
clusterca ca issued_certs dir:
file.directory:
- name: /etc/pki/clusterca/issued_certs
clusterca ca key file:
x509.private_key_managed:
- name: /etc/pki/clusterca/clusterca.key
- keysize: 2048
- backup: True
- require:
- file: clusterca ca dir
clusterca ca crt file:
x509.certificate_managed:
- name: /etc/pki/clusterca/clusterca.crt
- signing_private_key: /etc/pki/clusterca/clusterca.key
- CN: ca.vagrant.lan
- C: US
- ST: UT
- L: Salt Lake
- basicConstraints: "critical CA:true"
- keyUsage: "critical cRLSign, keyCertSign"
- subjectKeyIdentifier: hash
- authorityKeyIdentifier: keyid:always,issuer
- days_valid: 3650
- days_remaining: 0
- backup: True
- managed_private_key:
name: /etc/pki/clusterca/clusterca.key
keysize: 2048
backup: True
- require:
- file: clusterca ca dir
clusterca certs clusterca dir:
file.directory:
- name: /opt/clusterca
clusterca certs host key:
x509.private_key_managed:
- name: /opt/clusterca/host.key
- keysize: 2048
clusterca system crt:
x509.certificate_managed:
- name: /opt/clusterca/host.crt
- ca_server: {{ grains['id'] }}
- signing_policy: server
- private_key: /opt/clusterca/host.key
- CN: {{ grains['fqdn'] }}
- days_remaining: 30
- require:
- file: clusterca certs clusterca dir
- x509: clusterca certs host key
Run it
[root@saltmaster ~]# sudo salt saltmaster state.sls trigger
saltmaster:
----------
ID: clusterca ca dir
Function: file.directory
Name: /etc/pki/clusterca
Result: True
Comment:
Started: 00:27:30.857020
Duration: 5.807 ms
Changes:
----------
/etc/pki/clusterca:
----------
directory:
new
----------
ID: clusterca ca issued_certs dir
Function: file.directory
Name: /etc/pki/clusterca/issued_certs
Result: True
Comment:
Started: 00:27:30.862976
Duration: 1.201 ms
Changes:
----------
/etc/pki/clusterca/issued_certs:
----------
directory:
new
----------
ID: clusterca ca key file
Function: x509.private_key_managed
Name: /etc/pki/clusterca/clusterca.key
Result: True
Comment: The private key has been created
Started: 00:27:30.948729
Duration: 386.364 ms
Changes:
----------
created:
/etc/pki/clusterca/clusterca.key
----------
ID: clusterca ca key file
Function: file.managed
Name: /etc/pki/clusterca/clusterca.key
Result: True
Comment: File /etc/pki/clusterca/clusterca.key updated
Started: 00:27:31.320030
Duration: 14.692 ms
Changes:
----------
diff:
New file
mode:
0400
----------
ID: clusterca ca crt file
Function: x509.certificate_managed
Name: /etc/pki/clusterca/clusterca.crt
Result: True
Comment: The certificate has been created
Started: 00:27:31.335358
Duration: 371.109 ms
Changes:
----------
created:
/etc/pki/clusterca/clusterca.crt
----------
ID: clusterca ca crt file
Function: file.managed
Name: /etc/pki/clusterca/clusterca.crt
Result: True
Comment: File /etc/pki/clusterca/clusterca.crt updated
Started: 00:27:31.692379
Duration: 13.768 ms
Changes:
----------
diff:
New file
----------
ID: clusterca certs clusterca dir
Function: file.directory
Name: /opt/clusterca
Result: True
Comment:
Started: 00:27:31.706637
Duration: 1.283 ms
Changes:
----------
/opt/clusterca:
----------
directory:
new
----------
ID: clusterca certs host key
Function: x509.private_key_managed
Name: /opt/clusterca/host.key
Result: True
Comment: The private key has been created
Started: 00:27:31.708084
Duration: 407.191 ms
Changes:
----------
created:
/opt/clusterca/host.key
----------
ID: clusterca certs host key
Function: file.managed
Name: /opt/clusterca/host.key
Result: True
Comment: File /opt/clusterca/host.key updated
Started: 00:27:32.101522
Duration: 13.372 ms
Changes:
----------
diff:
New file
mode:
0400
----------
ID: clusterca system crt
Function: x509.certificate_managed
Name: /opt/clusterca/host.crt
Result: True
Comment: The certificate has been created
Started: 00:27:32.115932
Duration: 708.609 ms
Changes:
----------
created:
/opt/clusterca/host.crt
----------
ID: clusterca system crt
Function: file.managed
Name: /opt/clusterca/host.crt
Result: True
Comment: File /opt/clusterca/host.crt updated
Started: 00:27:32.811757
Duration: 12.441 ms
Changes:
----------
diff:
New file
Summary for saltmaster
-------------
Succeeded: 11 (changed=11)
Failed: 0
-------------
Total states run: 11
Total run time: 1.936 s
In theory I should be able to trigger this with a curl command similar to what saltgui is requesting correct?
I think I got it...interestingly enough it works with yaml but not json:
[root@saltmaster ~]# curl -sSKi --insecure http://localhost:3333/run -H 'Accept: application/x-yaml' -d client='runner' -d tgt='saltmaster' -d arg='20230714002732316211' -d fun='jobs.list_job' -d username='saltuser1' -d password='password' -d eauth='pam'
return:
- Arguments:
- server
- CN: saltmaster.vagrant.lan
csr: null
days_valid: 365
digest: sha256
not_after: null
not_before: null
public_key: !!binary |
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwjUUz5Sw2ZeeoMcwtgc0NSDTCwB8JBj2
Bv3vPrtyV+r1FKFGYTEAcDX/C8A3+YyUTS0tbtpFNqUm+fxANR1InTvtiSzRfgC0gYNS67oej/FN
XAcl+1dk59KxPP4SfeZt2AF3pCVOJPRkLYgALN0cj8LiVGt7YRKathCo+wrNUR9BDy7gqNHF99xU
LPLfwc7eFflBRMcEpGGGs4TRBXFtAuMPHuWGCEQuWHiFpS/nnIhy4OK1PoZ/YwDCE+1e/SubAl8f
USjpb2Yt6CIQwpJRwTHyghKGpEEM0zJlnPSQMYjhp8fWFrXXRxe40g32GV0qdOYAQ5aPM1pY7P3B
JRY91wIDAQAB
serial_number: null
signing_cert: null
signing_private_key: null
signing_private_key_passphrase: null
subject: null
- false
Function: x509.sign_remote_certificate
Minions:
- saltmaster
Result:
saltmaster:
retcode: 0
return:
data: !!binary |
MIIDvDCCAqSgAwIBAgIUeRm+YBsEsT1y0gIol0aifBSazIswDQYJKoZIhvcNAQELBQAwRzELMAkG
A1UEBhMCVVMxCzAJBgNVBAgMAlVUMRIwEAYDVQQHDAlTYWx0IExha2UxFzAVBgNVBAMMDmNhLnZh
Z3JhbnQubGFuMB4XDTIzMDcxNDAwMjczMloXDTI2MDcxMzAwMjczMlowITEfMB0GA1UEAwwWc2Fs
dG1hc3Rlci52YWdyYW50LmxhbjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMI1FM+U
sNmXnqDHMLYHNDUg0wsAfCQY9gb97z67clfq9RShRmExAHA1/wvAN/mMlE0tLW7aRTalJvn8QDUd
SJ077Yks0X4AtIGDUuu6Ho/xTVwHJftXZOfSsTz+En3mbdgBd6QlTiT0ZC2IACzdHI/C4lRre2ES
mrYQqPsKzVEfQQ8u4KjRxffcVCzy38HO3hX5QUTHBKRhhrOE0QVxbQLjDx7lhghELlh4haUv55yI
cuDitT6Gf2MAwhPtXv0rmwJfH1Eo6W9mLegiEMKSUcEx8oIShqRBDNMyZZz0kDGI4afH1ha110cX
uNIN9hldKnTmAEOWjzNaWOz9wSUWPdcCAwEAAaOBxTCBwjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB
/wQEAwIFoDAdBgNVHQ4EFgQUKgbVPQajmNm1BQ/3BHX/ON45tfYwgYIGA1UdIwR7MHmAFNNaJ3uJ
Xj5BCM/TiOup93CcB152oUukSTBHMQswCQYDVQQGEwJVUzELMAkGA1UECAwCVVQxEjAQBgNVBAcM
CVNhbHQgTGFrZTEXMBUGA1UEAwwOY2EudmFncmFudC5sYW6CFFixxOEWpgpTlF7kg/W7d+3ezkF6
MA0GCSqGSIb3DQEBCwUAA4IBAQB/OvHNcOfK7sjFRBcW82O+XXhX/P+QbMjIFSDmGP1AKRctQdgb
hCjWwH/sPfA1DcGOc5K0eOvb1Qi7R3BQ65EkuwjSsSTlfm48+HUXVu1i8UGdOiz7GIcT/swYgC1m
ShwyAwND1MF4sL8P3lmFa7IQjmZlKXPiW5CgWzv2ngda+3uQ6r/LeiYEgTNEVOwaBB+yJqtK/QPz
TtnPuUDflqG5eHX8I8XWrywlDFYja4Tmh+QYDWdwQZuI9KgiriXZ8ok0PPktepqHVHXulNSJeh6B
Q6mP8Cbqzg3AQkfqfBdjdS7vkNdI0F0TIzHXrhT+RZHfQskeVC/h32154oohZiNq
errors: []
success: true
StartTime: 2023, Jul 14 00:27:32.316211
Target: saltmaster
Target-type: glob
User: root
jid: '20230714002732316211'
Heres the json that errors
[root@saltmaster ~]# curl -sSKi --insecure http://localhost:3333/run -H 'Content-type: application/json' -d '[{"client": "runner","fun": "jobs.list_job","arg":"20230714002732316211","username":"saltuser1","password":"password","eauth":"pam"}]'
<!DOCTYPE html PUBLIC
"-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
<title>500 Internal Server Error</title>
<style type="text/css">
#powered_by {
margin-top: 20px;
border-top: 2px solid black;
font-style: italic;
}
#traceback {
color: red;
}
</style>
</head>
<body>
<h2>500 Internal Server Error</h2>
<p>Could not serialize the return data from Salt.</p>
<pre id="traceback"></pre>
<div id="powered_by">
<span>
Powered by <a href="http://www.cherrypy.org">CherryPy 18.6.1</a>
</span>
</div>
</body>
</html>
- the rule to create
/etc/pkiis missing... - m2crypto unavailable
for m2crypto, I needed:
- sudo pip install swig
- sudo salt-pip install M2Crypto
even after that, it runs much better, except one state.
not sure what is missing for me here:
ID: clusterca system crt
Function: x509.certificate_managed
Name: /opt/clusterca/host.crt
Result: False
Comment: An exception occurred in this state: Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2385, in call
ret = self.states[cdata["full"]](
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1232, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1247, in _run_as
return _func_or_method(*args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1280, in wrapper
return f(*args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/x509.py", line 675, in certificate_managed
raise salt.exceptions.SaltInvocationError(
salt.exceptions.SaltInvocationError: public_key, signing_private_key, or csr must be specified.
Started: 09:37:53.662131
Duration: 14.986 ms
Changes:
Is this the related issue you were referring to? saltstack/salt#59620
yes, that's the one!
could not find it yesterday because I assumed that I was the submitter...
I was also using the x509v2 Which requires another config... https://docs.saltproject.io/en/latest/ref/modules/all/salt.modules.x509_v2.html
[root@saltmaster ~]# cat /etc/salt/minion.d/x509v2.conf
features:
x509_v2: true
for me most states are successful, except:
ID: clusterca system crt
Function: x509.certificate_managed
Name: /opt/clusterca/host.crt
Result: False
Comment: ca_server did not respond. Salt master must permit peers to call the sign_remote_certificate function.
Started: 16:49:08.192773
Duration: 108.068 ms
Changes:
Man sorry I forgot how much I customized this vagrant setup. Missing the peer config:
[root@saltmaster salt]# cat /etc/salt/master.d/x509_peer.conf
peer:
.*:
- x509.sign_remote_certificate
that changes the output to:
ID: clusterca system crt
Function: x509.certificate_managed
Name: /opt/clusterca/host.crt
Result: False
Comment: ca_server reported errors:
signing_policy must be specified and defined on signing minion
Started: 17:27:49.336941
Duration: 449.101 ms
Changes:
I believe the signing policy comes from pillar data in the /srv/pillar/x509.sls above.
I did have that /srv/pillar/x509.sls file present, and still had the error signing_policy must be specified and defined on signing minion.
I was only trying to reproduce it so that you could submit a consistent/complete issue on the
but it may be sufficient to just show that function x509.sign_remote_certificate returns binary data, as proven by the yaml output.
ah, never mind, I have reproduced the error.
I just investigated the binary content and found that it was never intended to be a regular string.
if this was intended as a regular string, I could have proposed a solution for salt-api where all binary strings are converted to regular strings. that would have been a simple find+replace operation.
but in this case is just is binary blob, so it cannot be converted to text.
I found that the error only occurs in the JSON from salt-api, and not in the JSON from sudo salt-run --out=json jobs.list_jobs.
When you look at that output, you'll see that the error actually is in the ARGUMENTS to that function, and not in the RESULT of that function. Since your statefile did not call this function, it must be a function that is used internally.
oh btw, function jobs.list_jobs never returns any result-data from the listed jobs, so the problem could not have been in the result-data anyway.
I'll still try my workaround in json_out.py, but only to suppress such fields. That may help you and others until the real cause is fixed.
The workaround should go in netapi/rest_cherrypy/app.py (forget about json_out.py)
add the following function (anywhere in that file):
def fix_data_for_json(o, level):
if type(o) is dict:
n = {}
for k,v in o.items():
n[k] = fix_data_for_json(v, level+1)
return n
if type(o) is list:
n = [fix_data_for_json(x, level+1) for x in o]
return n
if type(o) is bytes:
# convert bytes-string into a representation of it
return 'bytes:' + ''.join(format(b, '02x') for b in o)
return o
add the following code in function hypermedia_handler, just before response = out(ret):
# first eliminate all binary fields
# known to exist in /stats
# and in x509.sign_remote_certificate
ret = fix_data_for_json(ret, 1)
Restart both salt-api and salt-master.
The result no longer crashes the salt-api, and thus SaltGUI can present the results.
In this case, the presence of the public_key in the function arguments means that the display on the screen for an individual job is lousy and will need improvement. see #533
Confirmed this fixes the issue: I can see the job listings again. Any suggestions on what we should do? Do we we just need to wait and hope they fix this or?
Also thanks so much! This was a wild ride!
I also realized I can just hide these troubled jobs in the ui via
saltgui_hide_jobs:
- x509.sign_remote_certificate
I also realized I can just hide these troubled jobs in the ui via[...]
that should not be possible...
this parameter works as a selection after the complete job-listing is retrieved.
and by then the error has already occurred
Any suggestions on what we should do? Do we we just need to wait and hope they fix this or?
My suggestion is that you create an issue on https://github.com/saltstack/salt/issues
You should mention the error appears i.c.w. salt-api and indicate that the error appears also when using only salt-api and it is thus unrelated to SaltGUI (if you mention SaltGUI at all).
I'll react on that by mentioning saltstack/salt#59620 and also by proposing a PR with the above fix.
The SaltStack team can then decide to fix the original problem and/or to apply my workaround.
Unfortunately, the team has a huge backlog, causing most solutions to take many months or even years.
there are a few issues in the SaltStack issues database that may be related to this problem:
@tazaki
do you intend to add you own issue there?
I'd like too I just haven't had time to thoroughly post the issue.
since this is no longer strictly related to SaltGUI, I'll close the issue.
@tazaki feel free to add any useful information to this issue as needed.