Ceseal Container keep restating

[Quang Tuyen]

My node keep crasing and entering this looping :

EBUG RS - deno_runtime::permissions:86 - ⚠️️  Granted read access to "/opt/ceseal/releases/24013112/data/protected_files/runtime-data.seal"
runtime-data.seal exists, no need to handover
Work dir '/opt/ceseal/releases/24013112'
Data dir '/opt/ceseal/releases/24013112/data'
Starting Ceseal with extra opts '--role=full '
Ceseal will running in hardware mode
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)
  - sys.insecure__allow_eventfd = true         (host-based eventfd is enabled)
  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

Running in Gramine-SGX
mr_enclave  : 0x9b97f27980336c73a43719e9f53092051ee7d4df281390b2e9d7451a8b8f8e2a
mr_signer   : 0x8ab66057ac1cbbd49fa645d937008f550c5bd355af21dbea7a73d589df738265
isv_svn     : 0x0000
isv_prod_id : 0x0000
git revision: 55dff8c
2024-02-04T16:41:56.871103Z  INFO main: ceseal: Starting ceseal... sgx=true
2024-02-04T16:41:56.871149Z  INFO main: ceseal: cores=16
2024-02-04T16:41:56.871161Z  INFO main: ceseal: init_args: InitArgs {
    sealing_path: "/data/protected_files",
    storage_path: "/data/storage_files",
    version: "0.1.0",
    git_revision: "55dff8c-2024-01-31T12:48:46.117109259Z",
    enable_checkpoint: true,
    checkpoint_interval: 900,
    remove_corrupted_checkpoint: false,
    max_checkpoint_files: 5,
    cores: 16,
    public_port: None,
    safe_mode_level: 0,
    no_rcu: false,
    ra_timeout: 8s,
    ra_max_retries: 1,
    role: Full,
}
2024-02-04T16:41:56.871904Z  INFO main: cestory: Loading checkpoint from file "/data/storage_files/checkpoint.seal-000037141"
2024-02-04T16:41:58.367709Z  INFO main: cestory: Succeeded to load checkpoint file "/data/storage_files/checkpoint.seal-000037141"
2024-02-04T16:41:58.367751Z  INFO main: cestory: Loaded checkpoint
2024-02-04T16:41:58.367762Z  INFO main: cestory: keyfairy is ready by restore
2024-02-04T16:41:58.367911Z  INFO main: cestory: Enclave init OK
2024-02-04T16:41:58.368250Z  INFO cestory: Ceseal internal server will listening on 0.0.0.0:8000
2024-02-04T16:42:01.370553Z DEBUG cestory: Successfully load podr2 key public key is: [48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 57, 227, 166, 30, 237, 82, 178, 235, 233, 170, 96, 37, 140, 170, 182, 135, 66, 188, 24, 189, 190, 149, 75, 125, 94, 230, 1, 154, 122, 80, 226, 15, 190, 6, 192, 61, 114, 197, 53, 116, 64, 156, 224, 226, 128, 169, 128, 95, 145, 109, 102, 52, 148, 190, 12, 71, 175, 101, 91, 66, 188, 9, 166, 173, 225, 22, 114, 98, 31, 144, 8, 175, 29, 165, 171, 189, 126, 33, 97, 184, 52, 69, 163, 231, 126, 205, 17, 210, 34, 97, 218, 144, 82, 164, 20, 164, 215, 20, 181, 180, 37, 179, 99, 108, 177, 125, 70, 7, 217, 206, 103, 78, 11, 146, 150, 198, 230, 166, 221, 25, 118, 8, 187, 143, 18, 223, 78, 83, 206, 178, 39, 110, 240, 123, 5, 107, 181, 31, 125, 210, 29, 191, 238, 173, 211, 194, 243, 239, 74, 31, 241, 71, 29, 148, 100, 118, 134, 169, 132, 183, 231, 139, 200, 189, 3, 77, 234, 118, 165, 223, 164, 199, 26, 176, 236, 165, 7, 232, 102, 197, 161, 59, 217, 1, 183, 177, 64, 60, 132, 6, 234, 226, 60, 157, 133, 219, 115, 213, 18, 145, 86, 229, 123, 189, 113, 147, 196, 180, 217, 15, 179, 109, 161, 255, 60, 202, 94, 228, 98, 209, 13, 19, 205, 40, 161, 189, 169, 190, 211, 217, 150, 67, 180, 242, 60, 173, 223, 209, 166, 4, 221, 206, 238, 96, 147, 199, 225, 203, 200, 241, 247, 225, 54, 20, 161, 2, 3, 1, 0, 1]
error: There are no available TCS pages left for a new thread. Please try to increase sgx.max_threads in the manifest. The current value is 128
[P1:T2:ceseal] error: PalStreamOpen failed: Operation denied (PAL_ERROR_DENIED)
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }', /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/threadpool-1.8.1/src/lib.rs:777:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: There are no available TCS pages left for a new thread. Please try to increase sgx.max_threads in the manifest. The current value is 128
[P1:T125:ceseal] error: PalStreamOpen failed: Operation denied (PAL_ERROR_DENIED)
[P1:T125:ceseal] error: failed to set up async cleanup_thread (exiting without clear child tid), return code: Cannot allocate memory (ENOMEM)
error: There are no available TCS pages left for a new thread. Please try to increase sgx.max_threads in the manifest. The current value is 128
[P1:T97:ceseal] error: PalStreamOpen failed: Operation denied (PAL_ERROR_DENIED)
[P1:T97:ceseal] error: failed to set up async cleanup_thread (exiting without clear child tid), return code: Cannot allocate memory (ENOMEM)
error: Host injected malicious signal 2
ceseal exited with code 1

[Demos Chiang]

There is a bug in this version of nodeadm, which causes restarting ceseal to fail. I will be solved it when next version release. There is currently a temporary solution if you need......

[Quang Tuyen]

Thanks.. will wait for new version