Does not handle JSON secrets

[Norbert Csibra]

We have a secret in a kv2 engine that has a big JSON object as data.
Exporting this secret with medusa works fine, both in yaml and json format, but when importing back, it tries to create separate secrets from every key/value pair instead of creating just one secret with the whole JSON object.
Plus during this try, it's unable to handle some keys with this error: Error while writing secret. json: unsupported type: importer.RawYaml.

[Jonas Vinther]

Hi @ncsibra

I just tested this, and you are totally right that if you export as json it will mess up everything. I hadn't thought about that scenario. But I just tried to export using the yaml format and it looks like yaml is handling json values more nicely. Can you try and export and import using yaml?

This is my export result when using --format="yaml":

A:
  B:
    json-example: |-
      {
        "Da": "value 1",
        "Db": {
          "DBa": 13,
          "DBb": "value 2"
        }
      }

[Norbert Csibra]

Hi @jonasvinther,

I tried before, but now tried again and it does not work.
Sample json created under "medusa" kv2 secret engine and "cake" secret name through vault GUI:

{
  "id": "0001",
  "type": "donut",
  "name": "Cake",
  "ppu": 0.55,
  "batters": {
    "batter": [
      {
        "id": "1001",
        "type": "Regular"
      },
      {
        "id": "1002",
        "type": "Chocolate"
      },
      {
        "id": "1003",
        "type": "Blueberry"
      },
      {
        "id": "1004",
        "type": "Devil's Food"
      }
    ]
  },
  "topping": [
    {
      "id": "5001",
      "type": "None"
    },
    {
      "id": "5002",
      "type": "Glazed"
    },
    {
      "id": "5005",
      "type": "Sugar"
    },
    {
      "id": "5007",
      "type": "Powdered Sugar"
    },
    {
      "id": "5006",
      "type": "Chocolate with Sprinkles"
    },
    {
      "id": "5003",
      "type": "Chocolate"
    },
    {
      "id": "5004",
      "type": "Maple"
    }
  ]
}

Export: medusa export medusa --format="yaml" -o ~/tmp/test.yml
Result:

cake:
  batters:
    batter:
    - id: "1001"
      type: Regular
    - id: "1002"
      type: Chocolate
    - id: "1003"
      type: Blueberry
    - id: "1004"
      type: Devil's Food
  id: "0001"
  name: Cake
  ppu: 0.55
  topping:
  - id: "5001"
    type: None
  - id: "5002"
    type: Glazed
  - id: "5005"
    type: Sugar
  - id: "5007"
    type: Powdered Sugar
  - id: "5006"
    type: Chocolate with Sprinkles
  - id: "5003"
    type: Chocolate
  - id: "5004"
    type: Maple
  type: donut

Import:

➜  ~ medusa import medusa ~/tmp/test.yml
Error while writing secret. json: unsupported type: importer.RawYaml
Error while writing secret. json: unsupported type: importer.RawYaml

In this case does not import anything to vault.

Environment

Arch Linux under WSL2.

Vault: 1.5.4.
Medusa: 0.2.2. (amd64)
Go: go1.15.8 (linux/amd64)

[Jonas Vinther]

Thank you for the additional information. I'm now able to recreate the issue.
This happens when you use the Vault ui and then use the json interface for creating the secret. If you use the key/value field interface this doesn't happen, and the json value is being stringified by vault.

I need to do some investigation in order to fix this issue. I hope to get time to look into this during this week.

[Jonas Vinther]

Hi @ncsibra
I don't know if you are still hanging around, but I have finally tried to fix this issue.

I have made a pre-release that I hope will solve this issue. I would love if you have time to test it and report back to me?
The release can be found here: https://github.com/jonasvinther/medusa/releases/tag/v0.3.1-pre
I have also added a new test data file, that shows the different types of secret that is handled by this fix. https://github.com/jonasvinther/medusa/blob/handle-json-secrets/test/data/example-different-datatypes.yaml

Also let me know if it's still failing for you guys. Maybe you have some kind of special scenario that is not covered yet.

[Norbert Csibra]

Hi @jonasvinther

I tested it, works fine with our data, imports correctly the whole json.
Thanks for the fix!

[Jonas Vinther]

@ncsibra that sounds awesome! Thank you for testing this. Don't forget to leave a ⭐ if you feel for it :)

[magnuszetterberg]

Hi @jonasvinther !

Thank you for the brilliant patch. It works now for me in 99% of the cases, but I have one edge-case. =(

I have a path with a secret in my vault which has a empty json payload:
path
coolproject/config/datalake/some_secret/config

and this path contains a json payload which looks like this
{}

Thus, an empty config.json.

From your export and then import, this 'path' is not created at all and is then missing after the other data has been imported. As noted, this is a edge-case which I understand might not get fixed by you, but it would be VERY nice if you would take a look at it anyway. =)

Thank you so much for spending time on this during the weekend!!

[Jonas Vinther]

@magnuszetterberg I see that you have found the last piece in the puzzle which is actually already rapported #58. I marked it as a enhancement because I thought that only a few people had empty secrets in their Vault. But now I found two people. So it might be something I should look into real soon. Thank you for letting me know about this.
And thank you for the kind words! You are welcome to ⭐ this project, if you feel for it :)

[magnuszetterberg]

I will absolutely put a star on this project! =)