Unable to create KMS encrypted EBS volumes

Question

Unable to create KMS encrypted EBS volumes

curiositycasualty opened this issue 3 years ago · comments

Summary

I'm hoping I'm just using rexray incorrectly, but I seem to be unable to tease it into creating KMS encrypted EBS volumes. I feel like I'm using it in a fairly vanilla manner. Any aid would be appreciated.

I'm using an Amazon Linux 2 ECS Optimized AMI: ami-06e650acd294da294 where I am installing the rexray/ebs docker plugin as part of the user-date:

    #!/bin/bash

    echo ECS_CLUSTER=dev >> /etc/ecs/ecs.config
    # echo ECS_LOGLEVEL=debug >> /etc/ecs/ecs.config

    sed -i '/After=cloud-final.service/d' /usr/lib/systemd/system/ecs.service
    systemctl daemon-reload

    systemctl restart ecs

    # verify that the agent is running
    until curl -s http://localhost:51678/v1/metadata
    do
      sleep 1
    done

    yum install -y amazon-efs-utils nvme-cli

    udevadm control --reload-rules && udevadm trigger

    # install the Docker volume plugin
    docker plugin install \
      rexray/ebs:0.11.4 \
      REXRAY_PREEMPT=true \
      REXRAY_LOGLEVEL=debug \
      EBS_REGION=us-east-2 \
      --alias=rexray/ebs \
      --grant-all-permissions

    systemctl restart docker
    systemctl restart ecs

My ECS task definition includes my KMS key, passed in as kmsKeyID:

{
      "dockerVolumeConfiguration": {
        "autoprovision": true,
        "labels": {
          "Alias": "cd98441",
          "Environment": "dev",
        },
        "scope": "shared",
        "driver": "rexray/ebs",
        "driverOpts": {
          "volumetype": "gp3",
          "encrypted": "true",
          "size": "5",
          "kmsKeyID": "arn:aws:kms:us-east-2:012345678910:key/abcd1234-a123-456a-a12b-a123b4cd56ef",
          "region": "us-east-2"
        }
      }
}

Here's a more direct example using docker volume create locally on the EC2 host:

docker volume create -d rexray/ebs:0.11.4 --name $(uuidgen) --opt=size=1 --opt=kmsKeyID=arn:aws:kms:us-east-2:012345678910:key/abcd1234-a123-456a-a12b-a123b4cd56ef

Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=info msg=\"Entering go-plugins-helpers capabilitiesPath\" time=1619215546391 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=info msg=\"Entering go-plugins-helpers getPath\" time=1619215546392 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"inspecting volume\" host=\"unix:///var/run/rexray/821099705.sock\" integrationDriver=linux opts=map[] osDriver=linux service=ebs storageDriver=libstorage time=1619215546392 volumeName=61ee6ae6-062e-4cb0-a472-c30fd3301c37 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=info msg=\"inspecting volume\" host=\"unix:///var/run/rexray/821099705.sock\" integrationDriver=linux opts=map[] osDriver=linux service=ebs storageDriver=libstorage time=1619215546392 volumeName=61ee6ae6-062e-4cb0-a472-c30fd3301c37 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"executor using large device range\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" integrationDriver=linux osDriver=linux server=plain-fancier-ke service=ebs storageDriver=libstorage time=1619215546494 txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=warning msg=\"device does not match\" deviceName=name deviceRX=^xvd[f-p]$ host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" integrationDriver=linux osDriver=linux server=plain-fancier-ke service=ebs storageDriver=libstorage time=1619215546500 txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=warning msg=\"device does not match\" deviceName=nvme0n1 deviceRX=^xvd[f-p]$ host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" integrationDriver=linux osDriver=linux server=plain-fancier-ke service=ebs storageDriver=libstorage time=1619215546507 txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=warning msg=\"device does not match\" deviceName=nvme0n1p1 deviceRX=^xvd[f-p]$ host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" integrationDriver=linux osDriver=linux server=plain-fancier-ke service=ebs storageDriver=libstorage time=1619215546509 txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=warning msg=\"device does not match\" deviceName=nvme0n1p128 deviceRX=^xvd[f-p]$ host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" integrationDriver=linux osDriver=linux server=plain-fancier-ke service=ebs storageDriver=libstorage time=1619215546511 txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"xli localdevices success\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" integrationDriver=linux osDriver=linux server=plain-fancier-ke service=ebs storageDriver=libstorage time=1619215546511 txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=info msg=\"http request\" host=\"unix:///var/run/rexray/821099705.sock\" route=volumeInspect server=plain-fancier-ke time=1619215546513 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=schema-validator route=volumeInspect server=plain-fancier-ke time=1619215546513 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=storage-session-handler route=volumeInspect server=plain-fancier-ke time=1619215546514 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=auth-svc-handler route=volumeInspect server=plain-fancier-ke time=1619215546514 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=service-validator route=volumeInspect server=plain-fancier-ke time=1619215546515 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=transaction-handler route=volumeInspect server=plain-fancier-ke time=1619215546515 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=local-devices-handler route=volumeInspect server=plain-fancier-ke time=1619215546515 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=instanceIDs-handler route=volumeInspect server=plain-fancier-ke time=1619215546516 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=auth-global-handler route=volumeInspect server=plain-fancier-ke time=1619215546516 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=error-handler route=volumeInspect server=plain-fancier-ke time=1619215546516 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=transaction-handler route=volumeInspect server=plain-fancier-ke time=1619215546517 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=query-params-handler route=volumeInspect server=plain-fancier-ke time=1619215546517 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/821099705.sock\" key=attachments len(value)=1 route=volumeInspect server=plain-fancier-ke time=1619215546517 tls=false value=[0] " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"query param\" host=\"unix:///var/run/rexray/821099705.sock\" key=byName len(value)=1 route=volumeInspect server=plain-fancier-ke time=1619215546518 tls=false value=[] " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"http header\" Libstorage-Tx=\"txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21, txCR=1619215546\" host=\"unix:///var/run/rexray/821099705.sock\" route=volumeInspect server=plain-fancier-ke time=1619215546519 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"skipping global auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/821099705.sock\" route=volumeInspect server=plain-fancier-ke time=1619215546519 tls=false txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"http header\" Libstorage-Instanceid=[ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2] host=\"unix:///var/run/rexray/821099705.sock\" route=volumeInspect server=plain-fancier-ke time=1619215546519 tls=false txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"http header\" Libstorage-Localdevices=[ebs=] host=\"unix:///var/run/rexray/821099705.sock\" route=volumeInspect server=plain-fancier-ke time=1619215546520 tls=false txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"getting storage service\" host=\"unix:///var/run/rexray/821099705.sock\" route=volumeInspect server=plain-fancier-ke service=ebs time=1619215546521 tls=false txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"skipping svc auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeInspect server=plain-fancier-ke service=ebs storageDriver=ebs time=1619215546521 tls=false txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=info msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"using cached ebs service\" cacheKey=7676e42b666082f9421d859b3a646e92 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeInspect server=plain-fancier-ke service=ebs storageDriver=ebs time=1619215546522 tls=false txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=info msg=\"executing task\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeInspect server=plain-fancier-ke service=ebs storageDriver=ebs task=0 time=1619215546523 tls=false txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"driver is not StorageDriverVolInspectByName\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeInspect server=plain-fancier-ke service=ebs storageDriver=ebs task=0 time=1619215546523 tls=false txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeInspect server=plain-fancier-ke service=ebs storageDriver=ebs time=1619215546527 tls=false txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=error msg=\"resource not found\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" resourceID=61ee6ae6-062e-4cb0-a472-c30fd3301c37 route=volumeInspect server=plain-fancier-ke service=ebs storageDriver=ebs task=0 time=1619215546664 tls=false txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=error msg=\"error: api call failed\" error.resourceID=61ee6ae6-062e-4cb0-a472-c30fd3301c37 host=\"unix:///var/run/rexray/821099705.sock\" route=volumeInspect server=plain-fancier-ke time=1619215546666 tls=false txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"api call error json\" apiErr=\"{\\\"message\\\":\\\"resource not found\\\",\\\"status\\\":404,\\\"error\\\":{\\\"resourceID\\\":\\\"61ee6ae6-062e-4cb0-a472-c30fd3301c37\\\"}}\" host=\"unix:///var/run/rexray/821099705.sock\" route=volumeInspect server=plain-fancier-ke time=1619215546666 tls=false txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"task completed\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeInspect server=plain-fancier-ke service=ebs storageDriver=ebs task=0 time=1619215546664 tls=false txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=error msg=\"docker-legacy: Get: 61ee6ae6-062e-4cb0-a472-c30fd3301c37: failed: resource not found\" host=\"unix:///var/run/rexray/821099705.sock\" integrationDriver=linux osDriver=linux service=ebs storageDriver=libstorage time=1619215546666 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=info msg=\"Entering go-plugins-helpers createPath\" time=1619215546668 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"creating volume\" host=\"unix:///var/run/rexray/821099705.sock\" integrationDriver=linux opts=&{<nil> <nil> 0xc4201e23e0 <nil> <nil> <nil> map[kmskeyid:arn:aws:kms:us-east-2:012345678910:key/abcd1234-a123-456a-a12b-a123b4cd56ef size:1]} osDriver=linux service=ebs storageDriver=libstorage time=1619215546669 volumeName=61ee6ae6-062e-4cb0-a472-c30fd3301c37 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"removing task\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" removedAfter=0s route=volumeInspect server=plain-fancier-ke service=ebs storageDriver=ebs task=0 tasksLen=1 time=1619215546670 tls=false txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"removed task\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeInspect server=plain-fancier-ke service=ebs storageDriver=ebs task=0 tasksLen=0 time=1619215546670 tls=false txCR=1619215546 txID=623ed1d7-a511-43f4-45d7-b88d9ad45d21 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=info msg=\"creating volume\" IOPS=0 availabilityZone= encrypted=<nil> encryptionKey=<nil> host=\"unix:///var/run/rexray/821099705.sock\" integrationDriver=linux opts=&{<nil> <nil> 0xc4201e23e0 <nil> <nil> <nil> map[kmskeyid:arn:aws:kms:us-east-2:012345678910:key/abcd1234-a123-456a-a12b-a123b4cd56ef size:1]} osDriver=linux service=ebs size=1 storageDriver=libstorage time=1619215546743 volumeName=61ee6ae6-062e-4cb0-a472-c30fd3301c37 volumeType= " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"executor using large device range\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" integrationDriver=linux osDriver=linux server=plain-fancier-ke service=ebs storageDriver=libstorage time=1619215546878 txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=warning msg=\"device does not match\" deviceName=name deviceRX=^xvd[f-p]$ host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" integrationDriver=linux osDriver=linux server=plain-fancier-ke service=ebs storageDriver=libstorage time=1619215546887 txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=warning msg=\"device does not match\" deviceName=nvme0n1 deviceRX=^xvd[f-p]$ host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" integrationDriver=linux osDriver=linux server=plain-fancier-ke service=ebs storageDriver=libstorage time=1619215546889 txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=warning msg=\"device does not match\" deviceName=nvme0n1p1 deviceRX=^xvd[f-p]$ host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" integrationDriver=linux osDriver=linux server=plain-fancier-ke service=ebs storageDriver=libstorage time=1619215546891 txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=warning msg=\"device does not match\" deviceName=nvme0n1p128 deviceRX=^xvd[f-p]$ host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" integrationDriver=linux osDriver=linux server=plain-fancier-ke service=ebs storageDriver=libstorage time=1619215546893 txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"xli localdevices success\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" integrationDriver=linux osDriver=linux server=plain-fancier-ke service=ebs storageDriver=libstorage time=1619215546893 txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=info msg=\"http request\" host=\"unix:///var/run/rexray/821099705.sock\" route=volumeCreate server=plain-fancier-ke time=1619215546894 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=post-args-handler route=volumeCreate server=plain-fancier-ke time=1619215546894 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=schema-validator route=volumeCreate server=plain-fancier-ke time=1619215546894 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=storage-session-handler route=volumeCreate server=plain-fancier-ke time=1619215546895 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=auth-svc-handler route=volumeCreate server=plain-fancier-ke time=1619215546895 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added route middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=service-validator route=volumeCreate server=plain-fancier-ke time=1619215546895 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=transaction-handler route=volumeCreate server=plain-fancier-ke time=1619215546895 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=local-devices-handler route=volumeCreate server=plain-fancier-ke time=1619215546895 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=instanceIDs-handler route=volumeCreate server=plain-fancier-ke time=1619215546895 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=auth-global-handler route=volumeCreate server=plain-fancier-ke time=1619215546895 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=error-handler route=volumeCreate server=plain-fancier-ke time=1619215546895 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=transaction-handler route=volumeCreate server=plain-fancier-ke time=1619215546895 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"added global middleware\" host=\"unix:///var/run/rexray/821099705.sock\" middleware=query-params-handler route=volumeCreate server=plain-fancier-ke time=1619215546896 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"http header\" Libstorage-Tx=\"txID=b8b75546-874e-417b-7d6e-0b034489a502, txCR=1619215546\" host=\"unix:///var/run/rexray/821099705.sock\" route=volumeCreate server=plain-fancier-ke time=1619215546896 tls=false " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"skipping global auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/821099705.sock\" route=volumeCreate server=plain-fancier-ke time=1619215546896 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"http header\" Libstorage-Instanceid=[ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2] host=\"unix:///var/run/rexray/821099705.sock\" route=volumeCreate server=plain-fancier-ke time=1619215546896 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"http header\" Libstorage-Localdevices=[ebs=] host=\"unix:///var/run/rexray/821099705.sock\" route=volumeCreate server=plain-fancier-ke time=1619215546896 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"getting storage service\" host=\"unix:///var/run/rexray/821099705.sock\" route=volumeCreate server=plain-fancier-ke service=ebs time=1619215546896 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"skipping svc auth handler; empty allow & deny lists\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeCreate server=plain-fancier-ke service=ebs storageDriver=ebs time=1619215546897 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"validating schema\" body=\"{\\\"name\\\":\\\"61ee6ae6-062e-4cb0-a472-c30fd3301c37\\\",\\\"availabilityZone\\\":\\\"\\\",\\\"iops\\\":0,\\\"size\\\":1,\\\"type\\\":\\\"\\\",\\\"opts\\\":{\\\"kmskeyid\\\":\\\"arn:aws:kms:us-east-2:012345678910:key/abcd1234-a123-456a-a12b-a123b4cd56ef\\\",\\\"size\\\":\\\"1\\\"}}\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeCreate schema=\"{\\n    \\\"$schema\\\": \\\"http://json-schema.org/draft-04/schema#\\\",\\n    \\\"$ref\\\": \\\"https://github.com/rexray/libstorage#/definitions/volumeCreateRequest\\\"\\n}\" server=plain-fancier-ke service=ebs storageDriver=ebs time=1619215546897 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=info msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"using cached ebs service\" cacheKey=7676e42b666082f9421d859b3a646e92 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeCreate server=plain-fancier-ke service=ebs storageDriver=ebs time=1619215546908 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=info msg=\"executing task\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeCreate server=plain-fancier-ke service=ebs storageDriver=ebs task=0 time=1619215546909 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"creating volume\" availabilityZone=0xc420246b80 host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" iops=0xc420246b88 route=volumeCreate server=plain-fancier-ke service=ebs size=0xc420246b90 storageDriver=ebs task=0 time=1619215546909 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 type=0xc420246b98 volumeName=61ee6ae6-062e-4cb0-a472-c30fd3301c37 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=debug msg=\"getting task service\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeCreate server=plain-fancier-ke service=ebs storageDriver=ebs time=1619215546911 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:47 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:47Z" level=error msg="time=\"2021-04-23T22:05:47Z\" level=debug msg=\"still waiting for action\" action=create host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeCreate server=plain-fancier-ke service=ebs storageDriver=ebs task=0 time=1619215547724 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:47 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:47Z" level=error msg="time=\"2021-04-23T22:05:47Z\" level=debug msg=\"still waiting for action\" action=create host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeCreate server=plain-fancier-ke service=ebs storageDriver=ebs task=0 time=1619215547877 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:48 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:48Z" level=error msg="time=\"2021-04-23T22:05:48Z\" level=debug msg=\"still waiting for action\" action=create host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeCreate server=plain-fancier-ke service=ebs storageDriver=ebs task=0 time=1619215548130 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:48 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:48Z" level=error msg="time=\"2021-04-23T22:05:48Z\" level=debug msg=\"still waiting for action\" action=create host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeCreate server=plain-fancier-ke service=ebs storageDriver=ebs task=0 time=1619215548580 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:49 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:49Z" level=error msg="time=\"2021-04-23T22:05:49Z\" level=debug msg=\"still waiting for action\" action=create host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeCreate server=plain-fancier-ke service=ebs storageDriver=ebs task=0 time=1619215549431 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:51 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:51Z" level=error msg="time=\"2021-04-23T22:05:51Z\" level=debug msg=\"success creating volume\" availabilityZone=0xc420246b80 host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" iops=0xc420246b88 route=volumeCreate server=plain-fancier-ke service=ebs size=0xc420246b90 storageDriver=ebs task=0 time=1619215551164 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 type=0xc420246b98 volumeName=61ee6ae6-062e-4cb0-a472-c30fd3301c37 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:51 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:51Z" level=error msg="time=\"2021-04-23T22:05:51Z\" level=debug msg=\"skipping response schema validation; disabled\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeCreate server=plain-fancier-ke service=ebs storageDriver=ebs task=0 time=1619215551164 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:51 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:51Z" level=error msg="time=\"2021-04-23T22:05:51Z\" level=debug msg=\"task completed\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeCreate server=plain-fancier-ke service=ebs storageDriver=ebs task=0 time=1619215551164 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:51 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:51Z" level=error msg="time=\"2021-04-23T22:05:51Z\" level=info msg=\"volume created\" host=\"unix:///var/run/rexray/821099705.sock\" integrationDriver=linux osDriver=linux service=ebs storageDriver=libstorage time=1619215551168 vol=&{[] available us-east-2b false 100 61ee6ae6-062e-4cb0-a472-c30fd3301c37  1 available vol-0e8355cf00cedd807 gp2 map[]} volumeName=61ee6ae6-062e-4cb0-a472-c30fd3301c37 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
61ee6ae6-062e-4cb0-a472-c30fd3301c37
Apr 23 22:05:51 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:51Z" level=error msg="time=\"2021-04-23T22:05:51Z\" level=debug msg=\"removing task\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" removedAfter=0s route=volumeCreate server=plain-fancier-ke service=ebs storageDriver=ebs task=0 tasksLen=1 time=1619215551169 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16
Apr 23 22:05:51 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:51Z" level=error msg="time=\"2021-04-23T22:05:51Z\" level=debug msg=\"removed task\" host=\"unix:///var/run/rexray/821099705.sock\" instanceID=\"ebs=i-0caf217be4b9eb934,availabilityZone=us-east-2b&region=us-east-2\" route=volumeCreate server=plain-fancier-ke service=ebs storageDriver=ebs task=0 tasksLen=0 time=1619215551169 tls=false txCR=1619215546 txID=b8b75546-874e-417b-7d6e-0b034489a502 " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16

Where the seemingly salient line is:

Apr 23 22:05:46 ip-10-90-11-62.us-east-2.compute.internal dockerd[4857]: time="2021-04-23T22:05:46Z" level=error msg="time=\"2021-04-23T22:05:46Z\" level=info msg=\"creating volume\" IOPS=0 availabilityZone= encrypted=<nil> encryptionKey=<nil> host=\"unix:///var/run/rexray/821099705.sock\" integrationDriver=linux opts=&{<nil> <nil> 0xc4201e23e0 <nil> <nil> <nil> map[kmskeyid:arn:aws:kms:us-east-2:012345678910:key/abcd1234-a123-456a-a12b-a123b4cd56ef size:1]} osDriver=linux service=ebs size=1 storageDriver=libstorage time=1619215546743 volumeName=61ee6ae6-062e-4cb0-a472-c30fd3301c37 volumeType= " plugin=f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16

But my EBS volumes are being created w/o encryption enabled:

$ aws ec2 describe-volumes --filter='[{"Name":"tag:Name","Values":["61ee6ae6-062e-4cb0-a472-c30fd3301c37"]}]' --query='Volumes[].Encrypted' | cat
[
    false
]

Version

[ec2-user@ip-10-90-11-62 ~]$ sudo chroot /var/lib/docker/plugins/f4c384f2347f4f97bcf5e8135e69335bbbe0419feae6dcf3ea6bef558bd9da16/rootfs ./rexray.sh rexray version
REX-Ray
-------
Binary: /
Flavor: client+agent+controller
SemVer: 0.11.4
OsArch: Linux-x86_64
Commit: e7414eaa971b27977d2283f2882825393493179d
Formed: Tue, 15 Jan 2019 16:03:57 UTC

Expected Behavior

EBS volumes are created encrypted.

Actual Behavior

EBS volumes are created without having been encrypted.

Isa Farnik · Answer 1 · Tue Apr 27 2021 08:51:08 GMT+0800 (China Standard Time)

I ended up resolving my issue by also supplying an encrypted driver option with the value true:

docker volume create -d rexray/ebs '--opt=kmsKeyID=arn:aws:kms:us-east-2:012345678910:key/abcd1234-a123-456a-a12b-a123b4cd56ef' --opt=encrypted=true --opt=volumetype=gp3 --opt=size=1