datafibers / simple_encryption

effective local aes encryption and decryption

Simple encryption leverages AES algorithm to encrypt and decrypt data. It uses secret key, iv with proper padding for detailed implementation.

Key Prepare

1. Generate security key with key folder and number of keys needed

java -jar simple_encryption.jar AESUtil dev 31

2. Distribute the file to where application can access, such as HDFS or GitHub with Spring Cloud Config.

Encryption and Decryption from Application

1. Read all key files to keyCache with encoded

val keyCache = cacheKeyFromFolders("key")

2. Encrypt the data (string) with rule-based version of keys

val cipherText = AesUtil.encryptWithVer(input, keyCache)

3. Decrypt the data with the version of keys in cipher

val plainText = AesUtil.decryptWithVer(cipherText, keyCache)

Encryption and Decryption from Spark

1. Cache the keyfiles and choose which key version to use with rules

val keyCache = cacheKeyFromFolders("key")

2. Encrypt the dataframe (df) string column with rule-based version of keys

val encryptDf = dsEncrypt(df, "email,address", keyCache)

3. Decrypt the dataframe (encryptDf) with the version of keys in cipher

val decryptDf = dsDecrypt(encryptDf, "email,address", keyCache)

4. You can also use chained call in spark

val decryptDf = df
  .transform(dsEncrypt("sin", keyCache))
  .transform(dsDecrypt("sin", keyCache))

Key Rotation

1. The key can rotate from the encryption side as follows

   rotate rule	comments
   always	rotate keys for every run
   day	rotate keys on every day
   month	rotate keys on every month, default
   year	rotate keys on every year

2. Once all keys are cached, decryption works all the time.

3. If keys are destroyed, the cache keys should be removed carefully (make sure not being used on history data). This usually applies to your data has retention period.

4. If new keys are added, do not reuse the old version as follows. It creates additional 5 keys starting from version 31

java -jar simple_encryption.jar AESUtil dev 5 31 

Format

• The default key file format is version (3 byte), key (16 byte/128 bit), and iv (16 byte/128 bit).
• The cipher text format is key_version (3 byte), cipher text.

TODO

• Add support to for key generation dynamically from DES/KMS
• Add support encryption with hashing so that data can also be used as join conditions
• Add spark decryption function
• Add performance test cases