OpenSSL error in ASN1_get_object: too long

Question

OpenSSL error in ASN1_get_object: too long

sdanbury opened this issue 6 years ago · comments

I am getting the error when the jwt_decode_sig function calls the signature_verify function. Any ideas on where to look at first?

If I copy the jwt_split code and pass the sig through it, it works fine and I can get all of the information out of the JWT signature. However, when I use the jwt_decode_sig function directly, it fails with the error, even though the public key is valid, and I assume the sig is valid.

Sam Danbury · Answer 1 · Wed Dec 05 2018 16:50:56 GMT+0800 (China Standard Time)

I am trying to do the following Python, in R:

import jwt
import requests
import base64
import json

# Step 1: Get the key id from JWT headers (the kid field)
encoded_jwt = headers.dict['x-amzn-oidc-data']
jwt_headers = encoded_jwt.split('.')[0]
decoded_jwt_headers = base64.b64decode(jwt_headers)
decoded_json = json.loads(decoded_jwt_headers)
kid = decoded_json['kid']

# Step 2: Get the public key from regional endpoint
url = 'https://public-keys.auth.elb.' + region + '.amazonaws.com/' + kid
req = requests.get(url)
pub_key = req.text

# Step 3: Get the payload
payload = jwt.decode(encoded_jwt, pub_key, algorithms=['ES256'])

This is so that I can verify the JWT sent back from the AWS ALB and in turn safely use the information stored within it.

Jeroen Ooms · Answer 2 · Wed Dec 05 2018 20:10:43 GMT+0800 (China Standard Time)

Do you have an example of the code that is giving this error?

Sam Danbury · Answer 3 · Wed Dec 05 2018 21:31:50 GMT+0800 (China Standard Time)

library(openssl)
library(jose)

jwt <- "<jwt-here-as-string>"

pubkeystring <- "-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEkWo7tYMmQslyqwidviEsTQYfU6So
+Osx64z2wGRKzMpxxxxxxxxxxxxxxxxxxxxxxxxxx4g49Xz9A==
-----END PUBLIC KEY-----"

pubkey = read_pubkey(pubkeystring)

jwt_decode_sig(jwt, pubkey = pubkey)

The public key I got from passing the kid to the AWS URL as mentioned in the previous comments.

I would give you the JWT I was using exactly, but it contains sensitive information, apologies. I will try and recreate my setup in a throwaway AWS account in the meantime. Here is the basic structure of the header and payload of the JWT if it helps:

Header
{
  "typ": "JWT",
  "kid": "xxxxxxxxxxxxxxxxxxxxxx",
  "alg": "ES256",
  "iss": "https://mydomain.eu.auth0.com/",
  "client": "Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "signer": "arn:aws:elasticloadbalancing:eu-west-1:123456765434:loadbalancer/app/my-alb/xxxxxxxxxxxxxx",
  "exp": 1544016026
}

Payload
{
  "sub": "hello|world|helloworldhelloworld",
  "https://example.com/roles": "role1,role2",
  "https://example.com/permissions": []
}

Jeroen Ooms · Answer 4 · Wed Dec 05 2018 22:05:08 GMT+0800 (China Standard Time)

Are you sure your jwt string contains a signature? Does this example work for you?

library(jose)

pubkeystring <- "-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDdlatRjRjogo3WojgGHFHYLugd
UWAY9iR3fy4arWNA1KoS8kVw33cJibXr8bvwUAUparCwlvdbH6dvEOfou0/gCFQs
HUfQrSDv+MuSUMAe8jzKE4qW+jK+xQU9a03GUnKHkkle+Q0pX/g6jXZ7r1/xAK5D
o2kQ+X5xK9cipRgEKwIDAQAB
-----END PUBLIC KEY-----"
sig <- "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiYWRtaW4iOnRydWV9.EkN-DOsnsuRjRO6BxXemmJDm3HbxrbRzXglbN2S4sOkopdU4IsDxTI8jO19W_A4K8ZPJijNLis4EZsHeY559a4DFOd50_OqgHGuERTqYZyuhtF39yxJPAjUESwxk2J5k_4zM3O-vtd1Ghyo4IbqKKSy6J9mTniYJPenn5-HIirE"
pk <- openssl::read_pubkey(pubkeystring)

# Should return: 1234567890
jwt_decode_sig(sig, pk)

Sam Danbury · Answer 5 · Wed Dec 05 2018 22:16:49 GMT+0800 (China Standard Time)

That does work yes.

I copied the whole JWT string from the x-amzn-oidc-data header that is returned by the ALB after a successful login (more info found here).

I used this gist to write a shiny app behind the ALB which in turn gets me the x-amzn-oidc-data header. I then copied that into the R code that I sent to you, so I assume it is correct.

So you think it is a problem with the third part of the JWT (the signature)? If it helps, I put the public key, and the JWT into the https://jwt.io/ debugger and it said "valid signature".

Jeroen Ooms · Answer 6 · Thu Dec 06 2018 00:59:04 GMT+0800 (China Standard Time)

It's hard to say from here. You can debug(jwt_decode_sig) and then see where things are going wrong. Does this at least work for your jwt? jose:::jwt_split(jwt) ?

Jeroen Ooms · Answer 7 · Thu Dec 06 2018 01:08:09 GMT+0800 (China Standard Time)

Also what is your version of OpenSSL? openssl::openssl_config()

Sam Danbury · Answer 8 · Thu Dec 06 2018 18:51:19 GMT+0800 (China Standard Time)

I did as you suggested and tried to debug(jwt_decode_sig).

The error occurs here: https://github.com/jeroen/jose/blob/c1385e453a47042144aa6cff4f213a74a0269851/R/jwt.R#L117

and then here:

https://github.com/jeroen/openssl/blob/ff46f36c8de85854c09ca2996d030ea5697aaf1a/R/signing.R#L52

OpenSSL version:

$ R

R version 3.4.4 (2018-03-15) -- "Someone to Lean On"
...
> openssl::openssl_config()
$version
[1] "OpenSSL 1.0.2g  1 Mar 2016"

$ec
[1] TRUE

Jeroen Ooms · Answer 9 · Thu Dec 06 2018 19:06:15 GMT+0800 (China Standard Time)

What is the length of out$sig?

Sam Danbury · Answer 10 · Thu Dec 06 2018 19:16:27 GMT+0800 (China Standard Time)

...
> debug(jwt_decode_sig)        
> jwt_decode_sig(sig, pk)
debugging in: jwt_decode_sig(sig, pk)
debug: {
    out <- jwt_split(jwt)
    if (out$type != "RSA" && out$type != "ECDSA") 
        stop("Invalid algorithm: ", out$type)
    key <- read_pubkey(pubkey)
    if ((!inherits(key, "rsa") && !inherits(key, "ecdsa")) || 
        !inherits(key, "pubkey")) 
        stop("Key must be rsa/ecdsa public key")
    dgst <- sha2(out$data, size = out$keysize)
    if (!signature_verify(dgst, out$sig, hash = NULL, pubkey = key)) 
        stop(out$type, " signature verification failed!", call. = FALSE)
    structure(out$payload, class = c("jwt_claim", "list"))
}
Browse[2]> 
debug: out <- jwt_split(jwt)
Browse[2]> 
debug: if (out$type != "RSA" && out$type != "ECDSA") stop("Invalid algorithm: ", 
    out$type)
Browse[2]> out$sig
 [1] 82 55 33 5f d9 21 26 2e b8 f8 60 52 03 52 63 94 5b 6d 4b 57 51 55 2f 42 8b
[26] 57 54 b6 12 7e b9 85 af 36 0c 35 b0 65 15 64 68 a0 48 7e f6 f9 44 b0 0e bb
[51] e0 57 ee 9d 36 68 13 2b 1a 3e 9e cc 43 f4
Browse[2]> length(out$sig)
[1] 64
Browse[2]> nchar(out$sig)
 [1] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
[39] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
Browse[2]>

Jeroen Ooms · Answer 11 · Thu Dec 06 2018 20:04:13 GMT+0800 (China Standard Time)

Hmm I am confused 🤔The spec says that the signature has to be 64 bytes but openssl seems to expect signatures of 70 or 71 bytes:

library(openssl)
test <- charToRaw('hello')
signature_create(test, key = ec_keygen('P-256'))

Perhaps there is an issue with the DER encoding of the signature in openssl. But it seems to work just fine for other applications.

Possibly related: auth0/java-jwt#187, mpdavis/python-jose#47

Sam Danbury · Answer 12 · Thu Dec 06 2018 20:18:13 GMT+0800 (China Standard Time)

Very interesting. So are you saying that there could be an issue in the library that is used to create the JWT signature that I am getting back from the AWS ALB because this library (which follows the spec) is unable to verify it?

Jeroen Ooms · Answer 13 · Thu Dec 06 2018 20:55:05 GMT+0800 (China Standard Time)

No, there is a bug here. Apparently the jwt spec uses an unconventional way to encode the signature to make sure it is always 64 bytes. We need to convert it from what openssl gives us, exactly the same problem as the java and python people have run into above.

Jeroen Ooms · Answer 14 · Fri Dec 07 2018 05:14:04 GMT+0800 (China Standard Time)

I have pushed a fix based on what the python/java implementations linked above are doing. Could you test this please?

remotes::install_github("jeroen/jose")

I should add better test cases because roundtripping apparently isn't enough.

Sam Danbury · Answer 15 · Fri Dec 07 2018 21:25:20 GMT+0800 (China Standard Time)

Thanks for the really quick turnaround on this!

I tried it, by doing the following:

$ mkdir libs
$ R_LIBS=libs Rscript -e 'install.packages("remotes")'
...
$ R_LIBS=libs Rscript -e 'remotes::install_github("jeroen/jose")'
...
$ R_LIBS=libs Rscript jwt.R
Loading required package: openssl

 *** caught segfault ***
address 0x7, cause 'memory not mapped'

Traceback:
 1: .Call(R_write_ecdsa, r, s)
 2: openssl::ecdsa_write(r, s)
 3: jwt_decode_sig(sig, pk)
An irrecoverable exception occurred. R is aborting now ...
Segmentation fault (core dumped)

Where jwt.R is:

library(jose)

pubkeystring <- "-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEkWo7tYMmQslyqwidviEsTQYfU6So
+Osx64z2wGRKzMpliIBpMAGqS1Eof5F+6ZJfWudUn7eeMioNPJ4g49Xz9A==
-----END PUBLIC KEY-----"

sig <- "blahblahblah"

pk = openssl::read_pubkey(pubkeystring)

jwt_decode_sig(sig, pk)

So yeah, it segfaults. Would you like me to get you an actual JWT to work with? Or have you recreated it with Auth0 and AWS ALB on your side?

Jeroen Ooms · Answer 16 · Fri Dec 07 2018 21:39:22 GMT+0800 (China Standard Time)

I think something went wrong in updating the openssl package. Can you try again please? Make sure neither jose or openssl is loaded in your R session when you install the update:

remotes::install_github("jeroen/openssl")
remotes::install_github("jeroen/jose")

Sam Danbury · Answer 17 · Sat Dec 08 2018 00:18:16 GMT+0800 (China Standard Time)

I am still getting the error with a clean environment. Starting from scratch, into a fresh libs directory, I install remotes, then run the two lines you just posted, then run the code, and it gives:

> 
> pk = openssl::read_pubkey(pubkeystring)
> jwt_decode_sig(sig, pk)
Error in jwt_decode_sig(sig, pk) : 
  could not find function "jwt_decode_sig"
> library(jose)
Loading required package: openssl
> jwt_decode_sig(sig, pk)

 *** caught segfault ***
address 0x7, cause 'memory not mapped'

Traceback:
 1: .Call(R_write_ecdsa, r, s)
 2: openssl::ecdsa_write(r, s)
 3: jwt_decode_sig(sig, pk)

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 1
R is aborting now ...
Segmentation fault (core dumped)

Jeroen Ooms · Answer 18 · Sun Dec 09 2018 21:04:15 GMT+0800 (China Standard Time)

Hmm that's very odd. Which OS are you on?

Sam Danbury · Answer 19 · Mon Dec 10 2018 18:33:49 GMT+0800 (China Standard Time)

Linux Ubuntu 16.04

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.5 LTS"

Jeroen Ooms · Answer 20 · Tue Dec 11 2018 22:57:02 GMT+0800 (China Standard Time)

Do you only get this crash for your particular signature, or also when running example(ecdsa_write)

Sam Danbury · Answer 21 · Wed Dec 12 2018 19:36:36 GMT+0800 (China Standard Time)

I am guessing a lot of these problems are to do with the fact that AWS ALB returns a base64 encoded token, not a base64 URL encoded token. I have been experiencing issues with javascript libraries as well, this issue pretty much sums it up: auth0/node-jsonwebtoken#514.

Possible that the same is happening here? Maybe the == padding on each of the three sections of the JWT is causing issues.

Anyway, I will try your code this afternoon with example(ecdsa_write) and see what happens. Thanks again.

Jeroen Ooms · Answer 22 · Thu Dec 13 2018 23:56:54 GMT+0800 (China Standard Time)

Hmm invalid input should never cause a crash.

I do have an amazon account, can you explain me where I would create a test jwt token?

Sam Danbury · Answer 23 · Fri Dec 14 2018 00:37:46 GMT+0800 (China Standard Time)

You would need to create the following:

AWS Application Load Balanacer (ALB). It is the ALB that does the OIDC dance with an identity provider like Auth0 (I am using Auth0) and then passes the JWT token down to the services that run behind the ALB.
An EC2 instance that will sit behind the ALB instance. On this instance you need some sort of web app that will print out the headers, so you can see the JWT header that is sent by the ALB.
You need to create a target group and register the EC2 instance in that target group, and then create an ALB rule to route all traffic / to your target group (your EC2 instance).

This post does a good job of explaining how it all fits together.

Basically people can use the ALB built-in OIDC auth to authenticate apps so the apps behind the load balancer don't have to (or can't authenticate themselves).

Jeroen Ooms · Answer 24 · Fri Dec 14 2018 01:05:25 GMT+0800 (China Standard Time)

OK that is more complicated than I thought. Is there no simpler application of jwt's in amazon?

If you can generate a dummy jwt that would be super helpful. You can revoke it immediately and email it to me because we only need to test the parser.

Jeroen Ooms · Answer 25 · Fri Jan 18 2019 03:59:44 GMT+0800 (China Standard Time)

I finally found the problem. Should be fixed in the openssl package 1.2.1.

Sam Danbury · Answer 26 · Fri Jan 18 2019 16:40:19 GMT+0800 (China Standard Time)

Great! Thanks so much for continuing the effort. What was the issue in the end?

Do you have a patreon or similar that I can contribute to?

Jeroen Ooms · Answer 27 · Tue Jan 22 2019 08:59:43 GMT+0800 (China Standard Time)

The issue was some uninitiated null pointer in openssl, that could make it crash. I uploaded jose 1.0 to cran. Does everything work now for you as expected?

I don't have patreon, your feedback is a contribution in itself :) You can always donate to WWF if you want to make me happy.

Sam Danbury · Answer 28 · Fri Mar 29 2019 18:24:24 GMT+0800 (China Standard Time)

@jeroen sorry for the late response! From initial testing, it looks like it is now fixed. When I find some time to do some more thorough testing, I will raise a new issue if anything pops up.

Thanks again for all of the effort. I made a donation to WWF on your behalf.