DeviceAuth: "invalid auth request: cannot decode public key"

dellgreen · February 3, 2021, 4:30pm

I have been successfully running mender-server 2.4.1 for a while now.

I wanted to change one of my devices device-auth keys, and decided to change from previous RSA key to ECDSA secp384r1 keys.

I decommissioned the device.

Now when i use the deviceauth api the server responds “invalid auth request: cannot decode public key”, if i change back to original RSA key it works fine again and the device is pending.

So i went about validating everything to ensure the ECDSA keys are valid, they are. Tried secp256r1 same problem. I then broke out the deviceauth ParsePubKey golang function that looks like the code that’s failing, and when i run this code locally against the public key that i’m sending to the server, it all passes fine.

github.com

mendersoftware/deviceauth/blob/2.4.1/utils/crypto.go#L97


      
          	default:
          		return errors.Wrap(errors.Errorf(
          			"public key algorithm (%T) not supported", pubkey,
          		), ErrMsgVerify)
          	}
          
          	return nil
          }
          
          //ParsePubKey
          func ParsePubKey(pubkey string) (crypto.PublicKey, error) {
          	block, _ := pem.Decode([]byte(pubkey))
          	if block == nil || block.Type != PubKeyBlockType {
          		return nil, errors.New("cannot decode public key")
          	}
          
          	key, err := x509.ParsePKIXPublicKey(block.Bytes)
          	if err != nil {
          		return nil, errors.Wrap(err, "cannot decode public key")
          	}

And this behaviour for me is repeatable in my live cloud instance and my local vm instance all running exactly the same versions and using the exact OS and rootfs. (VM was built from snapshot of cloud disk).

During the process of debugging I decided to checkout 2.4.1 and compile the deviceauth 2.4.1 docker service for myself in the VM with the intent to try and get more debug info on the problem. Before making any changes i rebuilt the deviceauth 2.4.1 image locally and replaced the existing image and container.

sudo docker build -t mendersoftware/deviceauth:mender-2.4.1 .

Now the problem is fixed just by building the docker service locally.

So on my cloud instance i threw away the existing deviceauth image and container and then ran the run script which repulled the image from upstream, and this didn’t fix the problem.

So i repeated the process of building the deviceauth docker image locally on the cloud server and installing it and just like the VM instance the cloud instance is also now fixed.

So is there a problem with the upstream docker deviceauth 2.4.1 image? is it being built with older dependencies than mine? or is it something else?

drewmoseley · February 3, 2021, 4:48pm

It does sound like something is wrong with the image in the Docker hub. @peter any ideas?

Drew

dellgreen · February 3, 2021, 5:49pm

Don’t know if this is relevant, but the VM and the cloud server that i built the docker images on are both running Ubuntu Server 18.04.5 LTS

peter · February 5, 2021, 9:04am

I really missed you Drew.

peter

peter · February 5, 2021, 9:06am

Hello @dellgreen

and thank you for using Mender.
I really appreciate a detailed description and the tests you have made. Not much I could add. I will have to try to replicate that and come back to you.
I gather that for the time being you are not blocked?

thanks,
peter

dellgreen · February 5, 2021, 9:44am

Hi @peter

It’s definitely a strange one. It makes no sense to me how a local build could fix the problem and caught me by surprise.

No it’s not a blocker, I’m currently running the locally built docker image in production and my VM.

peter · February 5, 2021, 10:38am

thanks!
could you try to send output of ldd on the deviceauth binary?

peter

dellgreen · February 5, 2021, 10:42am

Yep will get that next for you

The following is the list of docker images on a now working system

REPOSITORY                              TAG                            IMAGE ID       CREATED        SIZE
mendersoftware/deviceauth               mender-2.4.1                   8648a9a3c121   43 hours ago   34MB
<none>                                  <none>                         60d75e059132   43 hours ago   413MB
golang                                  1.14-alpine3.12                e91ef1a7b5ef   2 weeks ago    287MB
alpine                                  3.12                           389fef711851   7 weeks ago    5.57MB
mongo                                   3.6                            ab27d3bb28c6   8 weeks ago    446MB
mendersoftware/workflows-worker         mender-2.4.1                   bd7f64fae715   3 months ago   35.9MB
mendersoftware/workflows                mender-2.4.1                   f900df9fa1f7   3 months ago   35.9MB
mendersoftware/useradm                  mender-2.4.1                   1e21828d50e5   3 months ago   26.5MB
mendersoftware/inventory                mender-2.4.1                   ec9b7796eacd   3 months ago   28.7MB
mendersoftware/gui                      mender-2.4.1                   2d288f7d4f2b   3 months ago   33.8MB
mendersoftware/deployments              mender-2.4.1                   c9590fc987de   3 months ago   35MB
mendersoftware/create-artifact-worker   mender-2.4.1                   7603e2f3b9b6   3 months ago   89.8MB
mendersoftware/api-gateway              mender-2.4.1                   3f9004e80e30   3 months ago   54.1MB
minio/minio                             RELEASE.2018-09-25T21-34-43Z   fcc1e310a5cf   2 years ago    35.7MB
openresty/openresty                     1.13.6.2-0-alpine              08d5c926e4b6   2 years ago    49.3MB

I’m not a docker expert, but the created date of the “golang 1.14-alpine3.12” image of 2 weeks ago looks suspicious.

dellgreen · February 5, 2021, 10:43am

The history of that image is “golang 1.14-alpine3.12” :

IMAGE          CREATED       CREATED BY                                      SIZE      COMMENT
e91ef1a7b5ef   2 weeks ago   /bin/sh -c #(nop) WORKDIR /go                   0B        
<missing>      2 weeks ago   /bin/sh -c mkdir -p "$GOPATH/src" "$GOPATH/b…   0B        
<missing>      2 weeks ago   /bin/sh -c #(nop)  ENV PATH=/go/bin:/usr/loc…   0B        
<missing>      2 weeks ago   /bin/sh -c #(nop)  ENV GOPATH=/go               0B        
<missing>      2 weeks ago   /bin/sh -c set -eux;  apk add --no-cache --v…   281MB     
<missing>      2 weeks ago   /bin/sh -c #(nop)  ENV GOLANG_VERSION=1.14.14   0B        
<missing>      7 weeks ago   /bin/sh -c #(nop)  ENV PATH=/usr/local/go/bi…   0B        
<missing>      7 weeks ago   /bin/sh -c [ ! -e /etc/nsswitch.conf ] && ec…   17B       
<missing>      7 weeks ago   /bin/sh -c apk add --no-cache   ca-certifica…   497kB     
<missing>      7 weeks ago   /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B        
<missing>      7 weeks ago   /bin/sh -c #(nop) ADD file:ec475c2abb2d46435…   5.57MB

dellgreen · February 5, 2021, 11:07am

There’s no dynamic section in the deviceauth elf binary so ldd and readelf either fail or display an error message. Is there any other info i can get for you?

peter · February 5, 2021, 12:03pm

ah, ok. sorry for the noise then. I will try to reproduce in that case. thanks.

peter

Topic		Replies	Views
Device Auth API Post Help General Discussions api	6	1636	February 18, 2020
How to authenticate to device API when using official mender client General Discussions api , authentication	1	585	February 28, 2022
Preauthorization no longer working after upgrading to v2.6 General Discussions upgrade , preauthorization	2	425	March 18, 2021
Preauthorization on mender devauth 1.6.0 General Discussions	4	554	September 11, 2020
Update public key General Discussions	1	236	September 20, 2022

DeviceAuth: "invalid auth request: cannot decode public key"

Related topics