File Transfer using Mender-CLI and API

General Discussions
,
1

Hey guys,

So have a couple questions in regards to using mender-cli and the mender API with the new file transferring feature.

Question #1 - How do i properly use mender-cli to conduct a file transfer to/from a device.
Please note that I am using ‘.mender-clirc.json’
Currently for uploading to a device, i have used this full command:

mender-cli cp <device-id>:<device-download-path> <path-to-local-file> --server <server-address>

and get the error:

FAILURE: Missing X-MEN-FILE-MODE header

I have looked at all available documentation and even tried going through the actual source code, but it doesn’t mention anywhere how to add the file mode header to the command above.
To that point, I have no clue how to do the reverse command of requesting a file from remote device, to be locally downloaded, if someone would also be able to elaborate on this matter, I’d greatly appreciate it.

Question #2 - I would also like to try using the mender API (using JavaScript) for file transfers.
Do i need to login by means of a Token or something? Is using this feature of the mender API part of the Mender Open-Source model? or is it only available with a paid mender subscription?

Thanks guys, really appreciate someone taking the time to look into this, this has been driving me nuts.

2

hello @joseph

and thank you for trying File Transfer. as for the first question, could you tell me what mender-cli version are you using?
To use the API you need to have a valid JWT token, so yes, you need to authorize.
The File Transfer is part of the Troubleshoot add-on package

best regards,
peter

3

Hey @peter, thank you for your prompt reply.

So I’m using the latest mender-cli version (53b85e0).

For the JWT Token, are we able to acquire this token with an API Login request, while being on open-source? From my understanding this should be the functionality (and the JWT Token is different from a Two factor authentication token, is this correct?)

Thanks, looking forward to your reply :slight_smile:!

-Joseph

4

Hey @joseph

just to let you know I have not forgotten: I need to try to replicate it.

peter

5

Hey @peter,

Sounds good, I understand and thank you!

Little update in regards to using the API, I managed to get the JWT token but now I’m having issues figuring out the CORS policy headers that the API is looking for, for the file transfer. This is currently what I have, and also the error that I am receiving.

  let fileList = document.getElementById("FILEUPLOAD").files[0];
  let uploadPath = "/data/joseph/" // document.getElementById("PATHtoDEV").value;
  async function uploadFile(evt) {
    let fileBinary = evt.target.result;

    console.log(fileBinary)
    const inputBody =
      `{
      'path': ${uploadPath},
      'uid': 0,
      'gid': 0,
      'mode': '777',
      'file': ${fileBinary}
    }`;
    const headers = {
      'Content-Type':'multipart/form-data',
      'Accept':'application/json',
      'Authorization':'Bearer ' + jwtToken,
      'Access-Control-Allow-Origin':'*'
    };

    uploadResponse = await fetch(  serverAddr + 'api/management/v1/deviceconnect/devices/' + deviceID + '/upload',
      {
        method: 'PUT',
        body: inputBody,
        headers: headers
      })
    //   .then(function(res) {
    //     return res.json();
    //   }).then(function(body) {
    //   console.log(body);
    //   statusMessage.innerHTML = body
    // });
    if (uploadResponse.ok) {
      let response = await uploadResponse.json()
      console.log(response)
    }
    else {
      console.log("Upload error has occurred")
    }
  }

  let reader = new FileReader();
  reader.addEventListener('load', uploadFile)
  reader.readAsBinaryString(fileList)

Error that i received:

from origin 'null' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: It does not have HTTP ok status.

Please let me know if you have any thoughts about the API usage/if you can point me in the right direction.

Looking forward to your reply, thank you! :slight_smile:

6

regarding the API usage from js, is that an on-premise setup, meaning that you are running the server? (I can only gather that you are calling API from js from different domain that the server runs, am I mistaken?)

peter

7

Hey @peter,

The server that I am trying to access is an on-premise server (it will be an AWS server in the future). So it has a full server URL during the API call.

serverAddr + 'api/management/v1/deviceconnect/devices/' + deviceID + '/upload'

serverAddr - full https address with a / at the end.

Did this answer your question?

Thanks!

Joseph

8

yes, it did. I guess in that case you will have to modify the api-gw CORS headers, from the top of my head I can’t give you the solution, I would start by looking into the container, and playing with the configuration.

peter

9

Okay gotcha, I’ll do that as my first step in the right direction.

As for the mender-cli, I may have stumbled upon how the command needs to be called between a upload/download (by looking through the source code), it seems that these positions just need to be reversed, I have not confirmed this though (still unsure of how to provide the file mode, getting the same error “FAILURE: Missing X-MEN-FILE-MODE header”):

<device-id>:<path-on-device> <local-path-to-file>

<local-path-to-file> <device-id>:<path-on-device>

Thank you @peter!

Joseph

10

Hey @peter,

I think I may have misunderstood you. So the server is on a AWS cloud server. I am running my script outside of the server space, so it is local to me but remote to the server.

Does this change your input at all?

I have asked our administrator to look into the configuration setup of the server, but they are having a difficult time locating where the CORS policy would be kept. Would you happen to know the location of it on the server?

Thanks!

Joseph

11

Hey @joseph

I cannot replicate the mender-cli problem:

mender-cli cp ffe2ff6f-ed2d-4d24-8ff8-5c0dff65d1ff:/etc/profile /tmp/p --server https://myhostname --skip-verify
Configuration file not found. Continuing.
Successfully downloaded the file: "/etc/profile" from device ...

could you tell me what version of deviceconnect you are running with executing:

./deviceconnect --version

from the deviceconnect container?
the CORS: you should be able to do what you want, by checking /etc/traefik/config directory and using Headers - Traefik let me know if you run into problems, I can get some traefik experts online.

peter

12

Hey @peter,

Not sure what i’m doing that is different. I was thinking that maybe the mender-cli might have been out of date, even though we confirmed it, so I just downloaded the latest one from here: Downloads | Mender documentation, and I’m still getting the same error of ‘FAILURE: Missing X-MEN-FILE-MODE header’. I presume the link on the downloads page is the latest one?

So the deviceconnect version is 1.0.0.

Sounds good about the CORS, will dig into that and test.

Thanks!

Joseph

13

one follow up question: does the file transfer work from the UI?
could I have the docker ps output? the deviceconnect 1.0.0 is a bit ancient, and it does not add the file mode header. which version of the Mender Server are you running?

peter

14

Hey @peter ,

The file transfer does work from the UI, no problems there. The server is running on v2.7.

docker ps output

526202f16a84   mendersoftware/deployments:mender-master              "/entrypoint.sh --co…"   2 weeks ago   Up 10 minutes             8080/tcp                                        menderproduction_mender-deployments_1
f0c46a86d7a3   traefik:v2.4                                          "/entrypoint.sh --ac…"   2 weeks ago   Up 11 minutes             80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   menderproduction_mender-api-gateway_1
3b0cceb600bd   mendersoftware/deviceauth:mender-master               "/usr/bin/deviceauth…"   2 weeks ago   Up 11 minutes             8080/tcp                                        menderproduction_mender-device-auth_1
bc100f873a1b   mendersoftware/workflows-worker:mender-master         "/usr/bin/workflows …"   2 weeks ago   Up 11 minutes                                                             menderproduction_mender-workflows-worker_1
2ada955e2e04   mendersoftware/deviceconfig:mender-master             "/usr/bin/deviceconf…"   2 weeks ago   Up 11 minutes             8080/tcp                                        menderproduction_mender-deviceconfig_1
e8d3a661888a   mendersoftware/useradm:mender-master                  "/usr/bin/useradm --…"   2 weeks ago   Up 11 minutes             8080/tcp                                        menderproduction_mender-useradm_1
f88a0a09a001   mendersoftware/workflows:mender-master                "/usr/bin/workflows …"   2 weeks ago   Up 11 minutes             8080/tcp                                        menderproduction_mender-workflows-server_1
392986e880ba   mendersoftware/create-artifact-worker:mender-master   "/usr/bin/workflows …"   2 weeks ago   Up 11 minutes             8080/tcp                                        menderproduction_mender-create-artifact-worker_1
770d0649d0c5   mendersoftware/inventory:mender-master                "/usr/bin/inventory …"   2 weeks ago   Up 11 minutes             8080/tcp                                        menderproduction_mender-inventory_1
fe2f009fdc25   mendersoftware/deviceconnect:mender-master            "/usr/bin/deviceconn…"   2 weeks ago   Up 11 minutes             8080/tcp                                        menderproduction_mender-deviceconnect_1
9b252417d694   mendersoftware/gui:mender-master                      "/entrypoint.sh nginx"   2 weeks ago   Up 11 minutes (healthy)   80/tcp, 8080/tcp                                menderproduction_mender-gui_1
3ccc523d5be5   minio/minio:RELEASE.2019-04-23T23-50-36Z              "/usr/bin/docker-ent…"   2 weeks ago   Up 11 minutes (healthy)   9000/tcp                                        menderproduction_minio_1
b04619e1fc18   mongo:4.4                                             "docker-entrypoint.s…"   2 weeks ago   Up 11 minutes             27017/tcp                                       menderproduction_mender-mongo_1
ecc6c7edead8   nats:2.1.9-alpine3.12                                 "docker-entrypoint.s…"   2 weeks ago   Up 11 minutes             4222/tcp, 6222/tcp, 8222/tcp                    menderproduction_mender-nats_1

Thanks!

Joseph

15

Hey!

ok, I checked again on a clean setup, I cant replicate the problem with v2.7 and the file transfer, it works here. although I am trying demo, but it should be the same, since it pulls the same images. at the moment I am out of ideas, I will let you know when I have anything. When time permits I will run 2.7 production installation and check.
sorry I could not help more.

peter

16

@peter,

Thank you for putting in as much time as you already have into this, I really appreciate it!

So just to update what I’ve done on my end, I haven’t been able to get mender-cli working or the mender API with HTML/JavaScript/NodeJS (CORS Issues), but I was able to get it working using Python. Which makes me think that the CORS issues must be related to how I’m setting the headers or how the browser is interpreting it.

If you do happen to think of anything, please let me know! Thanks again for your help, till then!

Joseph