Hi there,
I was following this guide for Mender Server 3.7 and this guide for Mender Server 4.0 and they did not seem to be working. The hope for me is that I can learn from these setups and transition to a locally hosted production server. Although if there is an easier setup path using other versions I would greatly appreciate it!
Following the steps exactly for Mender Server 3.7 and Mender Server 4.0, I run these commands
# Mender Server 3.7
helm upgrade --install mender mender/mender --version 5.11.2 -f mender-3.7.8.yml --debug
# Mender Server 4.0
helm install mender mender/mender -f mender-values.yml --debug
This is the error message I get:
coalesce.go:286: warning: cannot overwrite table with non table for mender.nats.nats.image (map[pullPolicy:IfNotPresent repository:nats tag:2.9.20-alpine])
Error: INSTALLATION FAILED: template: mender/templates/useradm/secret.yaml:17:46: executing "mender/templates/useradm/secret.yaml" at <b64enc>: invalid value; expected string
helm.go:86: 2024-12-20 13:59:22.048774449 +1100 AEDT m=+0.230082954 [debug] template: mender/templates/useradm/secret.yaml:17:46: executing "mender/templates/useradm/secret.yaml" at <b64enc>: invalid value; expected string
INSTALLATION FAILED
main.newInstallCmd.func2
helm.sh/helm/v3/cmd/helm/install.go:158
github.com/spf13/cobra.(*Command).execute
github.com/spf13/cobra@v1.8.1/command.go:985
github.com/spf13/cobra.(*Command).ExecuteC
github.com/spf13/cobra@v1.8.1/command.go:1117
github.com/spf13/cobra.(*Command).Execute
github.com/spf13/cobra@v1.8.1/command.go:1041
main.main
helm.sh/helm/v3/cmd/helm/helm.go:85
runtime.main
runtime/proc.go:271
runtime.goexit
runtime/asm_amd64.s:1695
Troubleshooting this further, I decided to just modify mender-values.yml to the following:
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 | openssl rsa -out device_auth.key
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 | openssl rsa -out useradm.key
export MENDER_SERVER_DOMAIN="mender.example.com"
export MENDER_SERVER_URL="https://${MENDER_SERVER_DOMAIN}"
cat <<-EOF > mender-values.yml
ingress:
enabled: true
annotations:
cert-manager.io/issuer: "letsencrypt"
ingressClassName: traefik
path: /
hosts:
- ${MENDER_SERVER_DOMAIN}
tls:
# this secret must exists or it can be created from a working cert-manager instance
- secretName: mender-ingress-tls
hosts:
- ${MENDER_SERVER_DOMAIN}
global:
s3:
AWS_URI: "${MENDER_SERVER_URL}"
AWS_BUCKET: "${STORAGE_BUCKET}"
AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}"
AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}"
url: "${MENDER_SERVER_URL}"
api_gateway:
storage_proxy:
enabled: true
url: "${STORAGE_ENDPOINT}"
customRule: "PathRegexp(\`^/${STORAGE_BUCKET}\`)"
minio:
enabled: false
deployments:
customEnvs:
- name: DEPLOYMENTS_STORAGE_PROXY_URI
value: "${MENDER_SERVER_URL}"
device_auth:
certs:
key: |-
$(cat device_auth.key | sed -e 's/^/ /g')
useradm:
certs:
key: |-
$(cat useradm.key | sed -e 's/^/ /g')
tenantadm:
certs:
key: |-
$(cat useradm.key | sed -e 's/^/ /g')
EOF
This managed to satisfy the error message, although I feel that this is not the right way to provide certificates. Could anyone help me with this?
However, this led to further errors below:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/rancher/k3s/k3s.yaml
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /etc/rancher/k3s/k3s.yaml
install.go:224: 2024-12-20 16:07:52.884320083 +1100 AEDT m=+0.059427673 [debug] Original chart version: ""
install.go:241: 2024-12-20 16:07:52.967391691 +1100 AEDT m=+0.142499274 [debug] CHART PATH: /home/genesys-linux/.cache/helm/repository/mender-5.12.0.tgz
coalesce.go:286: warning: cannot overwrite table with non table for mender.nats.nats.image (map[pullPolicy:IfNotPresent repository:nats tag:2.9.20-alpine])
...
client.go:142: 2024-12-20 16:07:54.989065659 +1100 AEDT m=+2.164173238 [debug] creating 1 resource(s)
client.go:486: 2024-12-20 16:07:54.995998345 +1100 AEDT m=+2.171105919 [debug] Starting delete for "mender-auditlogs-migration" Job
client.go:490: 2024-12-20 16:07:54.998158098 +1100 AEDT m=+2.173265673 [debug] Ignoring delete failure for "mender-auditlogs-migration" batch/v1, Kind=Job: jobs.batch "mender-auditlogs-migration" not found
wait.go:104: 2024-12-20 16:07:54.998186058 +1100 AEDT m=+2.173293637 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2024-12-20 16:07:55.048067055 +1100 AEDT m=+2.223174646 [debug] creating 1 resource(s)
client.go:712: 2024-12-20 16:07:55.05996332 +1100 AEDT m=+2.235070905 [debug] Watching for changes to Job mender-auditlogs-migration with timeout of 5m0s
client.go:740: 2024-12-20 16:07:55.068276663 +1100 AEDT m=+2.243384252 [debug] Add/Modify event for mender-auditlogs-migration: ADDED
client.go:779: 2024-12-20 16:07:55.068322126 +1100 AEDT m=+2.243429703 [debug] mender-auditlogs-migration: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:740: 2024-12-20 16:07:55.076262403 +1100 AEDT m=+2.251369986 [debug] Add/Modify event for mender-auditlogs-migration: MODIFIED
client.go:779: 2024-12-20 16:07:55.076299091 +1100 AEDT m=+2.251406668 [debug] mender-auditlogs-migration: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
Error: INSTALLATION FAILED: failed pre-install: 1 error occurred:
* timed out waiting for the condition
helm.go:86: 2024-12-20 16:12:55.126814234 +1100 AEDT m=+302.301921813 [debug] failed pre-install: 1 error occurred:
* timed out waiting for the condition
INSTALLATION FAILED
main.newInstallCmd.func2
helm.sh/helm/v3/cmd/helm/install.go:158
github.com/spf13/cobra.(*Command).execute
github.com/spf13/cobra@v1.8.1/command.go:985
github.com/spf13/cobra.(*Command).ExecuteC
github.com/spf13/cobra@v1.8.1/command.go:1117
github.com/spf13/cobra.(*Command).Execute
github.com/spf13/cobra@v1.8.1/command.go:1041
main.main
helm.sh/helm/v3/cmd/helm/helm.go:85
runtime.main
runtime/proc.go:271
runtime.goexit
runtime/asm_amd64.s:1695
Running kubectl describe job mender-auditlogs-migration showed these lines:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 2m21s job-controller Created pod: mender-auditlogs-migration-fxmml
Finally, running kubectl describe pod mender-auditlogs-migration-fxmml showed these lines:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7m41s default-scheduler Successfully assigned default/mender-auditlogs-migration-fxmml to genesys-linux-to-be-filled-by-o-e-m
Normal Pulling 6m13s (x4 over 7m38s) kubelet Pulling image "registry.mender.io/mendersoftware/auditlogs:mender-3.7.8"
Warning Failed 6m13s (x4 over 7m37s) kubelet Failed to pull image "registry.mender.io/mendersoftware/auditlogs:mender-3.7.8": failed to pull and unpack image "registry.mender.io/mendersoftware/auditlogs:mender-3.7.8": failed to resolve reference "registry.mender.io/mendersoftware/auditlogs:mender-3.7.8": pull access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentials
Warning Failed 6m13s (x4 over 7m37s) kubelet Error: ErrImagePull
Warning Failed 5m43s (x6 over 7m37s) kubelet Error: ImagePullBackOff
Could anybody help see why there is a pull access denied for the required Image? Your help is greatly appreciated!