: invalid value; expected string
INSTALLATION FAILED
main.newInstallCmd.func2
helm.sh/helm/v3/cmd/helm/install.go:158
github.com/spf13/cobra.(*Command).execute
github.com/spf13/cobra@v1.8.1/command.go:985
github.com/spf13/cobra.(*Command).ExecuteC
github.com/spf13/cobra@v1.8.1/command.go:1117
github.com/spf13/cobra.(*Command).Execute
github.com/spf13/cobra@v1.8.1/command.go:1041
main.main
helm.sh/helm/v3/cmd/helm/helm.go:85
runtime.main
runtime/proc.go:271
runtime.goexit
runtime/asm_amd64.s:1695
```
Troubleshooting this further, I decided to just modify `mender-values.yml` to the following:
```auto
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 | openssl rsa -out device_auth.key
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 | openssl rsa -out useradm.key
export MENDER_SERVER_DOMAIN="mender.example.com"
export MENDER_SERVER_URL="https://${MENDER_SERVER_DOMAIN}"
cat <<-EOF > mender-values.yml
ingress:
enabled: true
annotations:
cert-manager.io/issuer: "letsencrypt"
ingressClassName: traefik
path: /
hosts:
- ${MENDER_SERVER_DOMAIN}
tls:
# this secret must exists or it can be created from a working cert-manager instance
- secretName: mender-ingress-tls
hosts:
- ${MENDER_SERVER_DOMAIN}
global:
s3:
AWS_URI: "${MENDER_SERVER_URL}"
AWS_BUCKET: "${STORAGE_BUCKET}"
AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}"
AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}"
url: "${MENDER_SERVER_URL}"
api_gateway:
storage_proxy:
enabled: true
url: "${STORAGE_ENDPOINT}"
customRule: "PathRegexp(\`^/${STORAGE_BUCKET}\`)"
minio:
enabled: false
deployments:
customEnvs:
- name: DEPLOYMENTS_STORAGE_PROXY_URI
value: "${MENDER_SERVER_URL}"
device_auth:
certs:
key: |-
$(cat device_auth.key | sed -e 's/^/ /g')
useradm:
certs:
key: |-
$(cat useradm.key | sed -e 's/^/ /g')
tenantadm:
certs:
key: |-
$(cat useradm.key | sed -e 's/^/ /g')
EOF
```
This managed to satisfy the error message, although I feel that this is not the right way to provide certificates. Could anyone help me with this?
However, this led to further errors below:
```auto
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/rancher/k3s/k3s.yaml
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /etc/rancher/k3s/k3s.yaml
install.go:224: 2024-12-20 16:07:52.884320083 +1100 AEDT m=+0.059427673 [debug] Original chart version: ""
install.go:241: 2024-12-20 16:07:52.967391691 +1100 AEDT m=+0.142499274 [debug] CHART PATH: /home/genesys-linux/.cache/helm/repository/mender-5.12.0.tgz
coalesce.go:286: warning: cannot overwrite table with non table for mender.nats.nats.image (map[pullPolicy:IfNotPresent repository:nats tag:2.9.20-alpine])
...
client.go:142: 2024-12-20 16:07:54.989065659 +1100 AEDT m=+2.164173238 [debug] creating 1 resource(s)
client.go:486: 2024-12-20 16:07:54.995998345 +1100 AEDT m=+2.171105919 [debug] Starting delete for "mender-auditlogs-migration" Job
client.go:490: 2024-12-20 16:07:54.998158098 +1100 AEDT m=+2.173265673 [debug] Ignoring delete failure for "mender-auditlogs-migration" batch/v1, Kind=Job: jobs.batch "mender-auditlogs-migration" not found
wait.go:104: 2024-12-20 16:07:54.998186058 +1100 AEDT m=+2.173293637 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2024-12-20 16:07:55.048067055 +1100 AEDT m=+2.223174646 [debug] creating 1 resource(s)
client.go:712: 2024-12-20 16:07:55.05996332 +1100 AEDT m=+2.235070905 [debug] Watching for changes to Job mender-auditlogs-migration with timeout of 5m0s
client.go:740: 2024-12-20 16:07:55.068276663 +1100 AEDT m=+2.243384252 [debug] Add/Modify event for mender-auditlogs-migration: ADDED
client.go:779: 2024-12-20 16:07:55.068322126 +1100 AEDT m=+2.243429703 [debug] mender-auditlogs-migration: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:740: 2024-12-20 16:07:55.076262403 +1100 AEDT m=+2.251369986 [debug] Add/Modify event for mender-auditlogs-migration: MODIFIED
client.go:779: 2024-12-20 16:07:55.076299091 +1100 AEDT m=+2.251406668 [debug] mender-auditlogs-migration: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
Error: INSTALLATION FAILED: failed pre-install: 1 error occurred:
* timed out waiting for the condition
helm.go:86: 2024-12-20 16:12:55.126814234 +1100 AEDT m=+302.301921813 [debug] failed pre-install: 1 error occurred:
* timed out waiting for the condition
INSTALLATION FAILED
main.newInstallCmd.func2
helm.sh/helm/v3/cmd/helm/install.go:158
github.com/spf13/cobra.(*Command).execute
github.com/spf13/cobra@v1.8.1/command.go:985
github.com/spf13/cobra.(*Command).ExecuteC
github.com/spf13/cobra@v1.8.1/command.go:1117
github.com/spf13/cobra.(*Command).Execute
github.com/spf13/cobra@v1.8.1/command.go:1041
main.main
helm.sh/helm/v3/cmd/helm/helm.go:85
runtime.main
runtime/proc.go:271
runtime.goexit
runtime/asm_amd64.s:1695
```
Running `kubectl describe job mender-auditlogs-migration` showed these lines:
```auto
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 2m21s job-controller Created pod: mender-auditlogs-migration-fxmml
```
Finally, running `kubectl describe pod mender-auditlogs-migration-fxmml` showed these lines:
```auto
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7m41s default-scheduler Successfully assigned default/mender-auditlogs-migration-fxmml to genesys-linux-to-be-filled-by-o-e-m
Normal Pulling 6m13s (x4 over 7m38s) kubelet Pulling image "registry.mender.io/mendersoftware/auditlogs:mender-3.7.8"
Warning Failed 6m13s (x4 over 7m37s) kubelet Failed to pull image "registry.mender.io/mendersoftware/auditlogs:mender-3.7.8": failed to pull and unpack image "registry.mender.io/mendersoftware/auditlogs:mender-3.7.8": failed to resolve reference "registry.mender.io/mendersoftware/auditlogs:mender-3.7.8": pull access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentials
Warning Failed 6m13s (x4 over 7m37s) kubelet Error: ErrImagePull
Warning Failed 5m43s (x6 over 7m37s) kubelet Error: ImagePullBackOff
```
Could anybody help see why there is a pull access denied for the required Image? Your help is greatly appreciated!
---
### Author:  [@robgio](https://hub.mender.io/u/robgio)
#### Post date: [December 20, 2024, 1:31pm UTC](https://hub.mender.io/t/production-installation-for-locally-hosted-mender-server-3-7-4-0-not-working/7414/2 "2024-12-20T13:31:07Z")
Hi @ianhgenesys
please don’t use the Mender Server v4.0 installation: it’s not released yet and the documentation you’re following is still in development.
For now you only have to follow the documentation for the v3.7 version and it seems you mixed up the two installations, because you didn’t specify the open source configuration:
```auto
global:
enterprise: false
```
which has to be specified for the Helm chart v5.2.
Can you start from scratch again, this time from the 3.7 only? Please don’t use MinIO because that is not working as documented, and will be fixed in the Helm Chart v6.0.0. The best is to use an Amazon S3 bucket.
Thanks
---
### Author:  [@ianhgenesys](https://hub.mender.io/u/ianhgenesys)
#### Post date: [December 20, 2024, 1:42pm UTC](https://hub.mender.io/t/production-installation-for-locally-hosted-mender-server-3-7-4-0-not-working/7414/3 "2024-12-20T13:42:35Z")
Hi @robgio thank you for your swift reply. I will follow the steps for v3.7 again on Monday when I have access to the Linux machine. Would there be alternatives to the Amazon S3 bucket for this version then? Is there documentation for an Azure blob storage or a locally hosted S3-like service?
---
### Author:  [@robgio](https://hub.mender.io/u/robgio)
#### Post date: [December 20, 2024, 1:45pm UTC](https://hub.mender.io/t/production-installation-for-locally-hosted-mender-server-3-7-4-0-not-working/7414/4 "2024-12-20T13:45:55Z")
Yes, sure, you can use any S3 compatible storage, even locally, or even Azure blob storage: [Mender Server | Mender documentation](https://docs.mender.io/3.7/server-installation/production-installation-with-kubernetes/mender-server#supported-artifact-storage-types)
---
### Author:  [@ianhgenesys](https://hub.mender.io/u/ianhgenesys)
#### Post date: [December 23, 2024, 12:00am UTC](https://hub.mender.io/t/production-installation-for-locally-hosted-mender-server-3-7-4-0-not-working/7414/5 "2024-12-23T00:00:52Z")
Hi @robgio, thank you for your input. I managed to get Mender setup with AWS S3 using the command `helm upgrade --install mender mender/mender --version 5.11.2 -f mender-3.7.8.yml --debug`. However, I am still facing some issues trying to connect to the server at [mender.example.com](http://mender.example.com). I have some debug messages that describe some errors but I don’t know how to begin troubleshooting it.
```auto
kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default cert-manager-5464c9ff99-kpg9m 1/1 Running 0 9m33s
default cert-manager-cainjector-658f64986d-lnfbr 1/1 Running 0 9m33s
default cert-manager-webhook-5696ccb5df-mcpj2 1/1 Running 0 9m33s
default mender-api-gateway-75f8dff5b5-xjsqf 1/1 Running 0 7m5s
default mender-create-artifact-worker-5c44889865-vzftz 1/1 Running 2 (6m36s ago) 7m5s
default mender-deployments-577bfdf64d-dx4lz 1/1 Running 0 7m5s
default mender-device-auth-78bcdd687-bljkk 1/1 Running 0 7m5s
default mender-deviceconfig-66967cd7c7-k57lz 1/1 Running 0 7m5s
default mender-deviceconnect-86b6f7565b-vf4d9 1/1 Running 0 7m4s
default mender-gui-7944884849-wbf8n 1/1 Running 0 7m5s
default mender-inventory-6f7df89889-wj2ss 1/1 Running 0 7m5s
default mender-iot-manager-7cc67d98bc-h94jl 1/1 Running 0 7m5s
default mender-mongodb-0 1/1 Running 0 8m56s
default mender-mongodb-1 1/1 Running 0 8m7s
default mender-mongodb-arbiter-0 1/1 Running 0 8m56s
default mender-nats-0 3/3 Running 0 7m5s
default mender-nats-1 3/3 Running 0 7m5s
default mender-nats-2 3/3 Running 0 7m5s
default mender-nats-box-7d447f45b-l8qmm 1/1 Running 0 7m5s
default mender-redis-master-0 1/1 Running 0 8m55s
default mender-redis-replicas-0 1/1 Running 0 8m54s
default mender-redis-replicas-1 1/1 Running 0 8m14s
default mender-redis-replicas-2 1/1 Running 0 7m49s
default mender-useradm-7775ccd87f-6xnqr 1/1 Running 0 7m4s
default mender-workflows-server-968c4fdcc-zth4b 1/1 Running 1 (6m38s ago) 7m5s
default mender-workflows-worker-7b6b94cb87-bk8hc 1/1 Running 2 (6m37s ago) 7m5s
kube-system coredns-ccb96694c-mjwjq 1/1 Running 0 9m33s
kube-system helm-install-traefik-5wwhh 0/1 Completed 1 9m34s
kube-system helm-install-traefik-crd-npghr 0/1 Completed 0 9m34s
kube-system local-path-provisioner-5cf85fd84d-5tlnb 1/1 Running 0 9m33s
kube-system metrics-server-5985cbc9d7-hfvx2 1/1 Running 0 9m33s
kube-system svclb-traefik-fb80b607-wrwkt 2/2 Running 0 9m11s
kube-system traefik-57b79cf995-vfvnh 1/1 Running 0 9m11s
```
```auto
nslookup mender.example.com
Server: 127.0.0.53
Address: 127.0.0.53#53
** server can't find mender.example.com: NXDOMAIN
```
```auto
dig mender.example.com
; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> mender.example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 39495
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;mender.example.com. IN A
;; AUTHORITY SECTION:
example.com. 1775 IN SOA ns.icann.org. noc.dns.icann.org. 2024081476 7200 3600 1209600 3600
;; Query time: 11 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Mon Dec 23 10:55:36 AEDT 2024
;; MSG SIZE rcvd: 103
```
```auto
kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
mender-ingress alb mender.example.com 80, 443 4m3s
```
```auto
kubectl describe ingress
Name: mender-ingress
Labels: app.kubernetes.io/instance=mender
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/part-of=mender
app.kubernetes.io/version=3.7.7
Namespace: default
Address:
Ingress Class: alb
Default backend:
TLS:
mender-ingress-tls terminates mender.example.com
Rules:
Host Path Backends
---- ---- --------
mender.example.com
/ ssl-redirect:use-annotation ()
/ mender-api-gateway:80 (10.42.0.43:9080)
Annotations: alb.ingress.kubernetes.io/actions.ssl-redirect:
{"Type": "redirect", "RedirectConfig":{ "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}
alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/healthcheck-path: /ui/
alb.ingress.kubernetes.io/listen-ports: [{"HTTP": 80}, {"HTTPS":443}]
alb.ingress.kubernetes.io/load-balancer-attributes: routing.http2.enabled=true,idle_timeout.timeout_seconds=600
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
alb.ingress.kubernetes.io/target-type: ip
cert-manager.io/issuer: letsencrypt
meta.helm.sh/release-name: mender
meta.helm.sh/release-namespace: default
```
Would you be able to assist with this please?
---
### Author:  [@ianhgenesys](https://hub.mender.io/u/ianhgenesys)
#### Post date: [December 23, 2024, 6:03am UTC](https://hub.mender.io/t/production-installation-for-locally-hosted-mender-server-3-7-4-0-not-working/7414/6 "2024-12-23T06:03:16Z")
These are my exact steps for this part of the [guide](https://docs.mender.io/3.7/server-installation/production-installation-with-kubernetes/mender-server) (except for the AWS\_ACCESS\_KEY\_ID and AWS\_SECRET\_ACCESS\_KEY):
```auto
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 | openssl rsa -out device_auth.key
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 | openssl rsa -out useradm.key
helm repo add mender https://charts.mender.io
helm repo update
export MENDER_SERVER_DOMAIN="mender.example.com"
export MENDER_SERVER_URL="https://${MENDER_SERVER_DOMAIN}"
export MENDER_VERSION_TAG="mender-3.7.8"
export MONGODB_ROOT_PASSWORD=$(pwgen 32 1)
export MONGODB_REPLICA_SET_KEY=$(pwgen 32 1)
cat >mender-3.7.8.yml <
AWS_SECRET_ACCESS_KEY:
AWS_FORCE_PATH_STYLE: "false"
url: "${MENDER_SERVER_URL}"
# This enables bitnami/mongodb sub-chart
mongodb:
enabled: true
auth:
enabled: true
rootPassword: ${MONGODB_ROOT_PASSWORD}
replicaSetKey: ${MONGODB_REPLICA_SET_KEY}
# This enabled nats sub-chart
nats:
enabled: true
api_gateway:
env:
SSL: false
device_auth:
certs:
key: |-
$(cat device_auth.key | sed -e 's/^/ /g')
useradm:
certs:
key: |-
$(cat useradm.key | sed -e 's/^/ /g')
ingress:
enabled: true
annotations:
cert-manager.io/issuer: "letsencrypt"
alb.ingress.kubernetes.io/healthcheck-path: /ui/
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":{ "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/load-balancer-attributes: routing.http2.enabled=true,idle_timeout.timeout_seconds=600
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
alb.ingress.kubernetes.io/target-type: ip
ingressClassName: alb
path: /
extraPaths:
- path: /
backend:
serviceName: ssl-redirect
servicePort: use-annotation
hosts:
- ${MENDER_SERVER_DOMAIN}
tls:
# this secret must exists or it can be created from a working cert-manager instance
- secretName: mender-ingress-tls
hosts:
- ${MENDER_SERVER_DOMAIN}
EOF
helm upgrade --install mender mender/mender --version 5.11.2 -f mender-3.7.8.yml --debug
```
I’m a bit new to these networking stuff and so I’m not very familiar with why these steps are meant to be done in this specific manner. Is this issue due to the example domain name? What’s the use of the domain name if I want to host this through a local web server like a localhost port? Should I change the [alb.ingress.kubernetes.io/scheme](http://alb.ingress.kubernetes.io/scheme) to internal instead and change the MENDER\_SERVER\_DOMAIN to localhost:8080?
Your assistance is greatly appreciated.
---
### Author:  [@robgio](https://hub.mender.io/u/robgio)
#### Post date: [December 23, 2024, 6:14am UTC](https://hub.mender.io/t/production-installation-for-locally-hosted-mender-server-3-7-4-0-not-working/7414/7 "2024-12-23T06:14:35Z")
Hello @ianhgenesys ,
you have to set a fully functional domain name, not the example one.
But If you want to host it locally only, then you have to switch from the ALB ingress controller to an internal Ingress controller, like Traefik or Ngnix, and modify the /etc/hosts file of your machine by setting, for example:
```auto
mender.example.com
```
The IP should be displayed in your ingress output: `kubectl get ingress`
---
### Author:  [@ianhgenesys](https://hub.mender.io/u/ianhgenesys)
#### Post date: [January 9, 2025, 1:17am UTC](https://hub.mender.io/t/production-installation-for-locally-hosted-mender-server-3-7-4-0-not-working/7414/8 "2025-01-09T01:17:00Z")
Hi @robgio, apologies for so many questions as I am quite new to this and thank you for all the help you have provided thus far. I have now managed to use the Traefik ingress controller to host the server locally. Is this the same as running it on my local WiFi network?
I am trying to connect to the site [mender.example.com](http://mender.example.com) or 192.168.159.156 from another laptop running on the same WiFi network but I keep getting “404 page not found” instead. The output of get ingress is as shown
```auto
kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
mender-ingress traefik mender.example.com 192.168.159.156 80, 443 97m
```
```auto
kubectl describe ingress
Name: mender-ingress
Labels: app.kubernetes.io/instance=mender
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/part-of=mender
app.kubernetes.io/version=3.7.7
Namespace: default
Address: 192.168.159.156
Ingress Class: traefik
Default backend:
TLS:
mender-ingress-tls terminates mender.example.org
Rules:
Host Path Backends
---- ---- --------
mender.example.com
/ mender-api-gateway:80 (10.42.0.41:9080)
Annotations: meta.helm.sh/release-name: mender
meta.helm.sh/release-namespace: default
Events:
```
Finally, I seem to have a lot of trouble with certificate management when trying to run it on a local server, something about how ACME challenges arent passing of some sort. Is this actually needed when running a local server? I seem to be getting a lot of certificate issues such as below:
```auto
kubectl logs traefik-57b79cf995-4948z -n kube-system
time="2025-01-08T23:31:52Z" level=info msg="Configuration loaded from flags."
time="2025-01-08T23:33:21Z" level=error msg="Error configuring TLS: secret default/mender-ingress-tls does not exist" providerName=kubernetes ingress=mender-ingress namespace=default
time="2025-01-08T23:33:21Z" level=error msg="Skipping service: no endpoints found" ingress=mender-ingress namespace=default serviceName=mender-api-gateway servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes
time="2025-01-08T23:33:21Z" level=error msg="Error configuring TLS: secret default/mender-ingress-tls does not exist" providerName=kubernetes ingress=mender-ingress namespace=default
time="2025-01-08T23:33:21Z" level=error msg="Skipping service: no endpoints found" ingress=mender-ingress namespace=default serviceName=mender-api-gateway servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes
time="2025-01-08T23:33:23Z" level=error msg="Error configuring TLS: secret default/mender-ingress-tls does not exist" providerName=kubernetes ingress=mender-ingress namespace=default
time="2025-01-08T23:33:23Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=mender-ingress namespace=default serviceName=mender-api-gateway servicePort="&ServiceBackendPort{Name:,Number:80,}"
```
Would you be able to assist with this please?
---
### Author:  [@robgio](https://hub.mender.io/u/robgio)
#### Post date: [January 21, 2025, 5:33am UTC](https://hub.mender.io/t/production-installation-for-locally-hosted-mender-server-3-7-4-0-not-working/7414/9 "2025-01-21T05:33:00Z")
Hello @ianhgenesys ,
a network test you can try on your laptop, is just to expose a TCP port on the WiFi network interface and see if you can reach from another laptop; this way you know for sure if your network setup is fine.
Talking about the certificate, you cannot get one for [example.com](http://example.com) from let’sencrypt; I would recommend using a free DNS provider (for example: [https://www.cloudns.net/](https://www.cloudns.net/)) suitable for testing. This way you have a temporay fqdn, the DNS, and you can get your certificate.
Cheers