Mender-conductor:1.3.0b1 error

General Discussions
1

Hi Team,

I am trying to use the new docker container for conductor, but I get this error.

I don’t see the variables in https://github.com/mendersoftware/integration/blob/2.0.0b1/conductor/server/config/config.properties

Nor do I see how I can disable this. Any suggestions would be welcome :slight_smile:

[2019-04-03 20:56:54,236] [main] INFO com.netflix.conductor.bootstrap.ModulesProvider - External payload storage is not configured, provided: , supported values are: [S3]
java.lang.IllegalArgumentException: No enum constant com.netflix.conductor.bootstrap.ModulesProvider.ExternalPayloadStorageType.
at java.lang.Enum.valueOf(Enum.java:238)
at com.netflix.conductor.bootstrap.ModulesProvider$ExternalPayloadStorageType.valueOf(ModulesProvider.java:38)
at com.netflix.conductor.bootstrap.ModulesProvider.selectModulesToLoad(ModulesProvider.java:111)
at com.netflix.conductor.bootstrap.ModulesProvider.get(ModulesProvider.java:51)
at com.netflix.conductor.bootstrap.Main.main(Main.java:53)

2

Hi @gil0109,

Do you have any problems running the server? We have seen conductor throwing errors like these but they have typically not impacted functionality, and this is most likely us not using certain parts of that component.

3

Short answer is yes… may not be related but the conductor keeps failing to start up as well. This is not an issue with v1.2

– conductor API is at http://localhost:8080
Starting Conductor server
Property file: config.properties
config.properties
Using ‘config.properties’
– Thu Apr 4 16:09:41 UTC 2019 waiting for conductor, attempt 1
Using config file/app/config/config.properties
[2019-04-04 16:09:43,701] [main] INFO com.netflix.conductor.bootstrap.ModulesProvider - Starting conductor server using dynomite/redis cluster.
[2019-04-04 16:09:43,764] [main] INFO com.netflix.conductor.bootstrap.ModulesProvider - External payload storage is not configured, provided: , supported values are: [S3]
java.lang.IllegalArgumentException: No enum constant com.netflix.conductor.bootstrap.ModulesProvider.ExternalPayloadStorageType.
at java.lang.Enum.valueOf(Enum.java:238)
at com.netflix.conductor.bootstrap.ModulesProvider$ExternalPayloadStorageType.valueOf(ModulesProvider.java:38)
at com.netflix.conductor.bootstrap.ModulesProvider.selectModulesToLoad(ModulesProvider.java:111)
at com.netflix.conductor.bootstrap.ModulesProvider.get(ModulesProvider.java:51)
at com.netflix.conductor.bootstrap.Main.main(Main.java:53)
[2019-04-04 16:09:44,643] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: HTTP
[2019-04-04 16:09:45,071] [main] INFO com.netflix.conductor.contribs.http.HttpTask - HttpTask initialized…
[2019-04-04 16:09:45,073] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: JSON_JQ_TRANSFORM
– Thu Apr 4 16:09:46 UTC 2019 waiting for conductor, attempt 2
[2019-04-04 16:09:47,243] [main] INFO com.netflix.archaius.guice.ConfigurationInjectingListener - Initializing ConfigurationInjectingListener
[2019-04-04 16:09:47,581] [main] INFO com.netflix.dyno.jedis.DynoJedisClient - Starting connection pool for app conductor
[2019-04-04 16:09:47,582] [main] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Getting Hosts from TokenMapSupplier
[2019-04-04 16:09:47,587] [pool-3-thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Adding host connection pool for host: Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-04 16:09:47,587] [pool-3-thread-1] INFO com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl - Priming connection pool for host:Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null], with conns:10
[2019-04-04 16:09:47,709] [pool-3-thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Successfully primed 10 of 10 to Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-04 16:09:47,720] [main] INFO com.netflix.dyno.connectionpool.TokenPoolTopology - Adding Host to TopologyHost [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-04 16:09:47,725] [main] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - registered mbean com.netflix.dyno.connectionpool.impl:type=MonitorConsole
[2019-04-04 16:09:47,733] [main] INFO com.netflix.dyno.jedis.DynoJedisClient - Starting connection pool for app conductor
[2019-04-04 16:09:47,733] [main] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Getting Hosts from TokenMapSupplier
[2019-04-04 16:09:47,733] [pool-6-thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Adding host connection pool for host: Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-04 16:09:47,734] [pool-6-thread-1] INFO com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl - Priming connection pool for host:Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null], with conns:10
[2019-04-04 16:09:47,741] [pool-6-thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Successfully primed 10 of 10 to Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-04 16:09:47,742] [main] INFO com.netflix.dyno.connectionpool.TokenPoolTopology - Adding Host to TopologyHost [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-04 16:09:47,742] [main] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - mbean com.netflix.dyno.connectionpool.impl:type=MonitorConsole has already been registered !
[2019-04-04 16:09:47,749] [main] INFO com.netflix.conductor.dyno.RedisQueuesProvider - DynoQueueDAO initialized with prefix conductor_queues.test!
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property ‘log4j2.debug’ to show Log4j2 internal initialization logging.
– Thu Apr 4 16:09:51 UTC 2019 waiting for conductor, attempt 3
[2019-04-04 16:09:52,657] [main] INFO com.netflix.conductor.core.events.EventProcessor - Event Processing is ENABLED. executorThreadCount set to 2
[2019-04-04 16:09:52,659] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - System Task Worker initialized with 10 threads and a callback time of 30 seconds and queue size: 100 with pollCount: 10 and poll interval: 50
[2019-04-04 16:09:52,660] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: SUB_WORKFLOW
[2019-04-04 16:09:52,660] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: WAIT
[2019-04-04 16:09:52,660] [Thread-3] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Started listening for system task: HTTP
[2019-04-04 16:09:52,660] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: EVENT
[2019-04-04 16:09:52,667] [main] INFO com.netflix.conductor.elasticsearch.EmbeddedElasticSearch - Deleting contents of data dir /tmp/elasticsearch_test_1554394192663data2779191640537963547
[2019-04-04 16:09:52,675] [main] INFO com.netflix.conductor.elasticsearch.es5.EmbeddedElasticSearchV5 - Starting ElasticSearch for cluster elasticsearch_test
[2019-04-04 16:09:53,739] [pool-13-thread-1] INFO com.netflix.dyno.queues.redis.RedisDynoQueue - com.netflix.dyno.queues.redis.RedisDynoQueue is ready to serve HTTP
– Thu Apr 4 16:09:56 UTC 2019 waiting for conductor, attempt 4
[2019-04-04 16:09:57,914] [main] INFO com.netflix.conductor.elasticsearch.es5.EmbeddedElasticSearchV5 - ElasticSearch cluster elasticsearch_test started in local mode on port 9200
– Thu Apr 4 16:10:01 UTC 2019 waiting for conductor, attempt 5
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{vwWSSrS-S8S9wsl-901ekQ}{mender-elasticsearch}{172.50.11.245:9300}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:366)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
at com.netflix.conductor.dao.es5.index.ElasticSearchDAOV5.setup(ElasticSearchDAOV5.java:149)
at com.netflix.conductor.bootstrap.Main.main(Main.java:71)
[2019-04-04 16:10:02,919] [Thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Removing host mender-redis from selectionStrategy, cpHealthTracker, cpMonitor
[2019-04-04 16:10:02,919] [Thread-2] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Removing host mender-redis from selectionStrategy, cpHealthTracker, cpMonitor
[2019-04-04 16:10:02,919] [Thread-2] INFO com.netflix.dyno.connectionpool.TokenPoolTopology - Removing Host from TopologyHost [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-04 16:10:02,919] [Thread-1] INFO com.netflix.dyno.connectionpool.TokenPoolTopology - Removing Host from TopologyHost [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-04 16:10:02,919] [Thread-2] INFO com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl - Shutting down connection pool for host:Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-04 16:10:02,919] [Thread-1] INFO com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl - Shutting down connection pool for host:Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-04 16:10:02,926] [Thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Remove host: Successfully removed hostPool for host mender-redis from connection pool
[2019-04-04 16:10:02,927] [Thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - deregistered mbean com.netflix.dyno.connectionpool.impl:type=MonitorConsole
[2019-04-04 16:10:02,935] [Thread-2] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Remove host: Successfully removed hostPool for host mender-redis from connection pool
– Thu Apr 4 16:10:06 UTC 2019 waiting for conductor, attempt 6
– Thu Apr 4 16:10:11 UTC 2019 waiting for conductor, attempt 7
– Thu Apr 4 16:10:16 UTC 2019 waiting for conductor, attempt 8
– Thu Apr 4 16:10:21 UTC 2019 waiting for conductor, attempt 9
– Thu Apr 4 16:10:26 UTC 2019 waiting for conductor, attempt 10
– Thu Apr 4 16:10:31 UTC 2019 waiting for conductor, attempt 11
– Thu Apr 4 16:10:36 UTC 2019 waiting for conductor, attempt 12
– Thu Apr 4 16:10:41 UTC 2019 waiting for conductor, attempt 13
– Thu Apr 4 16:10:46 UTC 2019 waiting for conductor, attempt 14
– Thu Apr 4 16:10:51 UTC 2019 waiting for conductor, attempt 15
– Thu Apr 4 16:10:56 UTC 2019 waiting for conductor, attempt 16
– Thu Apr 4 16:11:01 UTC 2019 waiting for conductor, attempt 17
– Thu Apr 4 16:11:06 UTC 2019 waiting for conductor, attempt 18
– Thu Apr 4 16:11:11 UTC 2019 waiting for conductor, attempt 19
– Thu Apr 4 16:11:16 UTC 2019 waiting for conductor, attempt 20
– Thu Apr 4 16:11:21 UTC 2019 conductor still down, exiting

4

Hi @gil0109
Based on the logs format I assume it’s conductor v.2.3.0 which is currently in master but in case if mender backend version 1.3.0b1 is used then conductor v.1.8.1 should be in use. Master conductor version won’t work with 1.3 backend version.

5

Correction: conductor v.1.8.1 came with Mender 1.4. Mender 1.3 used conductor v.1.7.7.

I am trying to use the new docker container for conductor, but I get this error.

Please also note that mender-conductor images includes tasks and workflows which are mender version specific. If you just pull never version of mender-conductor then most likely it won’t work.

If you want to update then you have to update all services completely from Mender 1.3 to Mender 1.4, partial updates in most cases won’t work.

6

He’s talking about mender-conductor 1.3, not Mender 1.3. Mender-conductor 1.3.0b1 is indeed the latest one, and comes with Mender 2.0.0.

@gil0109, are any other services not working correctly? What’s the output of

cd integration
./demo ps
7

All other services are working correctly with no errors. Just mendersoftware/mender-conductor:1.3.0b1 is failing.

Upgrade to the following:

  1. mendersoftware/api-gateway:1.6
  2. mendersoftware/deployments:1.7.0b1
  3. mendersoftware/deviceauth:2.0.0b1
  4. mendersoftware/gui:2.0.0b1
  5. mendersoftware/inventory:1.6.0b1
  6. mendersoftware/useradm:1.7

Minio/ElasticSearch/Redis/mongodb remains the same.

8

@gil0109
mender-elasticsearch and mender-redis are used only by mender-conductor and it depends from their versions.
mender-conductor:1.3.0b1 won’t work with ES:2.4 because it requires ES:5, so, you have to change all 3 of mender-conductor, mender-elasticsearch and mender-redis in one go.

9

I updated both mender-redis & mender-elasticsearch and still have the same issue.

– conductor API is at http://localhost:8080
Starting Conductor server
Property file: config.properties
config.properties
Using ‘config.properties’
– Tue Apr 9 15:51:15 UTC 2019 waiting for conductor, attempt 1
Using config file/app/config/config.properties
[2019-04-09 15:51:16,717] [main] INFO com.netflix.conductor.bootstrap.ModulesProvider - Starting conductor server using dynomite/redis cluster.
[2019-04-09 15:51:16,728] [main] INFO com.netflix.conductor.bootstrap.ModulesProvider - External payload storage is not configured, provided: , supported values are: [S3]
java.lang.IllegalArgumentException: No enum constant com.netflix.conductor.bootstrap.ModulesProvider.ExternalPayloadStorageType.
at java.lang.Enum.valueOf(Enum.java:238)
at com.netflix.conductor.bootstrap.ModulesProvider$ExternalPayloadStorageType.valueOf(ModulesProvider.java:38)
at com.netflix.conductor.bootstrap.ModulesProvider.selectModulesToLoad(ModulesProvider.java:111)
at com.netflix.conductor.bootstrap.ModulesProvider.get(ModulesProvider.java:51)
at com.netflix.conductor.bootstrap.Main.main(Main.java:53)
[2019-04-09 15:51:17,408] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: HTTP
[2019-04-09 15:51:17,834] [main] INFO com.netflix.conductor.contribs.http.HttpTask - HttpTask initialized…
[2019-04-09 15:51:17,884] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: JSON_JQ_TRANSFORM
[2019-04-09 15:51:19,518] [main] INFO com.netflix.archaius.guice.ConfigurationInjectingListener - Initializing ConfigurationInjectingListener
[2019-04-09 15:51:19,887] [main] INFO com.netflix.dyno.jedis.DynoJedisClient - Starting connection pool for app conductor
[2019-04-09 15:51:19,888] [main] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Getting Hosts from TokenMapSupplier
[2019-04-09 15:51:19,893] [pool-3-thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Adding host connection pool for host: Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-09 15:51:19,894] [pool-3-thread-1] INFO com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl - Priming connection pool for host:Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null], with conns:10
[2019-04-09 15:51:19,981] [pool-3-thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Successfully primed 10 of 10 to Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-09 15:51:19,993] [main] INFO com.netflix.dyno.connectionpool.TokenPoolTopology - Adding Host to TopologyHost [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-09 15:51:19,997] [main] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - registered mbean com.netflix.dyno.connectionpool.impl:type=MonitorConsole
[2019-04-09 15:51:20,004] [main] INFO com.netflix.dyno.jedis.DynoJedisClient - Starting connection pool for app conductor
[2019-04-09 15:51:20,004] [main] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Getting Hosts from TokenMapSupplier
[2019-04-09 15:51:20,004] [pool-6-thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Adding host connection pool for host: Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-09 15:51:20,005] [pool-6-thread-1] INFO com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl - Priming connection pool for host:Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null], with conns:10
[2019-04-09 15:51:20,009] [pool-6-thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Successfully primed 10 of 10 to Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-09 15:51:20,010] [main] INFO com.netflix.dyno.connectionpool.TokenPoolTopology - Adding Host to TopologyHost [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-09 15:51:20,010] [main] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - mbean com.netflix.dyno.connectionpool.impl:type=MonitorConsole has already been registered !
[2019-04-09 15:51:20,020] [main] INFO com.netflix.conductor.dyno.RedisQueuesProvider - DynoQueueDAO initialized with prefix conductor_queues.test!
– Tue Apr 9 15:51:20 UTC 2019 waiting for conductor, attempt 2
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property ‘log4j2.debug’ to show Log4j2 internal initialization logging.
[2019-04-09 15:51:24,821] [main] INFO com.netflix.conductor.core.events.EventProcessor - Event Processing is ENABLED. executorThreadCount set to 2
[2019-04-09 15:51:24,823] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - System Task Worker initialized with 10 threads and a callback time of 30 seconds and queue size: 100 with pollCount: 10 and poll interval: 50
[2019-04-09 15:51:24,823] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: SUB_WORKFLOW
[2019-04-09 15:51:24,829] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: WAIT
[2019-04-09 15:51:24,830] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: EVENT
[2019-04-09 15:51:24,830] [Thread-3] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Started listening for system task: HTTP
[2019-04-09 15:51:24,837] [main] INFO com.netflix.conductor.elasticsearch.EmbeddedElasticSearch - Deleting contents of data dir /tmp/elasticsearch_test_1554825084833data8389140209658292596
[2019-04-09 15:51:24,845] [main] INFO com.netflix.conductor.elasticsearch.es5.EmbeddedElasticSearchV5 - Starting ElasticSearch for cluster elasticsearch_test
– Tue Apr 9 15:51:25 UTC 2019 waiting for conductor, attempt 3
[2019-04-09 15:51:25,908] [pool-13-thread-1] INFO com.netflix.dyno.queues.redis.RedisDynoQueue - com.netflix.dyno.queues.redis.RedisDynoQueue is ready to serve HTTP
[2019-04-09 15:51:30,185] [main] INFO com.netflix.conductor.elasticsearch.es5.EmbeddedElasticSearchV5 - ElasticSearch cluster elasticsearch_test started in local mode on port 9200
– Tue Apr 9 15:51:30 UTC 2019 waiting for conductor, attempt 4
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{bUokLCcCRtOCA5JYBe1qdg}{mender-elasticsearch}{172.50.11.245:9300}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:366)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
at com.netflix.conductor.dao.es5.index.ElasticSearchDAOV5.setup(ElasticSearchDAOV5.java:149)
at com.netflix.conductor.bootstrap.Main.main(Main.java:71)
[2019-04-09 15:51:35,191] [Thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Removing host mender-redis from selectionStrategy, cpHealthTracker, cpMonitor
[2019-04-09 15:51:35,192] [Thread-2] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Removing host mender-redis from selectionStrategy, cpHealthTracker, cpMonitor
[2019-04-09 15:51:35,192] [Thread-1] INFO com.netflix.dyno.connectionpool.TokenPoolTopology - Removing Host from TopologyHost [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-09 15:51:35,192] [Thread-2] INFO com.netflix.dyno.connectionpool.TokenPoolTopology - Removing Host from TopologyHost [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-09 15:51:35,192] [Thread-1] INFO com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl - Shutting down connection pool for host:Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-09 15:51:35,192] [Thread-2] INFO com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl - Shutting down connection pool for host:Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-04-09 15:51:35,198] [Thread-2] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Remove host: Successfully removed hostPool for host mender-redis from connection pool
[2019-04-09 15:51:35,198] [Thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Remove host: Successfully removed hostPool for host mender-redis from connection pool
[2019-04-09 15:51:35,198] [Thread-2] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - deregistered mbean com.netflix.dyno.connectionpool.impl:type=MonitorConsole
– Tue Apr 9 15:51:35 UTC 2019 waiting for conductor, attempt 5
– Tue Apr 9 15:51:40 UTC 2019 waiting for conductor, attempt 6
– Tue Apr 9 15:51:45 UTC 2019 waiting for conductor, attempt 7
– Tue Apr 9 15:51:50 UTC 2019 waiting for conductor, attempt 8
– Tue Apr 9 15:51:55 UTC 2019 waiting for conductor, attempt 9
– Tue Apr 9 15:52:00 UTC 2019 waiting for conductor, attempt 10
– Tue Apr 9 15:52:05 UTC 2019 waiting for conductor, attempt 11
– Tue Apr 9 15:52:10 UTC 2019 waiting for conductor, attempt 12
– Tue Apr 9 15:52:15 UTC 2019 waiting for conductor, attempt 13
– Tue Apr 9 15:52:20 UTC 2019 waiting for conductor, attempt 14
– Tue Apr 9 15:52:25 UTC 2019 waiting for conductor, attempt 15
– Tue Apr 9 15:52:30 UTC 2019 waiting for conductor, attempt 16
– Tue Apr 9 15:52:35 UTC 2019 waiting for conductor, attempt 17

10

@gil0109 can you please share logs from mender-elasticsearch as well?

11

[2019-04-09T15:40:12,794][INFO ][o.e.n.Node ] [] initializing …
[2019-04-09T15:40:13,120][INFO ][o.e.e.NodeEnvironment ] [9ku_hwz] using [1] data paths, mounts [[/usr/share/elasticsearch/data (192.168.111.160:vol_053f6cb83825341a571a09057fa3f5c8)]], net usable_space [971.1mb], net total_space [1014mb], spins? [possibly], types [fuse.glusterfs]
[2019-04-09T15:40:13,120][INFO ][o.e.e.NodeEnvironment ] [9ku_hwz] heap size [503.6mb], compressed ordinary object pointers [true]
[2019-04-09T15:40:13,127][INFO ][o.e.n.Node ] node name [9ku_hwz] derived from node ID [9ku_hwzfRpaIzmufRDrQzQ]; set [node.name] to override
[2019-04-09T15:40:13,127][INFO ][o.e.n.Node ] version[5.6.16], pid[1], build[3a740d1/2019-03-13T15:33:36.565Z], OS[Linux/3.10.0-957.5.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_201/25.201-b08]
[2019-04-09T15:40:13,128][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch]
[2019-04-09T15:40:15,503][INFO ][o.e.p.PluginsService ] [9ku_hwz] loaded module [aggs-matrix-stats]
[2019-04-09T15:40:15,504][INFO ][o.e.p.PluginsService ] [9ku_hwz] loaded module [ingest-common]
[2019-04-09T15:40:15,504][INFO ][o.e.p.PluginsService ] [9ku_hwz] loaded module [lang-expression]
[2019-04-09T15:40:15,504][INFO ][o.e.p.PluginsService ] [9ku_hwz] loaded module [lang-groovy]
[2019-04-09T15:40:15,504][INFO ][o.e.p.PluginsService ] [9ku_hwz] loaded module [lang-mustache]
[2019-04-09T15:40:15,504][INFO ][o.e.p.PluginsService ] [9ku_hwz] loaded module [lang-painless]
[2019-04-09T15:40:15,504][INFO ][o.e.p.PluginsService ] [9ku_hwz] loaded module [parent-join]
[2019-04-09T15:40:15,504][INFO ][o.e.p.PluginsService ] [9ku_hwz] loaded module [percolator]
[2019-04-09T15:40:15,504][INFO ][o.e.p.PluginsService ] [9ku_hwz] loaded module [reindex]
[2019-04-09T15:40:15,504][INFO ][o.e.p.PluginsService ] [9ku_hwz] loaded module [transport-netty3]
[2019-04-09T15:40:15,504][INFO ][o.e.p.PluginsService ] [9ku_hwz] loaded module [transport-netty4]
[2019-04-09T15:40:15,505][INFO ][o.e.p.PluginsService ] [9ku_hwz] no plugins loaded
[2019-04-09T15:40:20,278][INFO ][o.e.d.DiscoveryModule ] [9ku_hwz] using discovery type [single-node]
[2019-04-09T15:40:21,705][INFO ][o.e.n.Node ] initialized
[2019-04-09T15:40:21,706][INFO ][o.e.n.Node ] [9ku_hwz] starting …
[2019-04-09T15:40:22,207][INFO ][o.e.t.TransportService ] [9ku_hwz] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2019-04-09T15:40:22,276][WARN ][o.e.b.BootstrapChecks ] [9ku_hwz] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2019-04-09T15:40:22,301][INFO ][o.e.c.s.ClusterService ] [9ku_hwz] new_master {9ku_hwz}{9ku_hwzfRpaIzmufRDrQzQ}{XyPkUXY0SjqSxzVGJ7knxQ}{127.0.0.1}{127.0.0.1:9300}, reason: single-node-start-initial-join[{9ku_hwz}{9ku_hwzfRpaIzmufRDrQzQ}{XyPkUXY0SjqSxzVGJ7knxQ}{127.0.0.1}{127.0.0.1:9300}]
[2019-04-09T15:40:22,390][INFO ][o.e.h.n.Netty4HttpServerTransport] [9ku_hwz] publish_address {172.51.83.19:9200}, bound_addresses {[::]:9200}
[2019-04-09T15:40:22,390][INFO ][o.e.n.Node ] [9ku_hwz] started
[2019-04-09T15:40:22,503][INFO ][o.e.g.GatewayService ] [9ku_hwz] recovered [0] indices into cluster_state

12

I am really sorry, I figured it out. I had the wrong settings in the elasticsearch.yml file:

I had: http.host: 0.0.0.0
But your team has: network.host: 0.0.0.0

Again, thank you for your patience and time on this issue. I appreciate you pointing me to the elasticsearch for this. This setting wasn’t in the previous version.