We recently upgraded from Mender 1.7.0 to Mender 2.0.1. We’ve intermittently had problems with conductor since the upgrade. In communications with the mender team, it was suggested that the problem was with elasticsearch needing a kernel parameter set with sysctl as was described in a warning message from elastic search. As a result I ran the recommended change:
sysctl -w vm.max_map_count=294912
It appears that elasticsearch is not restarting now but conductor is still in a reboot loop and not working.
The problem of conductor rebooting is similar to the one outlined in the documentation at https://docs.mender.io/development/troubleshooting/mender-server however it appears that the cause is not the same and the suggested fix of changing java vm parameters has been verified not to work.
Here is the log for elasticsearch:
[2019-07-30T19:01:37,701][INFO ][o.e.n.Node ] [] initializing ...
[2019-07-30T19:01:37,966][WARN ][o.e.d.e.NodeEnvironment ] ES has detected the [path.data] folder using the cluster name as a folder [/usr/share/elasticsearch/data], Elasticsearch 6.0 will not allow the cluster name as a folder within the data path
[2019-07-30T19:01:38,346][INFO ][o.e.e.NodeEnvironment ] [y3iuu7d] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/nvme0n1p1)]], net usable_space [5.4gb], net total_space [15.4gb], spins? [possibly], types [ext4]
[2019-07-30T19:01:38,347][INFO ][o.e.e.NodeEnvironment ] [y3iuu7d] heap size [1.9gb], compressed ordinary object pointers [true]
[2019-07-30T19:01:38,508][INFO ][o.e.n.Node ] node name [y3iuu7d] derived from node ID [y3iuu7dwSOqfBtt12iHCAw]; set [node.name] to override
[2019-07-30T19:01:38,508][INFO ][o.e.n.Node ] version[5.6.16], pid[1], build[3a740d1/2019-03-13T15:33:36.565Z], OS[Linux/4.15.0-1043-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_201/25.201-b08]
[2019-07-30T19:01:38,508][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2019-07-30T19:01:42,721][INFO ][o.e.p.PluginsService ] [y3iuu7d] loaded module [aggs-matrix-stats]
[2019-07-30T19:01:42,733][INFO ][o.e.p.PluginsService ] [y3iuu7d] loaded module [ingest-common]
[2019-07-30T19:01:42,733][INFO ][o.e.p.PluginsService ] [y3iuu7d] loaded module [lang-expression]
[2019-07-30T19:01:42,733][INFO ][o.e.p.PluginsService ] [y3iuu7d] loaded module [lang-groovy]
[2019-07-30T19:01:42,733][INFO ][o.e.p.PluginsService ] [y3iuu7d] loaded module [lang-mustache]
[2019-07-30T19:01:42,733][INFO ][o.e.p.PluginsService ] [y3iuu7d] loaded module [lang-painless]
[2019-07-30T19:01:42,733][INFO ][o.e.p.PluginsService ] [y3iuu7d] loaded module [parent-join]
[2019-07-30T19:01:42,733][INFO ][o.e.p.PluginsService ] [y3iuu7d] loaded module [percolator]
[2019-07-30T19:01:42,733][INFO ][o.e.p.PluginsService ] [y3iuu7d] loaded module [reindex]
[2019-07-30T19:01:42,734][INFO ][o.e.p.PluginsService ] [y3iuu7d] loaded module [transport-netty3]
[2019-07-30T19:01:42,734][INFO ][o.e.p.PluginsService ] [y3iuu7d] loaded module [transport-netty4]
[2019-07-30T19:01:42,734][INFO ][o.e.p.PluginsService ] [y3iuu7d] no plugins loaded
[2019-07-30T19:01:50,467][INFO ][o.e.d.DiscoveryModule ] [y3iuu7d] using discovery type [zen]
[2019-07-30T19:01:52,960][INFO ][o.e.n.Node ] initialized
[2019-07-30T19:01:52,961][INFO ][o.e.n.Node ] [y3iuu7d] starting ...
[2019-07-30T19:01:53,605][INFO ][o.e.t.TransportService ] [y3iuu7d] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2019-07-30T19:01:54,163][INFO ][o.e.m.j.JvmGcMonitorService] [y3iuu7d] [gc][1] overhead, spent [293ms] collecting in the last [1.1s]
[2019-07-30T19:01:56,881][INFO ][o.e.c.s.ClusterService ] [y3iuu7d] new_master {y3iuu7d}{y3iuu7dwSOqfBtt12iHCAw}{on0RQATlRyOYRFPtyANneQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2019-07-30T19:01:56,957][INFO ][o.e.h.n.Netty4HttpServerTransport] [y3iuu7d] publish_address {192.168.64.3:9200}, bound_addresses {0.0.0.0:9200}
[2019-07-30T19:01:56,957][INFO ][o.e.n.Node ] [y3iuu7d] started
[2019-07-30T19:01:57,286][WARN ][o.e.d.i.m.TypeParsers ] field [include_in_all] is deprecated, as [_all] is deprecated, and will be disallowed in 6.0, use [copy_to] instead.
[2019-07-30T19:01:57,536][INFO ][o.e.g.GatewayService ] [y3iuu7d] recovered [5] indices into cluster_state
[2019-07-30T19:01:58,733][WARN ][o.e.d.i.m.TypeParsers ] field [include_in_all] is deprecated, as [_all] is deprecated, and will be disallowed in 6.0, use [copy_to] instead.
[2019-07-30T19:01:59,344][INFO ][o.e.c.r.a.AllocationService] [y3iuu7d] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[conductor][0]] ...]).Here is the log for Conductor:
-- conductor API is at http://localhost:8080
Starting Conductor server
Property file: config.properties
config.properties
Using 'config.properties'
-- Tue Jul 30 19:01:35 UTC 2019 waiting for conductor, attempt 1
Using config file/app/config/config.properties
[2019-07-30 19:01:38,127] [main] INFO com.netflix.conductor.bootstrap.ModulesProvider - Starting conductor server using dynomite/redis cluster.
[2019-07-30 19:01:38,142] [main] INFO com.netflix.conductor.bootstrap.ModulesProvider - External payload storage is not configured, provided: , supported values are: [S3]
java.lang.IllegalArgumentException: No enum constant com.netflix.conductor.bootstrap.ModulesProvider.ExternalPayloadStorageType.
at java.lang.Enum.valueOf(Enum.java:238)
at com.netflix.conductor.bootstrap.ModulesProvider$ExternalPayloadStorageType.valueOf(ModulesProvider.java:38)
at com.netflix.conductor.bootstrap.ModulesProvider.selectModulesToLoad(ModulesProvider.java:111)
at com.netflix.conductor.bootstrap.ModulesProvider.get(ModulesProvider.java:51)
at com.netflix.conductor.bootstrap.Main.main(Main.java:53)
[2019-07-30 19:01:39,796] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: HTTP
[2019-07-30 19:01:40,395] [main] INFO com.netflix.conductor.contribs.http.HttpTask - HttpTask initialized...
[2019-07-30 19:01:40,396] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: JSON_JQ_TRANSFORM
-- Tue Jul 30 19:01:40 UTC 2019 waiting for conductor, attempt 2
[2019-07-30 19:01:42,947] [main] INFO com.netflix.archaius.guice.ConfigurationInjectingListener - Initializing ConfigurationInjectingListener
[2019-07-30 19:01:43,472] [main] INFO com.netflix.dyno.jedis.DynoJedisClient - Starting connection pool for app conductor
[2019-07-30 19:01:43,472] [main] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Getting Hosts from TokenMapSupplier
[2019-07-30 19:01:43,504] [pool-3-thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Adding host connection pool for host: Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:01:43,505] [pool-3-thread-1] INFO com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl - Priming connection pool for host:Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null], with conns:10
[2019-07-30 19:01:43,789] [pool-3-thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Successfully primed 10 of 10 to Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:01:43,815] [main] INFO com.netflix.dyno.connectionpool.TokenPoolTopology - Adding Host to TopologyHost [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:01:43,820] [main] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - registered mbean com.netflix.dyno.connectionpool.impl:type=MonitorConsole
[2019-07-30 19:01:43,852] [main] INFO com.netflix.dyno.jedis.DynoJedisClient - Starting connection pool for app conductor
[2019-07-30 19:01:43,852] [main] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Getting Hosts from TokenMapSupplier
[2019-07-30 19:01:43,865] [pool-6-thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Adding host connection pool for host: Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:01:43,866] [pool-6-thread-1] INFO com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl - Priming connection pool for host:Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null], with conns:10
[2019-07-30 19:01:43,869] [pool-6-thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Successfully primed 10 of 10 to Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:01:43,882] [main] INFO com.netflix.dyno.connectionpool.TokenPoolTopology - Adding Host to TopologyHost [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:01:43,883] [main] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - mbean com.netflix.dyno.connectionpool.impl:type=MonitorConsole has already been registered !
[2019-07-30 19:01:43,895] [main] INFO com.netflix.conductor.dyno.RedisQueuesProvider - DynoQueueDAO initialized with prefix conductor_queues.test!
-- Tue Jul 30 19:01:45 UTC 2019 waiting for conductor, attempt 3
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
-- Tue Jul 30 19:01:50 UTC 2019 waiting for conductor, attempt 4
[2019-07-30 19:01:51,746] [main] INFO com.netflix.conductor.core.events.EventProcessor - Event Processing is ENABLED. executorThreadCount set to 2
[2019-07-30 19:01:51,752] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - System Task Worker initialized with 10 threads and a callback time of 30 seconds and queue size: 100 with pollCount: 10 and poll interval: 50
[2019-07-30 19:01:51,758] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: SUB_WORKFLOW
[2019-07-30 19:01:51,759] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: WAIT
[2019-07-30 19:01:51,759] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: EVENT
[2019-07-30 19:01:51,765] [main] INFO com.netflix.conductor.elasticsearch.EmbeddedElasticSearch - Deleting contents of data dir /tmp/elasticsearch_test_1564513311762data5571683238847122862
[2019-07-30 19:01:51,766] [Thread-3] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Started listening for system task: HTTP
[2019-07-30 19:01:51,803] [main] INFO com.netflix.conductor.elasticsearch.es5.EmbeddedElasticSearchV5 - Starting ElasticSearch for cluster elasticsearch_test
[2019-07-30 19:01:52,943] [pool-13-thread-1] INFO com.netflix.dyno.queues.redis.RedisDynoQueue - com.netflix.dyno.queues.redis.RedisDynoQueue is ready to serve HTTP
-- Tue Jul 30 19:01:55 UTC 2019 waiting for conductor, attempt 5
[2019-07-30 19:01:58,364] [main] INFO com.netflix.conductor.elasticsearch.es5.EmbeddedElasticSearchV5 - ElasticSearch cluster elasticsearch_test started in local mode on port 9200
-- Tue Jul 30 19:02:00 UTC 2019 waiting for conductor, attempt 6
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{LcgP-N4rTzGNxBypaLQO_w}{mender-elasticsearch}{192.168.64.3:9300}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:366)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
at com.netflix.conductor.dao.es5.index.ElasticSearchDAOV5.setup(ElasticSearchDAOV5.java:149)
at com.netflix.conductor.bootstrap.Main.main(Main.java:71)
[2019-07-30 19:02:03,375] [Thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Removing host mender-redis from selectionStrategy, cpHealthTracker, cpMonitor
[2019-07-30 19:02:03,376] [Thread-2] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Removing host mender-redis from selectionStrategy, cpHealthTracker, cpMonitor
[2019-07-30 19:02:03,375] [Thread-1] INFO com.netflix.dyno.connectionpool.TokenPoolTopology - Removing Host from TopologyHost [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:02:03,382] [Thread-1] INFO com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl - Shutting down connection pool for host:Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:02:03,377] [Thread-2] INFO com.netflix.dyno.connectionpool.TokenPoolTopology - Removing Host from TopologyHost [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:02:03,389] [Thread-2] INFO com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl - Shutting down connection pool for host:Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:02:03,394] [Thread-2] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Remove host: Successfully removed hostPool for host mender-redis from connection pool
[2019-07-30 19:02:03,398] [Thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Remove host: Successfully removed hostPool for host mender-redis from connection pool
[2019-07-30 19:02:03,399] [Thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - deregistered mbean com.netflix.dyno.connectionpool.impl:type=MonitorConsole
-- Tue Jul 30 19:02:05 UTC 2019 waiting for conductor, attempt 7
-- Tue Jul 30 19:02:10 UTC 2019 waiting for conductor, attempt 8
-- Tue Jul 30 19:02:15 UTC 2019 waiting for conductor, attempt 9
-- Tue Jul 30 19:02:20 UTC 2019 waiting for conductor, attempt 10
-- Tue Jul 30 19:02:25 UTC 2019 waiting for conductor, attempt 11
-- Tue Jul 30 19:02:30 UTC 2019 waiting for conductor, attempt 12
-- Tue Jul 30 19:02:35 UTC 2019 waiting for conductor, attempt 13
-- Tue Jul 30 19:02:40 UTC 2019 waiting for conductor, attempt 14
-- Tue Jul 30 19:02:45 UTC 2019 waiting for conductor, attempt 15
-- Tue Jul 30 19:02:50 UTC 2019 waiting for conductor, attempt 16
-- Tue Jul 30 19:02:55 UTC 2019 waiting for conductor, attempt 17
-- Tue Jul 30 19:03:01 UTC 2019 waiting for conductor, attempt 18
-- Tue Jul 30 19:03:06 UTC 2019 waiting for conductor, attempt 19
-- Tue Jul 30 19:03:11 UTC 2019 waiting for conductor, attempt 20
-- Tue Jul 30 19:03:16 UTC 2019 conductor still down, exiting
-- conductor API is at http://localhost:8080
Starting Conductor server
Property file: config.properties
config.properties
Using 'config.properties'
-- Tue Jul 30 19:03:17 UTC 2019 waiting for conductor, attempt 1
Using config file/app/config/config.properties
[2019-07-30 19:03:17,774] [main] INFO com.netflix.conductor.bootstrap.ModulesProvider - Starting conductor server using dynomite/redis cluster.
[2019-07-30 19:03:17,780] [main] INFO com.netflix.conductor.bootstrap.ModulesProvider - External payload storage is not configured, provided: , supported values are: [S3]
java.lang.IllegalArgumentException: No enum constant com.netflix.conductor.bootstrap.ModulesProvider.ExternalPayloadStorageType.
at java.lang.Enum.valueOf(Enum.java:238)
at com.netflix.conductor.bootstrap.ModulesProvider$ExternalPayloadStorageType.valueOf(ModulesProvider.java:38)
at com.netflix.conductor.bootstrap.ModulesProvider.selectModulesToLoad(ModulesProvider.java:111)
at com.netflix.conductor.bootstrap.ModulesProvider.get(ModulesProvider.java:51)
at com.netflix.conductor.bootstrap.Main.main(Main.java:53)
[2019-07-30 19:03:18,136] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: HTTP
[2019-07-30 19:03:18,292] [main] INFO com.netflix.conductor.contribs.http.HttpTask - HttpTask initialized...
[2019-07-30 19:03:18,293] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: JSON_JQ_TRANSFORM
[2019-07-30 19:03:19,044] [main] INFO com.netflix.archaius.guice.ConfigurationInjectingListener - Initializing ConfigurationInjectingListener
[2019-07-30 19:03:19,190] [main] INFO com.netflix.dyno.jedis.DynoJedisClient - Starting connection pool for app conductor
[2019-07-30 19:03:19,191] [main] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Getting Hosts from TokenMapSupplier
[2019-07-30 19:03:19,196] [pool-3-thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Adding host connection pool for host: Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:03:19,197] [pool-3-thread-1] INFO com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl - Priming connection pool for host:Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null], with conns:10
[2019-07-30 19:03:19,334] [pool-3-thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Successfully primed 10 of 10 to Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:03:19,346] [main] INFO com.netflix.dyno.connectionpool.TokenPoolTopology - Adding Host to TopologyHost [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:03:19,351] [main] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - registered mbean com.netflix.dyno.connectionpool.impl:type=MonitorConsole
[2019-07-30 19:03:19,363] [main] INFO com.netflix.dyno.jedis.DynoJedisClient - Starting connection pool for app conductor
[2019-07-30 19:03:19,365] [main] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Getting Hosts from TokenMapSupplier
[2019-07-30 19:03:19,366] [pool-6-thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Adding host connection pool for host: Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:03:19,367] [pool-6-thread-1] INFO com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl - Priming connection pool for host:Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null], with conns:10
[2019-07-30 19:03:19,370] [pool-6-thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Successfully primed 10 of 10 to Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:03:19,371] [main] INFO com.netflix.dyno.connectionpool.TokenPoolTopology - Adding Host to TopologyHost [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:03:19,372] [main] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - mbean com.netflix.dyno.connectionpool.impl:type=MonitorConsole has already been registered !
[2019-07-30 19:03:19,382] [main] INFO com.netflix.conductor.dyno.RedisQueuesProvider - DynoQueueDAO initialized with prefix conductor_queues.test!
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
[2019-07-30 19:03:21,586] [main] INFO com.netflix.conductor.core.events.EventProcessor - Event Processing is ENABLED. executorThreadCount set to 2
[2019-07-30 19:03:21,593] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - System Task Worker initialized with 10 threads and a callback time of 30 seconds and queue size: 100 with pollCount: 10 and poll interval: 50
[2019-07-30 19:03:21,594] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: SUB_WORKFLOW
[2019-07-30 19:03:21,595] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: WAIT
[2019-07-30 19:03:21,596] [main] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding the queue for system task: EVENT
[2019-07-30 19:03:21,601] [main] INFO com.netflix.conductor.elasticsearch.EmbeddedElasticSearch - Deleting contents of data dir /tmp/elasticsearch_test_1564513401599data3627812782831074991
[2019-07-30 19:03:21,609] [main] INFO com.netflix.conductor.elasticsearch.es5.EmbeddedElasticSearchV5 - Starting ElasticSearch for cluster elasticsearch_test
[2019-07-30 19:03:21,610] [Thread-3] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Started listening for system task: HTTP
-- Tue Jul 30 19:03:22 UTC 2019 waiting for conductor, attempt 2
[2019-07-30 19:03:22,655] [pool-13-thread-1] INFO com.netflix.dyno.queues.redis.RedisDynoQueue - com.netflix.dyno.queues.redis.RedisDynoQueue is ready to serve HTTP
[2019-07-30 19:03:25,826] [main] INFO com.netflix.conductor.elasticsearch.es5.EmbeddedElasticSearchV5 - ElasticSearch cluster elasticsearch_test started in local mode on port 9200
-- Tue Jul 30 19:03:27 UTC 2019 waiting for conductor, attempt 3
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{Iq5wyGvySTyOTqR-V2Hdfw}{mender-elasticsearch}{192.168.64.3:9300}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:366)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
at com.netflix.conductor.dao.es5.index.ElasticSearchDAOV5.setup(ElasticSearchDAOV5.java:149)
at com.netflix.conductor.bootstrap.Main.main(Main.java:71)
[2019-07-30 19:03:30,835] [Thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Removing host mender-redis from selectionStrategy, cpHealthTracker, cpMonitor
[2019-07-30 19:03:30,836] [Thread-1] INFO com.netflix.dyno.connectionpool.TokenPoolTopology - Removing Host from TopologyHost [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:03:30,836] [Thread-1] INFO com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl - Shutting down connection pool for host:Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:03:30,841] [Thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Remove host: Successfully removed hostPool for host mender-redis from connection pool
[2019-07-30 19:03:30,844] [Thread-1] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - deregistered mbean com.netflix.dyno.connectionpool.impl:type=MonitorConsole
[2019-07-30 19:03:30,845] [Thread-2] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Removing host mender-redis from selectionStrategy, cpHealthTracker, cpMonitor
[2019-07-30 19:03:30,846] [Thread-2] INFO com.netflix.dyno.connectionpool.TokenPoolTopology - Removing Host from TopologyHost [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:03:30,846] [Thread-2] INFO com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl - Shutting down connection pool for host:Host [hostname=mender-redis, ipAddress=null, port=6379, rack: us-east-1b, datacenter: us-east-1, status: Up, hashtag=null]
[2019-07-30 19:03:30,856] [Thread-2] INFO com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl - Remove host: Successfully removed hostPool for host mender-redis from connection pool
-- Tue Jul 30 19:03:32 UTC 2019 waiting for conductor, attempt 4
-- Tue Jul 30 19:03:37 UTC 2019 waiting for conductor, attempt 5
-- Tue Jul 30 19:03:42 UTC 2019 waiting for conductor, attempt 6
-- Tue Jul 30 19:03:47 UTC 2019 waiting for conductor, attempt 7
-- Tue Jul 30 19:03:52 UTC 2019 waiting for conductor, attempt 8
-- Tue Jul 30 19:03:57 UTC 2019 waiting for conductor, attempt 9
-- Tue Jul 30 19:04:02 UTC 2019 waiting for conductor, attempt 10
-- Tue Jul 30 19:04:07 UTC 2019 waiting for conductor, attempt 11
-- Tue Jul 30 19:04:12 UTC 2019 waiting for conductor, attempt 12
-- Tue Jul 30 19:04:17 UTC 2019 waiting for conductor, attempt 13
-- Tue Jul 30 19:04:22 UTC 2019 waiting for conductor, attempt 14
-- Tue Jul 30 19:04:27 UTC 2019 waiting for conductor, attempt 15
-- Tue Jul 30 19:04:32 UTC 2019 waiting for conductor, attempt 16
-- Tue Jul 30 19:04:37 UTC 2019 waiting for conductor, attempt 17
-- Tue Jul 30 19:04:42 UTC 2019 waiting for conductor, attempt 18
-- Tue Jul 30 19:04:47 UTC 2019 waiting for conductor, attempt 19
-- Tue Jul 30 19:04:52 UTC 2019 waiting for conductor, attempt 20
-- Tue Jul 30 19:04:57 UTC 2019 conductor still down, exiting
ckelloug@cameron-laptop:~/mender_hub$ Here is elasticsearch.yml:
network.host: 0.0.0.0
discovery.type: single-node