TDH社区版安装，通过TOS启动prometheus报错
2023-07-22 10:40:10 2023-07-24 14:54:25 128
TDH社区版基础安装，安装过程中发生：
log:
2023-07-22T10:13:39.234944 [Master] ========== Task 145 start to run. ==========
2023-07-22T10:13:39.235262 [Master] Starting task local part ...
2023-07-22T10:13:39.235487 [Master] Start handle kubectl task
2023-07-22T10:13:39.238797 [Master] Target yaml path on manager is /var/lib/transwarp-manager/master/content/resources/services/aquila/prometheus.yaml
2023-07-22T10:13:39.238822 [Master] Start to generate prometheus.yaml on manager...
2023-07-22T10:13:39.297600 [Master] generated prometheus.yaml on [Manager]
2023-07-22T10:13:39.439819 [Master] Start [Create Roles] ...
2023-07-22T10:13:39.439852 [Master] Start executing [kubectl --server=https://127.0.0.1:6443 --certificate-authority=/srv/kubernetes/ca.pem --client-certificate=/srv/kubernetes/admin.pem --client-key=/srv/kubernetes/admin-key.pem create -f /var/lib/transwarp-manager/master/content/resources/services/aquila/prometheus.yaml]
2023-07-22T10:13:39.803493 [Master] [Create Roles] success.
2023-07-22T10:13:39.803544 [Master] Task local part ended.
2023-07-22T10:13:39.803559 [Master] Starting task remote part ...
2023-07-22T10:13:39.805377 [Master] Waiting 1 AQUILA_PROMETHEUS role(s) in Aquila to become Healthy within 600 s
2023-07-22T10:16:39.946380 [Master] Latest health check result of roles:
DAEMON_CHECK DOWN at 2023-07-22T10:16:38.829
AQUILA_PROMETHEUS has no Pod on tdh-masterVITAL_SIGN_CHECK DOWN at 2023-07-22T10:16:38.858
AQUILA_PROMETHEUS has no ready pod on tdh-master
2023-07-22T10:16:39.946789 [Master] Fail to run task remote part: io.transwarp.manager.master.operation.execution.entity.TaskDownException: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.KubectlTaskLocalRunner.postRemote(KubectlTaskLocalRunner.java:898)
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper.runTask(TaskDriver.java:287)
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper$$FastClassBySpringCGLIB$$e353057f.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.AbstractTaskLocalRunner.waitRolesHealthy(AbstractTaskLocalRunner.java:721)
at io.transwarp.manager.master.operation.execution.localrunner.KubectlTaskLocalRunner.postRemote(KubectlTaskLocalRunner.java:887)
... 11 more
Caused by: org.awaitility.core.ConditionTimeoutException: still DOWN within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.AbstractTaskLocalRunner.lambda$waitRolesHealthy$7(AbstractTaskLocalRunner.java:700)
at org.awaitility.core.CallableCondition$ConditionEvaluationWrapper.eval(CallableCondition.java:99)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:248)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:235)
... 4 more
2023-07-22T10:16:39.955739 [Master] The Task 145 run failed: java.lang.RuntimeException: io.transwarp.manager.master.operation.execution.entity.TaskDownException: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper.runTask(TaskDriver.java:292)
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper$$FastClassBySpringCGLIB$$e353057f.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.transwarp.manager.master.operation.execution.entity.TaskDownException: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.KubectlTaskLocalRunner.postRemote(KubectlTaskLocalRunner.java:898)
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper.runTask(TaskDriver.java:287)
... 10 more
Caused by: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.AbstractTaskLocalRunner.waitRolesHealthy(AbstractTaskLocalRunner.java:721)
at io.transwarp.manager.master.operation.execution.localrunner.KubectlTaskLocalRunner.postRemote(KubectlTaskLocalRunner.java:887)
... 11 more
Caused by: org.awaitility.core.ConditionTimeoutException: still DOWN within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.AbstractTaskLocalRunner.lambda$waitRolesHealthy$7(AbstractTaskLocalRunner.java:700)
at org.awaitility.core.CallableCondition$ConditionEvaluationWrapper.eval(CallableCondition.java:99)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:248)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:235)
... 4 more
2023-07-22T10:19:01.677310 [Master] ========== Task 145 start to run. ==========
2023-07-22T10:19:01.677811 [Master] Starting task local part ...
2023-07-22T10:19:01.678169 [Master] Start handle kubectl task
2023-07-22T10:19:01.680300 [Master] Target yaml path on manager is /var/lib/transwarp-manager/master/content/resources/services/aquila/prometheus.yaml
2023-07-22T10:19:01.680320 [Master] Start to generate prometheus.yaml on manager...
2023-07-22T10:19:01.794996 [Master] generated prometheus.yaml on [Manager]
2023-07-22T10:19:04.242798 [Master] Start [Create Roles] ...
2023-07-22T10:19:04.242859 [Master] Start executing [kubectl --server=https://127.0.0.1:6443 --certificate-authority=/srv/kubernetes/ca.pem --client-certificate=/srv/kubernetes/admin.pem --client-key=/srv/kubernetes/admin-key.pem scale --replicas=1 -f /var/lib/transwarp-manager/master/content/resources/services/aquila/prometheus.yaml]
2023-07-22T10:19:04.556451 [Master] [Create Roles] success.
2023-07-22T10:19:04.556505 [Master] Task local part ended.
2023-07-22T10:19:04.556522 [Master] Starting task remote part ...
2023-07-22T10:19:04.561581 [Master] Waiting 1 AQUILA_PROMETHEUS role(s) in Aquila to become Healthy within 600 s
2023-07-22T10:22:04.611202 [Master] Latest health check result of roles:
DAEMON_CHECK DOWN at 2023-07-22T10:22:03.676
AQUILA_PROMETHEUS has no Pod on tdh-masterVITAL_SIGN_CHECK DOWN at 2023-07-22T10:21:58.882
AQUILA_PROMETHEUS has no ready pod on tdh-master
2023-07-22T10:22:04.611312 [Master] Fail to run task remote part: io.transwarp.manager.master.operation.execution.entity.TaskDownException: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.KubectlTaskLocalRunner.postRemote(KubectlTaskLocalRunner.java:898)
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper.runTask(TaskDriver.java:287)
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper$$FastClassBySpringCGLIB$$e353057f.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.AbstractTaskLocalRunner.waitRolesHealthy(AbstractTaskLocalRunner.java:721)
at io.transwarp.manager.master.operation.execution.localrunner.KubectlTaskLocalRunner.postRemote(KubectlTaskLocalRunner.java:887)
... 11 more
Caused by: org.awaitility.core.ConditionTimeoutException: still DOWN within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.AbstractTaskLocalRunner.lambda$waitRolesHealthy$7(AbstractTaskLocalRunner.java:700)
at org.awaitility.core.CallableCondition$ConditionEvaluationWrapper.eval(CallableCondition.java:99)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:248)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:235)
... 4 more
2023-07-22T10:22:04.613657 [Master] The Task 145 run failed: java.lang.RuntimeException: io.transwarp.manager.master.operation.execution.entity.TaskDownException: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper.runTask(TaskDriver.java:292)
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper$$FastClassBySpringCGLIB$$e353057f.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.transwarp.manager.master.operation.execution.entity.TaskDownException: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.KubectlTaskLocalRunner.postRemote(KubectlTaskLocalRunner.java:898)
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper.runTask(TaskDriver.java:287)
... 10 more
Caused by: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.AbstractTaskLocalRunner.waitRolesHealthy(AbstractTaskLocalRunner.java:721)
at io.transwarp.manager.master.operation.execution.localrunner.KubectlTaskLocalRunner.postRemote(KubectlTaskLocalRunner.java:887)
... 11 more
Caused by: org.awaitility.core.ConditionTimeoutException: still DOWN within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.AbstractTaskLocalRunner.lambda$waitRolesHealthy$7(AbstractTaskLocalRunner.java:700)
at org.awaitility.core.CallableCondition$ConditionEvaluationWrapper.eval(CallableCondition.java:99)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:248)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:235)
... 4 more
log:
2023-07-22T10:13:39.234944 [Master] ========== Task 145 start to run. ==========
2023-07-22T10:13:39.235262 [Master] Starting task local part ...
2023-07-22T10:13:39.235487 [Master] Start handle kubectl task
2023-07-22T10:13:39.238797 [Master] Target yaml path on manager is /var/lib/transwarp-manager/master/content/resources/services/aquila/prometheus.yaml
2023-07-22T10:13:39.238822 [Master] Start to generate prometheus.yaml on manager...
2023-07-22T10:13:39.297600 [Master] generated prometheus.yaml on [Manager]
2023-07-22T10:13:39.439819 [Master] Start [Create Roles] ...
2023-07-22T10:13:39.439852 [Master] Start executing [kubectl --server=https://127.0.0.1:6443 --certificate-authority=/srv/kubernetes/ca.pem --client-certificate=/srv/kubernetes/admin.pem --client-key=/srv/kubernetes/admin-key.pem create -f /var/lib/transwarp-manager/master/content/resources/services/aquila/prometheus.yaml]
2023-07-22T10:13:39.803493 [Master] [Create Roles] success.
2023-07-22T10:13:39.803544 [Master] Task local part ended.
2023-07-22T10:13:39.803559 [Master] Starting task remote part ...
2023-07-22T10:13:39.805377 [Master] Waiting 1 AQUILA_PROMETHEUS role(s) in Aquila to become Healthy within 600 s
2023-07-22T10:16:39.946380 [Master] Latest health check result of roles:
DAEMON_CHECK DOWN at 2023-07-22T10:16:38.829
AQUILA_PROMETHEUS has no Pod on tdh-masterVITAL_SIGN_CHECK DOWN at 2023-07-22T10:16:38.858
AQUILA_PROMETHEUS has no ready pod on tdh-master
2023-07-22T10:16:39.946789 [Master] Fail to run task remote part: io.transwarp.manager.master.operation.execution.entity.TaskDownException: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.KubectlTaskLocalRunner.postRemote(KubectlTaskLocalRunner.java:898)
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper.runTask(TaskDriver.java:287)
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper$$FastClassBySpringCGLIB$$e353057f.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.AbstractTaskLocalRunner.waitRolesHealthy(AbstractTaskLocalRunner.java:721)
at io.transwarp.manager.master.operation.execution.localrunner.KubectlTaskLocalRunner.postRemote(KubectlTaskLocalRunner.java:887)
... 11 more
Caused by: org.awaitility.core.ConditionTimeoutException: still DOWN within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.AbstractTaskLocalRunner.lambda$waitRolesHealthy$7(AbstractTaskLocalRunner.java:700)
at org.awaitility.core.CallableCondition$ConditionEvaluationWrapper.eval(CallableCondition.java:99)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:248)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:235)
... 4 more
2023-07-22T10:16:39.955739 [Master] The Task 145 run failed: java.lang.RuntimeException: io.transwarp.manager.master.operation.execution.entity.TaskDownException: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper.runTask(TaskDriver.java:292)
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper$$FastClassBySpringCGLIB$$e353057f.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.transwarp.manager.master.operation.execution.entity.TaskDownException: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.KubectlTaskLocalRunner.postRemote(KubectlTaskLocalRunner.java:898)
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper.runTask(TaskDriver.java:287)
... 10 more
Caused by: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.AbstractTaskLocalRunner.waitRolesHealthy(AbstractTaskLocalRunner.java:721)
at io.transwarp.manager.master.operation.execution.localrunner.KubectlTaskLocalRunner.postRemote(KubectlTaskLocalRunner.java:887)
... 11 more
Caused by: org.awaitility.core.ConditionTimeoutException: still DOWN within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.AbstractTaskLocalRunner.lambda$waitRolesHealthy$7(AbstractTaskLocalRunner.java:700)
at org.awaitility.core.CallableCondition$ConditionEvaluationWrapper.eval(CallableCondition.java:99)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:248)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:235)
... 4 more
2023-07-22T10:19:01.677310 [Master] ========== Task 145 start to run. ==========
2023-07-22T10:19:01.677811 [Master] Starting task local part ...
2023-07-22T10:19:01.678169 [Master] Start handle kubectl task
2023-07-22T10:19:01.680300 [Master] Target yaml path on manager is /var/lib/transwarp-manager/master/content/resources/services/aquila/prometheus.yaml
2023-07-22T10:19:01.680320 [Master] Start to generate prometheus.yaml on manager...
2023-07-22T10:19:01.794996 [Master] generated prometheus.yaml on [Manager]
2023-07-22T10:19:04.242798 [Master] Start [Create Roles] ...
2023-07-22T10:19:04.242859 [Master] Start executing [kubectl --server=https://127.0.0.1:6443 --certificate-authority=/srv/kubernetes/ca.pem --client-certificate=/srv/kubernetes/admin.pem --client-key=/srv/kubernetes/admin-key.pem scale --replicas=1 -f /var/lib/transwarp-manager/master/content/resources/services/aquila/prometheus.yaml]
2023-07-22T10:19:04.556451 [Master] [Create Roles] success.
2023-07-22T10:19:04.556505 [Master] Task local part ended.
2023-07-22T10:19:04.556522 [Master] Starting task remote part ...
2023-07-22T10:19:04.561581 [Master] Waiting 1 AQUILA_PROMETHEUS role(s) in Aquila to become Healthy within 600 s
2023-07-22T10:22:04.611202 [Master] Latest health check result of roles:
DAEMON_CHECK DOWN at 2023-07-22T10:22:03.676
AQUILA_PROMETHEUS has no Pod on tdh-masterVITAL_SIGN_CHECK DOWN at 2023-07-22T10:21:58.882
AQUILA_PROMETHEUS has no ready pod on tdh-master
2023-07-22T10:22:04.611312 [Master] Fail to run task remote part: io.transwarp.manager.master.operation.execution.entity.TaskDownException: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.KubectlTaskLocalRunner.postRemote(KubectlTaskLocalRunner.java:898)
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper.runTask(TaskDriver.java:287)
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper$$FastClassBySpringCGLIB$$e353057f.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.AbstractTaskLocalRunner.waitRolesHealthy(AbstractTaskLocalRunner.java:721)
at io.transwarp.manager.master.operation.execution.localrunner.KubectlTaskLocalRunner.postRemote(KubectlTaskLocalRunner.java:887)
... 11 more
Caused by: org.awaitility.core.ConditionTimeoutException: still DOWN within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.AbstractTaskLocalRunner.lambda$waitRolesHealthy$7(AbstractTaskLocalRunner.java:700)
at org.awaitility.core.CallableCondition$ConditionEvaluationWrapper.eval(CallableCondition.java:99)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:248)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:235)
... 4 more
2023-07-22T10:22:04.613657 [Master] The Task 145 run failed: java.lang.RuntimeException: io.transwarp.manager.master.operation.execution.entity.TaskDownException: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper.runTask(TaskDriver.java:292)
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper$$FastClassBySpringCGLIB$$e353057f.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.transwarp.manager.master.operation.execution.entity.TaskDownException: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.KubectlTaskLocalRunner.postRemote(KubectlTaskLocalRunner.java:898)
at io.transwarp.manager.master.operation.execution.TaskDriver$TaskRunHelper.runTask(TaskDriver.java:287)
... 10 more
Caused by: java.lang.IllegalStateException: 1 AQUILA_PROMETHEUS role(s) in Aquila didn't become healthy within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.AbstractTaskLocalRunner.waitRolesHealthy(AbstractTaskLocalRunner.java:721)
at io.transwarp.manager.master.operation.execution.localrunner.KubectlTaskLocalRunner.postRemote(KubectlTaskLocalRunner.java:887)
... 11 more
Caused by: org.awaitility.core.ConditionTimeoutException: still DOWN within 180 s
at io.transwarp.manager.master.operation.execution.localrunner.AbstractTaskLocalRunner.lambda$waitRolesHealthy$7(AbstractTaskLocalRunner.java:700)
at org.awaitility.core.CallableCondition$ConditionEvaluationWrapper.eval(CallableCondition.java:99)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:248)
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:235)
... 4 more
回答
登录后可回答问题
.jpg)
