K8s deployment - getting "CrashLoopBackOff" for business logic pods #130

pulnara · 2021-04-26T14:53:13Z

I'm currently trying to run this application in Kubernetes, but with no luck. I fixed a couple of problems with the deployments in my forked version: https://github.com/pulnara/ftgo-application

I'm using a Minikube Kubernetes cluster and the kubernetes-deploy-all.sh script. After executing it, pods responsible for Kafka, MySQL etc. are successfully running, but the ones for business logic (like accounting service, consumer service etc.) not. After a while (and multiple restarts) they're eventually going into the CrashLoopBackOff status, since all the time they don't meet the liveness probe - getting "connection refused" for the Spring Boot's /actuator/health endpoint.

This is my result of the kubectl get pods command:

NAME                                         READY   STATUS    RESTARTS   AGE
ftgo-accounting-service-55bd94d788-sh62d     0/1     Running   4          11m
ftgo-api-gateway-79948fdfbc-mxndr            0/1     Running   5          11m
ftgo-cdc-service-58fcf454b5-fdf67            0/1     Running   5          11m
ftgo-consumer-service-59fd7cd94d-wstrr       0/1     Running   4          11m
ftgo-dynamodb-local-0                        1/1     Running   0          13m
ftgo-dynamodb-local-init-4kpf5               1/1     Running   0          11m
ftgo-dynamodb-local-init-mntsj               0/1     Error     0          13m
ftgo-kafka-0                                 1/1     Running   0          13m
ftgo-kitchen-service-868476cb5c-rpxfc        0/1     Running   4          11m
ftgo-mysql-0                                 1/1     Running   0          13m
ftgo-order-history-service-b84cb6f49-8qtv6   0/1     Running   5          11m
ftgo-order-service-58b776fc7c-q4l22          0/1     Running   4          11m
ftgo-restaurant-service-5fdf567678-ffdvc     0/1     Running   4          11m
ftgo-zookeeper-0                             1/1     Running   0          13m

This are events from the ftgo-accounting-service pod's description:
Events:

  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  5m52s                  default-scheduler  Successfully assigned default/ftgo-accounting-service-55bd94d788-sh62d to minikube
  Normal   Pulled     5m25s                  kubelet            Successfully pulled image "msapatterns/ftgo-accounting-service:latest" in 8.6026105s
  Normal   Pulled     2m55s                  kubelet            Successfully pulled image "msapatterns/ftgo-accounting-service:latest" in 2.0851995s
  Normal   Created    2m54s (x2 over 5m25s)  kubelet            Created container ftgo-accounting-service
  Normal   Started    2m53s (x2 over 5m24s)  kubelet            Started container ftgo-accounting-service
  Warning  Unhealthy  68s (x6 over 4m7s)     kubelet            Liveness probe failed: Get "http://172.17.0.9:8080/actuator/health": dial tcp 172.17.0.9:8080: connect: connection refused
  Normal   Killing    67s (x2 over 3m27s)    kubelet            Container ftgo-accounting-service failed liveness probe, will be restarted
  Warning  Unhealthy  49s (x8 over 4m9s)     kubelet            Readiness probe failed: Get "http://172.17.0.9:8080/actuator/health": dial tcp 172.17.0.9:8080: connect: connection refused
  Normal   Pulling    35s (x3 over 5m33s)    kubelet            Pulling image "msapatterns/ftgo-accounting-service:latest"
  Normal   Pulled     31s                    kubelet            Successfully pulled image "msapatterns/ftgo-accounting-service:latest" in 3.8507742s

Did anyone happen to have a similar problem? I tried multiple times and on 2 different OS (Ubuntu, Mac OS), and getting the same error on both. I have no idea how to fix it.

The text was updated successfully, but these errors were encountered:

cer · 2021-04-26T15:00:09Z

I haven't ran this in a while.

Are there any logs for the ftgo-accounting-service pod?

pulnara · 2021-04-26T15:50:56Z

No, kubectl logs command (ran the script once again just now, which is why the pod IDs are different) returns nothing:

$ kubectl get pods
NAME                                         READY   STATUS    RESTARTS   AGE
ftgo-accounting-service-55bd94d788-cg6jp     0/1     Running   2          6m11s
ftgo-api-gateway-79948fdfbc-g5fgt            0/1     Running   2          6m10s
ftgo-cdc-service-58fcf454b5-9qvw6            0/1     Running   2          6m11s
ftgo-consumer-service-59fd7cd94d-bpbng       0/1     Running   2          6m9s
ftgo-dynamodb-local-0                        1/1     Running   0          7m36s
ftgo-dynamodb-local-init-chzrw               0/1     Error     0          7m36s
ftgo-dynamodb-local-init-nchmp               1/1     Running   0          4m
ftgo-kafka-0                                 1/1     Running   0          7m35s
ftgo-kitchen-service-868476cb5c-h7kg5        0/1     Running   2          6m8s
ftgo-mysql-0                                 1/1     Running   0          7m34s
ftgo-order-history-service-b84cb6f49-f4877   0/1     Running   2          6m7s
ftgo-order-service-58b776fc7c-9t6zl          0/1     Running   2          6m5s
ftgo-restaurant-service-5fdf567678-t5sk2     0/1     Running   1          6m4s
ftgo-zookeeper-0                             1/1     Running   0          7m33s

cer · 2021-04-26T16:03:02Z

Hmmm... what about kubectl logs -p?

pulnara · 2021-04-26T16:14:39Z

It returns the following:

$ kubectl logs -p ftgo-accounting-service-55bd94d788-cg6jp 
2021-04-26 16:02:49.885  INFO [,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.2.1.RELEASE)

2021-04-26 16:02:56.472  INFO [ftgo-accounting-service,,,] 8 --- [           main] n.c.f.a.main.AccountingServiceMain       : No active profile set, falling back to default profiles: default
2021-04-26 16:03:55.502  INFO [ftgo-accounting-service,,,] 8 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data repositories in DEFAULT mode.
2021-04-26 16:03:56.347  INFO [ftgo-accounting-service,,,] 8 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 552ms. Found 0 repository interfaces.
2021-04-26 16:04:07.025  INFO [ftgo-accounting-service,,,] 8 --- [           main] o.s.cloud.context.scope.GenericScope     : BeanFactory id=bb8bd870-2d57-3dd4-a7e9-669745dbf85a
2021-04-26 16:04:16.353  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:16.938  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'io.eventuate.javaclient.spring.EventuateJavaClientDomainConfiguration' of type [io.eventuate.javaclient.spring.EventuateJavaClientDomainConfiguration$$EnhancerBySpringCGLIB$$f16b79d7] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:17.258  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'io.eventuate.javaclient.spring.common.EventuateCommonConfiguration' of type [io.eventuate.javaclient.spring.common.EventuateCommonConfiguration$$EnhancerBySpringCGLIB$$1b424602] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:17.370  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'io.eventuate.local.java.spring.common.jdbckafkastore.EventuateLocalConfiguration' of type [io.eventuate.local.java.spring.common.jdbckafkastore.EventuateLocalConfiguration$$EnhancerBySpringCGLIB$$a5e7fd80] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:17.988  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'dataSource' of type [org.springframework.aop.scope.ScopedProxyFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:17.996  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'dataSource' of type [com.zaxxer.hikari.HikariDataSource$$EnhancerBySpringCGLIB$$15533138] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:18.229  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'spring.datasource-org.springframework.boot.autoconfigure.jdbc.DataSourceProperties' of type [org.springframework.boot.autoconfigure.jdbc.DataSourceProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:18.374  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jdbc.DataSourceInitializerInvoker' of type [org.springframework.boot.autoconfigure.jdbc.DataSourceInitializerInvoker] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:18.486  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'spring.jpa-org.springframework.boot.autoconfigure.orm.jpa.JpaProperties' of type [org.springframework.boot.autoconfigure.orm.jpa.JpaProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:18.535  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'spring.jpa.hibernate-org.springframework.boot.autoconfigure.orm.jpa.HibernateProperties' of type [org.springframework.boot.autoconfigure.orm.jpa.HibernateProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:18.611  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration' of type [org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:18.676  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'hikariPoolDataSourceMetadataProvider' of type [org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration$$Lambda$255/1732238286] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:19.019  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration' of type [org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:19.218  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.transaction.TransactionAutoConfiguration' of type [org.springframework.boot.autoconfigure.transaction.TransactionAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:19.251  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'spring.transaction-org.springframework.boot.autoconfigure.transaction.TransactionProperties' of type [org.springframework.boot.autoconfigure.transaction.TransactionProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:19.262  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'platformTransactionManagerCustomizers' of type [org.springframework.boot.autoconfigure.transaction.TransactionManagerCustomizers] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:20.637  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'jpaVendorAdapter' of type [org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:20.711  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'entityManagerFactoryBuilder' of type [org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-26 16:04:21.299  INFO [ftgo-accounting-service,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jdbc.DataSourceConfiguration$Hikari' of type [org.springframework.boot.autoconfigure.jdbc.DataSourceConfiguration$Hikari] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

It seems that introducing lazy initialization might help. I'm not very familiar with Spring beans yet, so I'm not sure where to fix it though.

cer · 2021-04-26T16:24:14Z

It seems that introducing lazy initialization might help. I'm not very familiar with Spring beans yet, so I'm not sure where to fix it though.

It's probably best to assume that the service code works at the application level. The probably is mostly likely with where it's running (i.e. K8s) and how it has been configured to run (e.g. k8s yaml)..

That log is useful. In particular it doesn't show any errors, which is interesting.

A couple of thoughts:

What's the configured memory of the Docker VM? The default settings are way too low to run Java services.
Another potential problem is that starting all of the services (including CDC) in one go is very resource intensive and can potentially overload a machine. I have a vague memory of situation where due to a lack of resources the services could not start in time and were repeatedly killed. I would recommend starting services one at a time to see whether that solves the problem.

pulnara · 2021-04-27T14:46:48Z

I've just tried running this on another machine - MacBook Pro with 16GB RAM (previously used MacBook Air/8 GB and Ubuntu machine with 16 GB as well). Here are my Docker resources settings:

After applying this change, I tried starting only the ftgo-application-service (after successfully deploying all of the stateful-services) and it didn't work either, resulting with similar logs.

What's your recommended memory/resources configuration?

pulnara · 2021-04-27T21:54:44Z

After a couple more attempts, and starting minikube with the following specification:
minikube start --cpus 4 --memory 4096

I can see that the problem might lay in the service being unable to reach Kafka topic.
Here is some part of the accounting service's pod logs that indicates it:

2021-04-27 21:44:07.319  INFO [ftgo-accounting-service,,,] 7 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 2.3.0
2021-04-27 21:44:07.319  INFO [ftgo-accounting-service,,,] 7 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: fc1aaa116b661c8a
2021-04-27 21:44:07.319  INFO [ftgo-accounting-service,,,] 7 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1619559847318
2021-04-27 21:44:07.324 DEBUG [ftgo-accounting-service,,,] 7 --- [           main] i.e.m.k.b.c.EventuateKafkaConsumer       : Verifying Topic net.chrisrichardson.ftgo.consumerservice.domain.Consumer
2021-04-27 21:45:07.339 ERROR [ftgo-accounting-service,,,] 7 --- [           main] i.e.m.k.b.c.EventuateKafkaConsumer       : Got exception: 

org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

2021-04-27 21:45:07.349 ERROR [ftgo-accounting-service,,,] 7 --- [           main] i.e.m.k.b.c.EventuateKafkaConsumer       : Error subscribing

java.lang.RuntimeException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
        at io.eventuate.messaging.kafka.basic.consumer.EventuateKafkaConsumer.verifyTopicExistsBeforeSubscribing(EventuateKafkaConsumer.java:73) ~[eventuate-messaging-kafka-basic-consumer-0.10.0.RELEASE.jar!/:na]
        at io.eventuate.messaging.kafka.basic.consumer.EventuateKafkaConsumer.start(EventuateKafkaConsumer.java:98) ~[eventuate-messaging-kafka-basic-consumer-0.10.0.RELEASE.jar!/:na]
        at io.eventuate.messaging.kafka.consumer.MessageConsumerKafkaImpl.subscribe(MessageConsumerKafkaImpl.java:55) [eventuate-messaging-kafka-consumer-0.10.0.RELEASE.jar!/:na]
        at io.eventuate.tram.consumer.kafka.EventuateTramKafkaMessageConsumer.subscribe(EventuateTramKafkaMessageConsumer.java:28) [eventuate-tram-consumer-kafka-0.25.1.RELEASE.jar!/:na]
        at io.eventuate.tram.consumer.common.MessageConsumerImpl.subscribe(MessageConsumerImpl.java:37) [eventuate-tram-consumer-common-0.25.1.RELEASE.jar!/:na]
        at io.eventuate.tram.events.subscriber.DomainEventDispatcher.initialize(DomainEventDispatcher.java:35) [eventuate-tram-events-0.25.1.RELEASE.jar!/:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_252]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_252]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_252]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_252]
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389) [spring-beans-5.2.1.RELEASE.jar!/:5.2.1.RELEASE]
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333) [spring-beans-5.2.1.RELEASE.jar!/:5.2.1.RELEASE]
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157) [spring-beans-5.2.1.RELEASE.jar!/:5.2.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:416) [spring-beans-5.2.1.RELEASE.jar!/:5.2.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1795) [spring-beans-5.2.1.RELEASE.jar!/:5.2.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595) [spring-beans-5.2.1.RELEASE.jar!/:5.2.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) [spring-beans-5.2.1.RELEASE.jar!/:5.2.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) [spring-beans-5.2.1.RELEASE.jar!/:5.2.1.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.2.1.RELEASE.jar!/:5.2.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) [spring-beans-5.2.1.RELEASE.jar!/:5.2.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) [spring-beans-5.2.1.RELEASE.jar!/:5.2.1.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:879) ~[spring-beans-5.2.1.RELEASE.jar!/:5.2.1.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878) ~[spring-context-5.2.1.RELEASE.jar!/:5.2.1.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) ~[spring-context-5.2.1.RELEASE.jar!/:5.2.1.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141) ~[spring-boot-2.2.1.RELEASE.jar!/:2.2.1.RELEASE]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747) ~[spring-boot-2.2.1.RELEASE.jar!/:2.2.1.RELEASE]
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) ~[spring-boot-2.2.1.RELEASE.jar!/:2.2.1.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-2.2.1.RELEASE.jar!/:2.2.1.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) ~[spring-boot-2.2.1.RELEASE.jar!/:2.2.1.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) ~[spring-boot-2.2.1.RELEASE.jar!/:2.2.1.RELEASE]
        at net.chrisrichardson.ftgo.accountingservice.main.AccountingServiceMain.main(AccountingServiceMain.java:24) ~[classes!/:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_252]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_252]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_252]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_252]
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) ~[service.jar:na]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) ~[service.jar:na]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) ~[service.jar:na]
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) ~[service.jar:na]
Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

Please find the full log attached here: accounting-service-logs.txt

I believe that the kafka service doesn't work properly. I'm attaching its logs as well:
kafka-pod-logs.txt

cer · 2021-04-27T22:19:48Z

Finally an error message!

The Kafka log indicates that the DNS name ftgo-zookeeper cannot be resolved. Is there a K8s ftgo-zookeeper service?

cer · 2021-04-27T22:20:42Z

PS. I am surprised that you need to run minikube start --cpus 4 --memory 4096 - why not the k8s built into Docker?

pulnara · 2021-04-27T22:30:54Z

Finally an error message!

The Kafka log indicates that the DNS name ftgo-zookeeper cannot be resolved. Is there a K8s ftgo-zookeeper service?

It's up and running. Plese find the full pod's log here:
zookeeper-pod-logs.txt

PS. I am surprised that you need to run minikube start --cpus 4 --memory 4096 - why not the k8s built into Docker?

I'm just used to using minikube :) I definitely should try the built-in K8s then.

Thank you very much for your assistance, I'm still learning :)

cer · 2021-04-27T23:17:44Z

I'm just used to using minikube :) I definitely should try the built-in K8s then.

Presumably, that's why increasing the Docker VM size did not have an effect.

It's up and running. Plese find the full pod's log here: zookeeper-pod-logs.txt

This indicates that there is a pod. It tell you whether there is a Service defined -

ftgo-application/deployment/kubernetes/stateful-services/ftgo-zookeeper-deployment.yml

Line 1 in a835e23

apiVersion: v1

- that configures the DNS name

cer · 2021-04-27T23:22:50Z

BTW I am bit confused about that's happening. I read more of the Kafka log. It seems to evaluating connect to Kafka. But then it (sometimes) has problems connecting to itself via its advertised name ftgo-kafka:9092 when trying to create the topic net.chrisrichardson.ftgo.consumerservice.domain.Consumer (the problem in the accounting service)

[2021-04-27 21:52:10,203] WARN [RequestSendThread controllerId=1001] Controller 1001's connection to broker ftgo-kafka:9092 (id: 1001 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.net.SocketTimeoutException: Failed to connect within 30000 ms
	at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:283)
	at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:236)
	at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)

vanillaxx · 2021-04-28T16:56:12Z

I've check and it seems that topic is created but I also cannot find the reason of the SocketTimeoutException

pulnara · 2021-04-29T12:50:08Z

Okay - I think I know what is the reason of this problem.

After you've drawn my attention to the DNS name issue, I found some discussion on StackOverflow which indicated that Minikube doesn't usually work well with names, preferring IPs. I decided to use the K8s built-in into Docker, and it worked! Even on low resources settings.

Example screenshot after I deployed 2 of the services:

I will switch to Docker built-in K8s or GKS then.
Since the app works like a charm after I've updated the outdated API in K8s objects, I took the liberty of creating a PR to integrate those changes: #131

@cer thank you very much for your patience and help in debugging! 😄

pulnara closed this as completed May 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

K8s deployment - getting "CrashLoopBackOff" for business logic pods #130

K8s deployment - getting "CrashLoopBackOff" for business logic pods #130

pulnara commented Apr 26, 2021

cer commented Apr 26, 2021

pulnara commented Apr 26, 2021 •

edited

Loading

cer commented Apr 26, 2021

pulnara commented Apr 26, 2021

cer commented Apr 26, 2021

pulnara commented Apr 27, 2021

pulnara commented Apr 27, 2021

cer commented Apr 27, 2021

cer commented Apr 27, 2021

pulnara commented Apr 27, 2021

cer commented Apr 27, 2021

cer commented Apr 27, 2021 •

edited

Loading

vanillaxx commented Apr 28, 2021

pulnara commented Apr 29, 2021 •

edited

Loading

K8s deployment - getting "CrashLoopBackOff" for business logic pods #130

K8s deployment - getting "CrashLoopBackOff" for business logic pods #130

Comments

pulnara commented Apr 26, 2021

cer commented Apr 26, 2021

pulnara commented Apr 26, 2021 • edited Loading

cer commented Apr 26, 2021

pulnara commented Apr 26, 2021

cer commented Apr 26, 2021

pulnara commented Apr 27, 2021

pulnara commented Apr 27, 2021

cer commented Apr 27, 2021

cer commented Apr 27, 2021

pulnara commented Apr 27, 2021

cer commented Apr 27, 2021

cer commented Apr 27, 2021 • edited Loading

vanillaxx commented Apr 28, 2021

pulnara commented Apr 29, 2021 • edited Loading

pulnara commented Apr 26, 2021 •

edited

Loading

cer commented Apr 27, 2021 •

edited

Loading

pulnara commented Apr 29, 2021 •

edited

Loading