Exception in Job-Manager execution

[raghupathiguduri]

Hi Team,
We are currently running the setup with version release-5.0.0 and we want to enable different consumption repoorts like wfs, content rating, course consumption.

We have added below two jobs in the crontab for daily schedule.

/bin/bash -lc "ruby /mount/data/analytics/scripts/submit-all-jobs.rb"
/mount/data/analytics/scripts/start-jobmanager.sh

But we are seeing below error in the log file ( /mount/data/analytics/logs/services/{{date}}-job-manager.log)

Caused by: java.lang.IllegalArgumentException: endpoint.getHost() is null for https:///telemetry-data-store?restype=container&comp=list&prefix=transient-data&include=metadata

In the above url it is missing storage account name before the container name.

We did check the config files and env variables to find where this storage details are defined. We don't see any missing value from config.

Can you help us to understand where this config is missing ?

[sowmya-dixit]

@raghupathiguduri
Can you please share the complete job manager logs which will help us to understand the issue better.

[raghupathiguduri]

@sowmya-dixit Please find the complete log file of job-manager

Killing currently running job-manager process
Starting the job manager
config: {"jobsCount":3,"topic":"production.analytics.job_queue","bootStrapServer":"10.1.1.6:9092,10.1.1.7:9092,10.1.1.8:9092","zookeeperConnect":"10.1.1.6:2181,10.1.1.7:2181,10.1.1.8:2181","consumerGroup":"jobmanager","slackChannel":"#test_channel","slackUserName":"JobManager","tempBucket":"telemetry-data-store","tempFolder":"transient-data"}
Job manager is started.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$2 (file:/mount/data/analytics/spark-3.1.3-bin-hadoop2.7/jars/guice-3.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
org.sunbird.cloud.storage.exception.StorageServiceException: endpoint.getHost() is null for https:///telemetry-data-store?restype=container&comp=list&prefix=transient-data&include=metadata
at org.sunbird.cloud.storage.BaseStorageService.deleteObject(BaseStorageService.scala:158)
at org.sunbird.cloud.storage.BaseStorageService.deleteObject$(BaseStorageService.scala:153)
at org.sunbird.cloud.storage.service.AzureStorageService.deleteObject(AzureStorageService.scala:9)
at org.ekstep.analytics.job.JobManager$.init(JobManager.scala:37)
at org.ekstep.analytics.job.JobManager$.main(JobManager.scala:32)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at optional.Application.callWithOptions(application.scala:251)
at optional.Application.callWithOptions$(application.scala:223)
at org.ekstep.analytics.job.JobManager$.callWithOptions(JobManager.scala:22)
at optional.Application.main(application.scala:258)
at optional.Application.main$(application.scala:255)
at org.ekstep.analytics.job.JobManager$.main(JobManager.scala:22)
at org.ekstep.analytics.job.JobManager.main(JobManager.scala)
Caused by: org.sunbird.cloud.storage.exception.StorageServiceException: endpoint.getHost() is null for https:///telemetry-data-store?restype=container&comp=list&prefix=transient-data&include=metadata
at org.sunbird.cloud.storage.BaseStorageService.deleteObjects(BaseStorageService.scala:175)
at org.sunbird.cloud.storage.BaseStorageService.deleteObjects$(BaseStorageService.scala:162)
at org.sunbird.cloud.storage.service.AzureStorageService.deleteObjects(AzureStorageService.scala:9)
at org.sunbird.cloud.storage.BaseStorageService.deleteObject(BaseStorageService.scala:155)
... 15 more
Caused by: java.lang.IllegalArgumentException: endpoint.getHost() is null for https:///telemetry-data-store?restype=container&comp=list&prefix=transient-data&include=metadata
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:146)
at org.jclouds.http.HttpRequest.(HttpRequest.java:246)
at org.jclouds.rest.internal.GeneratedHttpRequest.(GeneratedHttpRequest.java:83)
at org.jclouds.rest.internal.GeneratedHttpRequest$Builder.build(GeneratedHttpRequest.java:63)
at org.jclouds.rest.internal.RestAnnotationProcessor.apply(RestAnnotationProcessor.java:337)
at org.jclouds.rest.internal.RestAnnotationProcessor.apply(RestAnnotationProcessor.java:137)
at org.jclouds.rest.internal.InvokeHttpMethod.toCommand(InvokeHttpMethod.java:189)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:85)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy50.listBlobs(Unknown Source)
at org.jclouds.azureblob.blobstore.AzureBlobStore.list(AzureBlobStore.java:168)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
at com.sun.proxy.$Proxy51.list(Unknown Source)
at org.sunbird.cloud.storage.BaseStorageService.$anonfun$deleteObjects$1(BaseStorageService.scala:166)
at org.sunbird.cloud.storage.BaseStorageService.$anonfun$deleteObjects$1$adapted(BaseStorageService.scala:164)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.sunbird.cloud.storage.BaseStorageService.deleteObjects(BaseStorageService.scala:164)
... 18 more
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mount/data/analytics/spark-3.1.3-bin-hadoop2.7/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mount/data/analytics/models-2.0/batch-models-2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mount/data/analytics/models-2.0/analytics-framework-2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mount/data/analytics/models-2.0/data-products-1.0/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
(Total Events Size,0)
(setting spark.cassandra.connection.host to lp-cassandra,10.1.2.4)
(Total Events Size,0)
org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.ekstep.analytics.framework.util.RestUtil$._call(RestUtil.scala:29)
at org.ekstep.analytics.framework.util.RestUtil$.post(RestUtil.scala:70)
at org.ekstep.analytics.model.MonitorSummaryModel$.$anonfun$postProcess$7(MonitorSummaryModel.scala:99)
at org.ekstep.analytics.model.MonitorSummaryModel$.$anonfun$postProcess$7$adapted(MonitorSummaryModel.scala:97)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at org.ekstep.analytics.model.MonitorSummaryModel$.postProcess(MonitorSummaryModel.scala:97)
at org.ekstep.analytics.framework.IBatchModelTemplate.execute(IBatchModelTemplate.scala:27)
at org.ekstep.analytics.framework.IBatchModelTemplate.execute$(IBatchModelTemplate.scala:20)
at org.ekstep.analytics.model.MonitorSummaryModel$.execute(MonitorSummaryModel.scala:39)
at org.ekstep.analytics.framework.driver.BatchJobDriver$.$anonfun$_processModel$1(BatchJobDriver.scala:106)
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
at org.ekstep.analytics.framework.util.CommonUtil$.time(CommonUtil.scala:548)
at org.ekstep.analytics.framework.driver.BatchJobDriver$._processModel(BatchJobDriver.scala:105)
at org.ekstep.analytics.framework.driver.BatchJobDriver$.$anonfun$_process$1(BatchJobDriver.scala:78)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.ekstep.analytics.framework.driver.BatchJobDriver$._process(BatchJobDriver.scala:66)
at org.ekstep.analytics.framework.driver.BatchJobDriver$.process(BatchJobDriver.scala:51)
at org.ekstep.analytics.framework.driver.BatchJobDriver$.process(BatchJobDriver.scala:23)
at org.ekstep.analytics.framework.JobDriver$.run(JobDriver.scala:31)
at org.ekstep.analytics.job.summarizer.MonitorSummarizer$.main(MonitorSummarizer.scala:19)
at org.ekstep.analytics.job.JobExecutorV2$.main(JobExecutor.scala:27)
at org.ekstep.analytics.job.JobRunner.executeJob(JobManager.scala:121)
at org.ekstep.analytics.job.JobRunner.run(JobManager.scala:94)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.http.ProtocolException: Target host is not specified
at org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:71)
at org.apache.http.impl.client.InternalHttpClient.determineRoute(InternalHttpClient.java:125)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
... 32 more
org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.ekstep.analytics.framework.util.RestUtil$._call(RestUtil.scala:29)
at org.ekstep.analytics.framework.util.RestUtil$.post(RestUtil.scala:70)
at org.ekstep.analytics.model.MonitorSummaryModel$.$anonfun$postProcess$7(MonitorSummaryModel.scala:99)
at org.ekstep.analytics.model.MonitorSummaryModel$.$anonfun$postProcess$7$adapted(MonitorSummaryModel.scala:97)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at org.ekstep.analytics.model.MonitorSummaryModel$.postProcess(MonitorSummaryModel.scala:97)
at org.ekstep.analytics.framework.IBatchModelTemplate.execute(IBatchModelTemplate.scala:27)
at org.ekstep.analytics.framework.IBatchModelTemplate.execute$(IBatchModelTemplate.scala:20)
at org.ekstep.analytics.model.MonitorSummaryModel$.execute(MonitorSummaryModel.scala:39)
at org.ekstep.analytics.framework.driver.BatchJobDriver$.$anonfun$_processModel$1(BatchJobDriver.scala:106)
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
at org.ekstep.analytics.framework.util.CommonUtil$.time(CommonUtil.scala:548)
at org.ekstep.analytics.framework.driver.BatchJobDriver$._processModel(BatchJobDriver.scala:105)
at org.ekstep.analytics.framework.driver.BatchJobDriver$.$anonfun$_process$1(BatchJobDriver.scala:78)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.ekstep.analytics.framework.driver.BatchJobDriver$._process(BatchJobDriver.scala:66)
at org.ekstep.analytics.framework.driver.BatchJobDriver$.process(BatchJobDriver.scala:51)
at org.ekstep.analytics.framework.driver.BatchJobDriver$.process(BatchJobDriver.scala:23)
at org.ekstep.analytics.framework.JobDriver$.run(JobDriver.scala:31)
at org.ekstep.analytics.job.summarizer.MonitorSummarizer$.main(MonitorSummarizer.scala:19)
at org.ekstep.analytics.job.JobExecutorV2$.main(JobExecutor.scala:27)
at org.ekstep.analytics.job.JobRunner.executeJob(JobManager.scala:121)
at org.ekstep.analytics.job.JobRunner.run(JobManager.scala:94)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.http.ProtocolException: Target host is not specified
at org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:71)
at org.apache.http.impl.client.InternalHttpClient.determineRoute(InternalHttpClient.java:125)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
... 32 more
org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.ekstep.analytics.framework.util.RestUtil$._call(RestUtil.scala:29)
at org.ekstep.analytics.framework.util.RestUtil$.post(RestUtil.scala:70)
at org.ekstep.analytics.model.MonitorSummaryModel$.$anonfun$postProcess$7(MonitorSummaryModel.scala:99)
at org.ekstep.analytics.model.MonitorSummaryModel$.$anonfun$postProcess$7$adapted(MonitorSummaryModel.scala:97)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at org.ekstep.analytics.model.MonitorSummaryModel$.postProcess(MonitorSummaryModel.scala:97)
at org.ekstep.analytics.framework.IBatchModelTemplate.execute(IBatchModelTemplate.scala:27)
at org.ekstep.analytics.framework.IBatchModelTemplate.execute$(IBatchModelTemplate.scala:20)
at org.ekstep.analytics.model.MonitorSummaryModel$.execute(MonitorSummaryModel.scala:39)
at org.ekstep.analytics.framework.driver.BatchJobDriver$.$anonfun$_processModel$1(BatchJobDriver.scala:106)
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
at org.ekstep.analytics.framework.util.CommonUtil$.time(CommonUtil.scala:548)
at org.ekstep.analytics.framework.driver.BatchJobDriver$._processModel(BatchJobDriver.scala:105)
at org.ekstep.analytics.framework.driver.BatchJobDriver$.$anonfun$_process$1(BatchJobDriver.scala:78)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.ekstep.analytics.framework.driver.BatchJobDriver$._process(BatchJobDriver.scala:66)
at org.ekstep.analytics.framework.driver.BatchJobDriver$.process(BatchJobDriver.scala:51)
at org.ekstep.analytics.framework.driver.BatchJobDriver$.process(BatchJobDriver.scala:23)
at org.ekstep.analytics.framework.JobDriver$.run(JobDriver.scala:31)
at org.ekstep.analytics.job.summarizer.MonitorSummarizer$.main(MonitorSummarizer.scala:19)
at org.ekstep.analytics.job.JobExecutorV2$.main(JobExecutor.scala:27)
at org.ekstep.analytics.job.JobRunner.executeJob(JobManager.scala:121)
at org.ekstep.analytics.job.JobRunner.run(JobManager.scala:94)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.http.ProtocolException: Target host is not specified
at org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:71)
at org.apache.http.impl.client.InternalHttpClient.determineRoute(InternalHttpClient.java:125)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
... 32 more
(Total Events Size,1)
[ERROR] [02/07/2024 12:14:54.289] [scruid-actor-system-akka.actor.default-dispatcher-17] [akka.stream.Log(akka://scruid-actor-system/system/StreamSupervisor-2)] [scruid-load-balancer] Upstream failed. (akka.stream.AbruptTerminationException: Processor actor [Actor[akka://scruid-actor-system/system/StreamSupervisor-2/flow-0-2-mapAsyncUnordered#246628504]] terminated abruptly)

[sowmya-dixit]

@raghupathiguduri
From the logs it looks like job-manager is able to run the reports submitted.
The below error message can be ignored as it logs error for checking tmp location and proceeds further for execution -
Caused by: java.lang.IllegalArgumentException: endpoint.getHost() is null for https:///telemetry-data-store?restype=container&comp=list&prefix=transient-data&include=metadata

You can find respective report logs under mount/data/analytics/scripts/logs/joblog.log file.
Let us know if need any other details.

[raghupathiguduri]

@sowmya-dixit How can we check what all reports are enabled as part of job-manager and if we want to add any new report to the job-manager list, how can we do that ?

How can we find where it is storing the reports in the blob ? as we are not able to find them in the reports container.

[sowmya-dixit]

@raghupathiguduri
Currently these are the reports enabled as part of job-manager by default. For any addition of reports, you can override this variable and add report ids.
Also we have report service APIs to enable reports on top of Druid which will also be executed via job-manager by default along the above job list. These Druid reports enabled are available via get reports list API or can be found under report_config postgres table. Any new Druid reports can be added using submit report API.

And for finding reports in the blob depends on report configurations. For example, WFS - sends data to kafka, doesn't write to blob. Druid reports will be available in the blob & path depends on report configuration, these report details can be found using get report API.

Note: Keep monitor-job-summ job at the end of the jobs list here as it will provide status of the reports execution at the end into slack channel.