Issue with Lern data-product

[AmiableAnil]

@ddevadat We are facing the issue while running the admin-user-reports in OCI env. We are getting the following error. Can you please help on it?
24/01/31 07:02:49 INFO Executor: Finished task 198.0 in stage 40.0 (TID 2738). 12145 bytes result sent to driver 24/01/31 07:02:49 INFO TaskSetManager: Finished task 198.0 in stage 40.0 (TID 2738) in 37 ms on dp.vmssubnetprivat.edhotfix.oraclevcn.com (executor driver) (200/200) 24/01/31 07:02:49 INFO TaskSchedulerImpl: Removed TaskSet 40.0, whose tasks have all completed, from pool 24/01/31 07:02:49 INFO DAGScheduler: ResultStage 40 (collectAsMap at StateAdminReportJob.scala:109) finished in 2.318 s 24/01/31 07:02:49 INFO DAGScheduler: Job 7 is finished. Cancelling potential speculative or zombie tasks for this job 24/01/31 07:02:49 INFO TaskSchedulerImpl: Killing all running tasks in stage 40: Stage finished 24/01/31 07:02:49 INFO DAGScheduler: Job 7 finished: collectAsMap at StateAdminReportJob.scala:109, took 2.328675 s Exception in thread "main" org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.ServiceException: Request Error: null at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:478) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:427) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleException(Jets3tNativeFileSystemStore.java:411) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:181) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at org.apache.hadoop.fs.s3native.$Proxy48.retrieveMetadata(Unknown Source) at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:476) at org.apache.hadoop.fs.s3native.NativeS3FileSystem.delete(NativeS3FileSystem.java:420) at org.ekstep.analytics.framework.util.HadoopFileUtil.$anonfun$delete$1(HadoopFileUtil.scala:36) at org.ekstep.analytics.framework.util.HadoopFileUtil.$anonfun$delete$1$adapted(HadoopFileUtil.scala:34) at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38) at scala.collection.TraversableLike.map(TraversableLike.scala:238) at scala.collection.TraversableLike.map$(TraversableLike.scala:231) at scala.collection.AbstractTraversable.map(Traversable.scala:108) at org.ekstep.analytics.framework.util.HadoopFileUtil.delete(HadoopFileUtil.scala:34) at org.ekstep.analytics.framework.util.DatasetExt.saveToBlobStore(DatasetUtil.scala:76) at org.sunbird.userorg.job.report.StateAdminReportJob$.saveUserSelfDeclaredExternalInfo(StateAdminReportJob.scala:173) at org.sunbird.userorg.job.report.StateAdminReportJob$.generateExternalIdReport(StateAdminReportJob.scala:97) at org.sunbird.userorg.job.report.StateAdminReportJob$.execute(StateAdminReportJob.scala:47) at org.sunbird.userorg.job.report.StateAdminReportJob$.main(StateAdminReportJob.scala:42) at org.ekstep.analytics.job.JobExecutor$.main(JobExecutor.scala:16) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at optional.Application.callWithOptions(application.scala:251) at optional.Application.callWithOptions$(application.scala:223) at org.ekstep.analytics.job.JobExecutor$.callWithOptions(JobExecutor.scala:8) at optional.Application.main(application.scala:258) at optional.Application.main$(application.scala:255) at org.ekstep.analytics.job.JobExecutor$.main(JobExecutor.scala:8) at org.ekstep.analytics.job.JobExecutor.main(JobExecutor.scala) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: org.jets3t.service.ServiceException: Request Error: null at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:625) at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:279) at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1052) at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2264) at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2193) at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1120) at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:575) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:174) ... 48 more Caused by: java.lang.NullPointerException at org.jets3t.service.utils.ServiceUtils.findBucketNameInHostOrPath(ServiceUtils.java:845) at org.jets3t.service.impl.rest.httpclient.RestStorageService.authorizeHttpRequest(RestStorageService.java:732) at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:326) ... 55 more 24/01/31 07:02:49 INFO SparkContext: Invoking stop() from shutdown hook

cc: @mohitga @gandham-santhosh @Hari-stackroute @kumarks1122

[AmiableAnil]

Thanks @ddevadat for scheduling the call.

We have updated the spark-3.1.3-bin-hadoop2.7/conf/jets3t.properties as suggested by you.
Now we are getting the following exception
24/02/01 07:30:37 INFO DAGScheduler: Job 7 finished: collectAsMap at StateAdminReportJob.scala:109, took 2.325728 s 24/02/01 07:30:37 INFO BlockManagerInfo: Removed broadcast_32_piece0 on dp.vmssubnetprivat.edhotfix.oraclevcn.com:45321 in memory (size: 65.2 KiB, free: 1925.9 MiB) Exception in thread "main" java.lang.ClassCastException: class java.lang.String cannot be cast to class java.util.Date (java.lang.String and java.util.Date are in module java.base of loader 'bootstrap') at org.jets3t.service.model.StorageObject.getLastModifiedDate(StorageObject.java:376) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:176) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at org.apache.hadoop.fs.s3native.$Proxy48.retrieveMetadata(Unknown Source) at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:483) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) at org.ekstep.analytics.framework.util.HadoopFileUtil.$anonfun$delete$1(HadoopFileUtil.scala:42) at org.ekstep.analytics.framework.util.HadoopFileUtil.$anonfun$delete$1$adapted(HadoopFileUtil.scala:37) at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38) at scala.collection.TraversableLike.map(TraversableLike.scala:238) at scala.collection.TraversableLike.map$(TraversableLike.scala:231) at scala.collection.AbstractTraversable.map(Traversable.scala:108) at org.ekstep.analytics.framework.util.HadoopFileUtil.delete(HadoopFileUtil.scala:37) at org.ekstep.analytics.framework.util.DatasetExt.saveToBlobStore(DatasetUtil.scala:85) at org.sunbird.userorg.job.report.StateAdminReportJob$.saveUserSelfDeclaredExternalInfo(StateAdminReportJob.scala:173) at org.sunbird.userorg.job.report.StateAdminReportJob$.generateExternalIdReport(StateAdminReportJob.scala:97) at org.sunbird.userorg.job.report.StateAdminReportJob$.execute(StateAdminReportJob.scala:47) at org.sunbird.userorg.job.report.StateAdminReportJob$.main(StateAdminReportJob.scala:42) at org.ekstep.analytics.job.JobExecutor$.main(JobExecutor.scala:16) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at optional.Application.callWithOptions(application.scala:251) at optional.Application.callWithOptions$(application.scala:223) at org.ekstep.analytics.job.JobExecutor$.callWithOptions(JobExecutor.scala:8) at optional.Application.main(application.scala:258) at optional.Application.main$(application.scala:255) at org.ekstep.analytics.job.JobExecutor$.main(JobExecutor.scala:8) at org.ekstep.analytics.job.JobExecutor.main(JobExecutor.scala) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 24/02/01 07:30:38 INFO SparkContext: Invoking stop() from shutdown hook

[ddevadat]

are you still getting the authentication error. Is the s3 access credentials working fine?

[AmiableAnil]

endpoint configuration was wrong in the config. After adding the write value that got resolved. Now getting the above issue.

[ddevadat]

What the deployment tags and repo used for

Build/DataPipeline/AnalyticsCore
Build/DataPipeline/CoreDataProducts
Build/DataPipeline/EdDataProducts
Build/Lern/LernDataProducts

[ddevadat]

is this issue resolved?

[AmiableAnil]

Yes @ddevadat, this issue is resolved.
This was happening due to jets3t library version. We have find out that OCI PR was reverted here https://github.com/Sunbird-Obsrv/sunbird-core-dataproducts/blob/release-5.1.1/batch-models/pom.xml
We have informed the Obsrv team regarding this.

OCI PR for reference: Sunbird-Obsrv/sunbird-analytics-core@3099d7d#diff-c23aca5ef227b852c7bef6813f0cc40221885f89741a1a072376fa6d8873685bL166

Thanks for your quick response and support.