Query on druid ingestion

[ddevadat]

We are trying to understand data consumption in druid. Have a few queries regarding this.

1. Which ingestion method is used in druid for sunbird . ie. streaming( from kafka ) or batch ( from object storage ) or both.
2. What is the data format. I am assuming its json, but wanted to confirm
3. Can i have a sample data for each of the ingestion method.
4. Can i have the data structure for each of the ingestion method.
5. Whats the typical data size
6. How frequently the data is refreshed. I understand the telemetery related jobs are more traffic intensive, but do we have any numbers regarding this

[manjudr]

@ddevadat

Q1 - The Sunbird OBSRV systems support both streaming ingestion through Kafka and batch ingestion from object storage. There are several jobs which generate records and ingest them into Druid in batch ingestion mode.

Q1 - Apache druid provides a capability to ingest both JSON and CSV formats. However, most of our data products and kafka ingestion ingest the data in json format only.

[manjudr]

Most of the ingestion specs are defined here - https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/tree/master/ansible/roles/druid-ingestion/templates and https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/tree/master/ansible/roles/data-products-deploy/files
You can use these to understand the structure of the data.

[manjudr]

Q5 - What's the typical data size - Could you please elaborate on this? do you mean the number of events that are processed per day?

[ddevadat]

@manjudr thanks for the responses

is there a sample json data available for a given ingestion spec. e.g for a streaming ingestion spec
https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/blob/master/ansible/roles/druid-ingestion/templates/raw_telemetry_events

do you have sample data file ?

Similarly for a batch ingestion spec
https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/blob/master/ansible/roles/druid-ingestion/templates/raw_ml_obs_domain_index

do you have sample data file.

Q5> Say if we talking about https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/blob/master/ansible/roles/druid-ingestion/templates/raw_telemetry_events

what will the events process per day on avg and what will be the size of the data file

[manjudr]

@ddevadat

1. Sample for raw_ml_obs_domain_index ingestion spec -

{"program_name":"CBSE SQAA","program_id":"60dacb6ff595770aafd6a8e6","solution_name":"CBSE-School Quality Assessment and Assurance","solution_id":"611395bf820a6555d7434699","parent_channel":"SHIKSHALOKAM","solution_type":"observation_with_rubric","private_program":"false","domain_name":"MANAGEMENT AND GOVERNANCE","domain_externalId":"D5","domain_level":"L1","criteria_name":"Overall: Domain 7 - Beneficiary Satisfaction","criteria_score":"","criteria_id":"611395bf820a6555d7434698","unique_submissions":5,"unique_entities":1,"unique_users":2,"unique_solution":1,"time_stamp":"2023-10-20T12:24:31.794+05:30"}

Please check with the ML team to understand more about the program dataset

Sample for the raw_telemetry_events -

{"eid":"INTERACT","ver":"3.0","syncts":1645775959825,"ets":1.645775959676E12,"flags":{"ex_processed":true,"pp_validation_processed":true},"mid":"INTERACT:5fb98fff1e9f3b8b8e72f3875e972e1e","type":"events","tags":["ORG_001"],"actor":{"id":"cae25d303f2c094a3cb23bc14f49d27f","type":"User"},"edata":{"id":"login","type":"click","pageid":"login"},"@timestamp":"2022-02-25T07:59:19.825Z","context":{"channel":"ORG_001","pdata":{"id":"dev.sunbird.portal","ver":"4.6.0","pid":"sunbird-portal"},"env":"public","sid":"b2f0c750-9610-11ec-9811-996022e74793","did":"cae25d303f2c094a3cb23bc14f49d27f","cdata":[],"rollup":{"l1":"ORG_001"},"uid":"anonymous"},"object":{}}