OpenPAI Backlog

[scarlett2018]

This issue is a long term backlog for planning and discussion. Please feel free to add "the top in your mind" to this issue directly.

• 2021 Feb Release Plan 2021 Feb Release Plan #5253
• 2021 March Draft Plan 2021 March Release Plan #5346

Last Updated on 02/01/2021

Top Focus Scenario

• Support HiveD Management experience
• AI Optimized Workload E2E support
  • use case study for internal & external typical workloads

---

Multi-Cloud/Multi-Cluster
- Hybrid installation
- multi/cross cluster management
- multi-cluster job scheduling

Job management

• Favorite Jobs

AutoScaler

• new added node's provisioning

Engineering Excellent

• CI/CD
• Build process

More examples for the current examples

Add user team support

• Jupyter Notebook support

• FC detect the preemption, gc, eviction, etc. @yqwang-ms

• FC rescale Framework @yqwang-ms

• Update the official docker image (comparing with deepo) @hzy46

Installation experience

• Topology planning
• Installation pre-check (especially for installation that start from non-clean env)
• Azure installation practices
• SKU support for complex job types
• Network plugin deployment
• migrate python2 to python3 Upgrade Python 2 to Python 3 #5042
• A kube runtime plugin to export parameters to env A kube runtime plugin to export parameters to env #4636

[fanyangCS]

#3872

[scarlett2018]

History Backlog Info Backup, only for reference.

Brainstorming on 2020/09/10

[Planned in #4898 ] Cell as sku in hived scheduler.

• Refactor duplicate deployment code.
• Refactor job submission form.

[Planned in #4898] Support dynamic sku types in different vc.

• Remaining works in New RestServer Architecture: RestServer -> DB -> ApiServer New RestServer Architecture: RestServer -> DB -> ApiServer #4651

@hzy46 :
[Planned in #4898] pod/event watcher to support #4649
[Planned in #4898 ] multi cluster management detailed design & review

• merge previous doc change Fix document using DRI tickets #4828 Add distributed training examples of PyTorch #4821
• db gcer
• archive job (tag) Support delete/archive jobs #4453
• dbcontroller optimization items Database poller will be dead locked when 30000+ concurrent RPC calls happen #4849 Memory issues for framework watcher and poller in DBController #4850 Database controller performance optimization candidate items #4794
• make DBController tolerant to wrong framework request make DBController tolerant to wrong framework request #4889
• ui to let admin add storage how to realize this sentence: "please use RESTful API to assign storage to the groups of the user"?  #4806

@suiguoxin: #4789
[Planned in #4898] Alert-manager: Kill low-gpu-utilization jobs, tag abnormal jobs, Cordon node with k8s API when GPU GCC Error
[Planned in #4898] Job tags : DB table, rest-server API, web-portal refactor

@yiyione:
[Planned in #4898] Group management page in webportal
[Planned in #4898] VC request management for user and admin

• VSCode client bug fix

@debuggy:

• marketplace related items 2020 Sept Release Plan openpaimarketplace#60
  [Planned in 2020 Sept ~ Oct release plan #4898]Grammar check
  • NNI job integration -- Feature Engineering
  • add info for items in submit job page
  • add "Upload" feature for dataset, set upload standard and workflow
• job submit refactor
  • new submit workflow design
  • team storage
  • sku
• job detail work part 2
  • job retry more detail
  • show events in job detail page

Per Task Retry History

• FC generates Task history snapshots (mocked Task CRD object with UID) @yqwang-ms
• Fluentd + DB to collect above snapshots @hzy46
• WebPortal expose the Per Task Retry History (and failure counts grouped by failure type within one job?) @debuggy

Brainstorming on 5/14 - last updated at 5/28

• Candidate ideas about PAI experience improvements Candidate ideas about PAI experience improvements #3338

Multi-Cloud/Multi-Cluster
@hzy46
- Hybrid installation
- multi/cross cluster management
- multi-cluster job scheduling

AutoScaler @ydye

• new added node's provisioning

Support for large scale cluster

• (June) set up latency benchmark report for 100 virtual nodes --> 1500 virtual nodes * 8 job @ydye
• based on the stress and performance test result, analyze and plan for improvements

GPU Utilization

• (June) improve training cost ROI for EDU scenario @hzy46

• improve current internal training workload utilization @yiyione

• (June) Performance improvement for DB (Use Database as History Server Use Database as History Server #4610) @hzy46

More examples for the current examples

Add user team support

Surfacing more backend error to users in Job Details Page #4649
@qfyin ,@yqwang-ms
potential failure case: storage mount failed

• Surfacing more backend error to users (GCC error, out of storage, etc.)

• K8S Pod keep waiting in creating container, such as ErrImagePull K8S Pod keep waiting in creating container, such as ErrImagePull #3572

• GPU Fairness Usage @Binyang2014

• Usability improvement based on user study and feedback @yangou1988

• Service plug-in (user customized) @hzy46

• Jupyter Notebook support @hzy46

• Runtime interface @Binyang2014

• HiveD Improvements (i.e. aware per GPU bad, etc.) @yqwang-ms @zhypku

• FC detect the preemption, gc, eviction, etc. @yqwang-ms

• FC rescale Framework @yqwang-ms

• Update the official docker image (comparing with deepo) @hzy46

Installation experience

• Topology planning

• Installation pre-check (especially for installation that start from non-clean env)

• Azure installation practices

• SKU support for complex job types

• Network plugin deployment

• Support for new hardware

• migrate python2 to python3 Upgrade Python 2 to Python 3 #5042

• A kube runtime plugin to export parameters to env A kube runtime plugin to export parameters to env #4636

These are incomplete items from v0.17

• Grafana metrics link does not have time interval

• Skip intra communication for elastic job (Binyang P1 - Skip intra communication for elastic tasks #3708)

• Bring back debug mode (Binyang Reserve container after job failed. #3643, Better Job Debugging Experience for End Users #2210)

• replace disk cleaner with k8s pod eviction policy ([pure-k8s] Disk usage monitor module #3340, [pure k8s] K8s's pod eviction may not trigger when job container uses too much disk space #3765)

• When job is waiting, show the detailed waiting reason somewhere in the job page. (Show queueing duration and running duration in job detail #3847)

• Archive logs to system provided storage

• Exec to job containers on webportal

• persist service log (persistent log for PAI services #2129)

• differentiate opportunistic job and normal jobs in VC usage page.

• improve user group adding to VC experience

• improve HiveD configuration experience

• Show job priority in webportal

• introduce a new job state "initialization" (A job (or a pod of a job) may get stuck in a state neither running nor waiting #4141)

• open ssh connection in browser

• ssh key management and simplifying user behavior of ssh plugin

• open ssh connection in browser

• favorite jobs My favorite jobs and Jobs shared to me #2749

• actual resource usage of a job V1.0.0/1 allocated job resource usage API #4695

• Favorite Jobs

---

Engineering Excellent

• CI/CD
• Build process

---

Low Priority Postponed items

• (Postponed from Aug 2020 release) Elastic DL and Job level scale up and scale down - @yqwang-ms check with Ming/NNI and other integration scenarios - TODO: summarize the to-the-customer level scenarios supports