20230904 developer call notes

[hammer]

Pre-notes

• Notes from previous meeting

PRs

• sgkit#1111 Update minimum Dask dependency versions
  • Xarray pin sgkit#934, Windows sgkit#1122
• sgkit#1119 Improve performance of variant_stats

Issues

• Jeff’s mustache
• Publication
• sgkit#1117 "show source" for function in docs shows rst code
• sgkit#1118 Test with Zarr V3

Discussion

• disc#1120 CZI EOSS Cycle 6

Notes

Attendees

1. Jeff Hammerbacher
2. Tom White
3. Jerome Kelleher
4. Ben Jeffery

Discussion

• Jeff’s mustache: gone!
• Build is broken in several ways
  • TW: none have an obvious fix
  • sgkit#934: Xarray pin due to scatterplot perf regression
  • sgkit#1122: Windows build breaking due to Pandas 2 changes
  • JH: remove scatterplot from GWAS tutorial?
  • TW: need to file xarray upstream issue
  • JK: just grab the values and use another scatterplot?
  • TW: Remove 3.8 build on Windows
• sgkit#1119 → Publication
  • Motivation: figure 2 for paper to show scaling properties
  • Comparing to BCF for storage space is good
  • Comparing to bcftools for compute speed is not as good
  • Using allele frequency distribution calculation as the comparison
  • Tim’s PR rescues performance
  • JK: some problem with Numba caching causing crashes
  • JH: show with Numba vs. no Numba?
  • JK: will confuse people to show Numba.
  • JH: compare to Plink?
  • JK: main point of section is to show BCF is a poor format for computation
  • JK: could do plink, but then might need to do hail, and where do you stop?
  • JK: had to make a copy of BCF to run af-dist; another sgkit advantage!
  • TW: likes emphasizing the “feature enrichment” model too
  • JK: hitting scaling issues with 1 million variants
  • BJ: need utility to sniff VCF and help with configuration for scale
• sgkit#1117
• sgkit#1118
  • TW: just opened issue, haven’t looked at it yet
• disc#1120
  • JK: Are we essential or promising?
  • JK: Getting Oxford involved can be a pain
  • JK: Tom getting set up as a supplier for Oxford, should be easier
  • JK: Tom to work on Zarr integrity, CLI, tskit support
  • JH: Let’s talk with NumFOCUS?
  • JH: What would Tom want to work on? Need to know what users want!
  • JK: Let’s check in with Tim M on user needs
  • JH: Could also proposed Cubed work as valuable for larger CZI ecosystem
• Housekeeping
  • Moved next call to 9/18
  • JK gone 9/25
  • Back to regular schedule 10/9