20231218 developer call notes

[hammer]

Pre-notes

• Notes from previous meeting

PRs

• sgkit PR#1138 Prevent parts of VCF being reparsed

Issues

• sgkit#1162 Main build is failing

Discussions

• JH call with NVIDIA summary

Notes

Attendees

1. Jeff Hammerbacher
2. Tom White
3. Jerome Kelleher
4. Ben Jeffery
5. Tim Millar
6. Rafal Wojdyla

Discussion

• 0.8.0 release
• Numba cache
  • BJ: Dask errors are different with same underlying cause
  • JK: disable cache for now until we can get to the bottom of it
  • TM: should we have 2 switches, one for JIT, one for gu vectorize?
• Big, messy VCF
  • BJ: instrumenting sgkit to report memory allocation when reading a large VCF
  • JK: motivated by a desire to demo QC
• sgkit PR#1138
  • BJ: can’t figure out why coverage check is failing
  • TW: me neither!
• sgkit#1162
  • TW: xarray upstream changes failing our builds
• sgkit-publication#46
  • JK: need a figure to show how Zarr stores data
• Publication story
  • Figure 1: Zarr storage vs other formats
  • Figure 2: afdist for computing on the whole genotype matrix: sgkit scales with threads, users can implement themselves.
    • JK: IO causes scaling to stop at 10^6 samples
    • BJ: GE has a 750 GB RAM box to test!
    • JH: fold speedup vs. wall clock time? JK: hard to see with wall clock.
  • Figure 3: afdist on a subset of the genotype matrix
  • Next story: scale out
    • JH: these figures end at 10^6; if our story is about big biobanks, we will need scale out
    • JK: will talk about scale out then point to Liangde’s thesis
    • JH: how do we talk about PLINK?
    • JK: users are pretty well served, it’s feature rich and fast. Let’s focus on people who want to use Python and want to be able to scale out.
    • TM: Are we going to get asked about TileDB?
    • JH: The TileDB array data storage manager (2016)
  • After that: applications
    • QC
    • Stat gen
    • Pop gen
    • Quant gen
    • Phylogenetics
  • TM: do we have a list of what uses GPUs?
    • JK: only pairwise distance!
    • TM: actually 653 has GPU stuff
    • TM: https://github.com/pystatgen/sgkit/pull/673 too!