feat(sdk): add infrastructure for roll-out of next generation python client

[dmitryduev]

Description

This PR adds infrastructure for the roll-out of the new Go-based python client.

_WANDB_NG=true python

import wandb

s = wandb.Session()
print(s.address)
s.teardown()

with wandb.Session() as s:
    print(s.address)

• I updated CHANGELOG.md, or it's not applicable

Testing

How was this PR tested?

[codecov]

Codecov Report

Attention: Patch coverage is 63.90533% with 61 lines in your changes missing coverage. Please review.

Project coverage is 75.76%. Comparing base (15f4422) to head (cda6fb4).
Report is 57 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7641      +/-   ##
==========================================
+ Coverage   74.29%   75.76%   +1.47%     
==========================================
  Files         500      502       +2     
  Lines       55727    54168    -1559     
==========================================
- Hits        41400    41041     -359     
+ Misses      13918    12718    -1200     
  Partials      409      409              

Flag	Coverage Δ
func	41.30% <58.75%> (-0.06%)	⬇️
system	63.43% <63.31%> (-0.08%)	⬇️
unit	56.30% <60.94%> (+0.94%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
core/cmd/wandb-core/main.go	71.64% <100.00%> (ø)
core/pkg/filestream/filestreamimpl.go	86.27% <100.00%> (+19.09%)	⬆️
core/pkg/observability/util.go	86.66% <100.00%> (+9.16%)	⬆️
core/pkg/observability/sentry.go	61.81% <50.00%> (+3.84%)	⬆️
wandb/sdk/wandb_init.py	84.02% <50.00%> (-0.16%)	⬇️
wandb/lib/__init__.py	0.00% <0.00%> (ø)
wandb/__init__.py	93.51% <93.39%> (-0.35%)	⬇️
wandb/lib/lib.py	0.00% <0.00%> (ø)

... and 100 files with indirect coverage changes

[timoffex]

invert if statement to reduce nesting

if x { doLotsOfStuff() } => if !x { return } doLotsOfStuff()

[timoffex]

it's not immediately clear to me that this refers to a connection to the wandb-core internal process

Session is a connection to the internal "core" process.

also, maybe it would be more informative to call this InternalProcess instead of Session. Each "session" corresponds to one process, but the naming leaves open the possibility that there could be multiple sessions per process.

[dmitryduev]

Elaborated what connection we're referring to in the docstring.

As for the naming, I think Session is more appropriate. We are making it public with the intent to allow Go users to connect to a wandb-core service already running either locally or on the network.

[timoffex]

the intent to allow Go users to connect to a wandb-core service already running

I see, that makes sense. It has a Start() method that launches a new internal process; I suppose we'd eventually add a Connect() method?

What if a user wants to start a process without connecting to it, with the intention to connect from other processes? It seems there's an opportunity to separate concepts:

• WandbProcess is a reference to a running process.
  • WandbProcess::Connect() Session opens and returns a connection to the process.
  • WandbProcess::Kill() kills the process.
• Session is a connection to a running W&B process.
  • Session::Close() closes the connection and frees any resources.
  • Other methods for communicating with the session.

These should be interfaces to make code that uses them testable.

Optionally, we can have WandbProcessLauncher and SessionStarter, but I think we can get away with top-level Launch(settings) WandbProcess and Connect(address) Session functions.

The simplest usage in Go would be:

process := wandb.Launch(...)
defer process.Kill()

session := process.Connect()
defer session.Close()

session.Send(...)

Or to connect to a process running elsewhere:

session := wandb.Connect(...)
defer session.Close()

session.Send(...)

[timoffex]

by the name, this sounds like it would launch the process, not manage it

[timoffex]

the intent to allow Go users to connect to a wandb-core service already running

I see, that makes sense. It has a Start() method that launches a new internal process; I suppose we'd eventually add a Connect() method?

What if a user wants to start a process without connecting to it, with the intention to connect from other processes? It seems there's an opportunity to separate concepts:

• WandbProcess is a reference to a running process.
  • WandbProcess::Connect() Session opens and returns a connection to the process.
  • WandbProcess::Kill() kills the process.
• Session is a connection to a running W&B process.
  • Session::Close() closes the connection and frees any resources.
  • Other methods for communicating with the session.

These should be interfaces to make code that uses them testable.

Optionally, we can have WandbProcessLauncher and SessionStarter, but I think we can get away with top-level Launch(settings) WandbProcess and Connect(address) Session functions.

The simplest usage in Go would be:

process := wandb.Launch(...)
defer process.Kill()

session := process.Connect()
defer session.Close()

session.Send(...)

Or to connect to a process running elsewhere:

session := wandb.Connect(...)
defer session.Close()

session.Send(...)