go-lazy

Description

go-lazy is a library that includes a lazy reader and writer that allows delayed opening of a resource. The LazyReader and LazyReadCloser can be used to delay opening a resource until the resource is actually read. The LazyWriterAt and LazyWriteCloser can be used to delay creating a resource until the first byte is actually written. The LazyFile can be used to delay opening a file and allocating a file descriptor until it is required.

Usage

Go

You can import go-lazy as a library with:

import (
  "github.com/spatialcurrent/go-lazy/pkg/lazy"
)

The easiest pattern is to simply use a closure function to open up a file or remote data source, so that you can delay opening the resource until it is actually needed.

return lazy.NewReader(func() (io.Reader, error) {
  // open up a file or remote data source and return as a reader.
})

return lazy.NewReadCloser(func() (io.ReadCloser, error) {
  // open up a file or remote data source and return as a reader.
})

writer := lazy.NewLazyWriteCloser(func() (io.WriterAt, error) {
  // open up a pipe
})

writer := lazy.NewLazyWriterAt(func() (io.WriterAt, error) {
  // open up a file or byte buffer
})

A good use case is for reading multiple gzip files in sequential order. The gzip reader can automatically read through multiple files. This approach keeps the number of file descriptors open to 1 at a time while still using an io.MultiReader for convenience.

r, err := gzip.NewReader(io.MultiReader(
  NewLazyReader(func() (io.Reader, error) {
    return first file
  }),
  NewLazyReader(func() (io.Reader, error) {
    // return second file
  }),
))

The LazyFile can also be used to delay opening or stating a file if it is passed to a callback. LazyFile matches the os.File API when possible, but is not a drop in replacement. The API is identical for Read, Stat, and Close methods, but the Fd method now returns the file descriptor with an error if any.

f := lazy.NewLazyFile(name, os.O_RDONLY, 0)
err := callback(f)
if err != nil {
  return err
}
// do something with the file

Another good use case if for using the s3manager Downloader to concurrently download multiple files. To avoid many empty files for downloads that haven't started yet, use LazyWriterAt.

writer := lazy.NewLazyWriterAt(func() (io.WriterAt, error) {
  file, err := os.Create(input.Path)
  if err != nil {
    return nil, fmt.Errorf("error creating destination file %q: %w", input.Path, err)
  }
  return file, nil
})
_, err = downloader.DownloadWithContext(input.Context, writer, &s3.GetObjectInput{
  Bucket: aws.String(input.Bucket),
  Key:    aws.String(input.Key),
})

See go.dev for information on how to use Go API.

Testing

To run Go tests use make test_go (or bash scripts/test.sh), which runs unit tests, go vet, go vet with shadow, errcheck, ineffassign, staticcheck, and misspell.

Contributing

Spatial Current, Inc. is currently accepting pull requests for this repository. We'd love to have your contributions! Please see Contributing.md for how to get started.

License

This work is distributed under the MIT License. See LICENSE file.