pkg/blobserver/diskpacked implement StreamBlobs TODO #1401
Conversation
|
@tgulacsi Can you rebase? I intend to review this soon, by the way, hoping for this weekend. Your renewed activity in perkeep is motivating for me, I'd love to keep collaborating on stuff with you :) |
Implement the TODO that raised a concert about a lot of garbage created by reading all the blobs, always. This now reads only when the blob.ReadFull is called.
tgulacsi
force-pushed
the
dp-streamblobs
branch
from
825fe62
to
21fad03
Compare
| data := make([]byte, size) | ||
| if _, err := io.ReadFull(r, data); err != nil { | ||
| return err | ||
| var br *blob.Blob |
There was a problem hiding this comment.
We usually use the short br for blobrefs, but this is a blob.
| { | ||
| fn := fd.Name() | ||
| size, offset := size, offset | ||
| if _, err = io.CopyN(ioutil.Discard, r, int64(size)); err != nil { |
There was a problem hiding this comment.
Does this cause us to read the full blob? Can we just not read it?
| fhc.m[fn] = fh | ||
| return fh, nil | ||
| } | ||
| func (fhc *fileHandleCache) Put(fh *fileRefCount) { |
There was a problem hiding this comment.
I may just be unfamiliar with the naming for similar datastructures, but what does "put" mean here?
I see that it decreases the refcount and possibly remove the file handle from the cache.
Looking at the code it seems that the maximum number of file handles isn't really enforced?
What if they all have a refcount of 1? What is going to stop us from caching an unlimited of file handles if the user does not read a single blob?
Optimize walking blobs for not reading them.
Second part of #1395