Shared folder used for big files needed by many VB tasks?

[hucker75]

This may already be available, and if so I'll ask LHC why they're not using it. Can a VB app download a large (say 1GB) data set which will probably be used by many other VB tasks, and place it in a shared folder to stay there for later, or concurrent on other cores, tasks? At the moment, LHC VB tasks download a huge dataset before starting, but within the virtual environment. Then another task starts and downloads the same dataset. Very inefficient on disk space, internet usage for the user, and internet usage for the project.

Yes I know there's squid, but even having 11 years experience in tech support, I can't get it to work. I'm sure most LHC users don't run it.

[davidpanderson]

That's a great idea. The natural way to implement is to share the project directory
(where the big file is) with the VM.
AFAIK Vbox lets a VM have multiple shared directories:
https://docs.oracle.com/en/virtualization/virtualbox/6.0/user/sharedfolders.html

This change would be entirely in vboxwrapper. It would be a small change (few dozen lines).

[computezrmle]

Basically a good idea if many tasks need exactly the same set of input data.
LHC@home covers this part using a large vdi file that includes a Linux OS, scientific software and a prefilled CVMFS cache (up to 2 GB, depending on the subproject).

The VM then downloads outdated data as well as task specific data via a CVMFS client and/or a Frontier client.
Both use HTTP for data transport.
Hundreds (if not thousands) of Squids run by CERN/Fermilab and partners around the world ensure a reliable and efficient data distribution.
To support LHC@home's transfer needs CERN/Fermilab even distribute data via Cloudflare's CDN which is also based on hundreds of Squids.

Based on decades of experience the CVMFS/Frontier experts recommend a local Squid if more than 5-10 nodes are running in the same LAN environment request CVMFS/Frontier data.
Posts at the LHC@home forum explain how to efficiently configure a local Squid, even on Windows.
Logs from LHC@home show that local Squids are widely used, especially (but not only) by high potential volunteers.

Efficiency of a local Squid depends on various parameters, e.g. number of cores running LHC@home or subproject mix.
Typical values are >90 %, in some cases >98 %.

[hucker75]

I did use squid, but I can't get it to run anymore, on either of my fast Windows machines. It needs to be made plug and play, not fiddling about with settings manually.

However, since squid is caching files needed by many LHC tasks, having Boinc do this itself would achieve the same outcome without any user intervention. Whenever a file is downloaded in side the VM, store it in a Boinc folder for use later, perhaps with an expiry date should it not be used for some time, to save disk space.

[computezrmle]

It needs to be made plug and play, not fiddling about with settings manually.

Not even BOINC works plug and play.
It needs to be actively attached to at least 1 project, or an account manager, preferences need to be correctly set ...

In case of Squid this is as easy as possible (in short):

1. Install Squid
2. Replace the default config file with the BOINC optimized config file from LHC@home
3. Add the local machines/networks to that config file
4. Restart Squid
5. Tell BOINC to use that Squid

... squid is caching files needed by many LHC tasks, having Boinc do this itself ...

Why implement a cache (from scratch) when there's a ready to use one based on decades of experience?

Whenever a file is downloaded in side the VM, store it in a Boinc folder ...

VMs run independent from BOINC - like a computer in the garage and another one in the house.
BOINC does not know if/when a VM downloads anything - like the computer in the garage doesn't know if/when the computer in the house downloads anything.

As for LHC@home:
Each CVMFS client inside a VM runs it's own local cache which is very similar to Squid's disk cache.
If an object requested from that cache is stale CVMFS downloads it from an upper level - here by default a nearby Squid from Cloudflare's CDN.

A local Squid can/should be inserted between a client and the default CDN to cover the following objectives:
6. cache objects that are requested by more than 1 VM running either concurrently or consecutively
7. cache objects that are not cached inside a VM like those requested by the VM's Frontier client
8. cache objects that are requested by other apps, e.g. independent CVMFS clients on Linux hosts

[hucker75]

Anyone can install Boinc without any technical knowledge, you just click buttons and add a project. But Squid is ported from Linux and it feels like I'm running Linux, I have to enter commands. I did get it to run for a couple of months, after following very longwinded instructions, which referred me to other instructions and so forth, but now it won't start. I reinstalled, uninstalled, tried to wipe all evidence of it and start again, but it won't run. It also won't run on another machine, I've given up with it, it's too complicated.

Boinc already remembers files for other projects, I don't see why it can't be done for a VM.