Just a simply query on what is the actual max file size supported by NFS emulation. I know that it is an abstraction upon MData, which has a max file size of 1MiB if I recall, but does NFS enable larger files than this limit?
In extension to that, if it can handle larger files, how have people found the performance to be with larger files?
Yes, because the files themselves are stored indirectly.
I.e. what is stored in MData itself is just a file descriptor structure, which you can find here. The data_map_name field points to a special structure, a data map, which can be thought of as a private encryption key and also a pointer to a data location. This data structure is created by our self-encryption library, which splits the file into chunks and stores them as separate Immutable Data objects.
These chunks themselves are limited to roughly 1 MiB in size for now.
The trick is, as the file data can be split into multiple Immutable Data objects (the logic for it is defined here), the file size becomes virtually unlimited. However, it will be limited by the size of the data map, which itself is stored in an Immutable Data object, hence the max. size of a data map is 1 MiB.
Edit: also worth noting that of course the data map itself can be split into chunks, of course, but that can quickly become impractical as far as the CPU usage and the network bandwidth are concerned.
This limit is not on the Alpha 2 network itself. See the Alpha 2 post:
When uploading files using Web Hosting Manager, the maximum file size is 20 MiB per file.
So it’s just a hard-coded limit for users who are uploading files using the Web Hosting Manager app. It doesn’t affect the SAFE API (Node.js API, Web API, etc.)
Okay so at the moment, I am presuming that this limit may be removed at some point, the way to achieve the largest files on the network is through NFS (max size of the data map)?
Which limit? It’s still possible to upload files of virtually unlimited size, by splitting them into multiple chunks of 1 MiB each (this is done automatically by the self_encryption library).
We can recursively self encrypt, so a large data map can itself be encrypted into chunks and a smaller datamap.
Okay I see what you are saying. So if my understanding is correct, something like Home - Documentation can write an almost arbitrary amount of data to the network? As the self_encryption library will chunk all the data in the background.
@DaBrown95 if you succeed in this it would be great to have a simple ‘how to’ post to refer to as I’m sure this will get asked and asked again. Good luck!