What is the current max File size supported by NFS?

Just a simply query on what is the actual max file size supported by NFS emulation. I know that it is an abstraction upon MData, which has a max file size of 1MiB if I recall, but does NFS enable larger files than this limit?

In extension to that, if it can handle larger files, how have people found the performance to be with larger files?

1 Like

There is a limit for alpha 2 on each immutable data which I think is 20 MBytes.

3 Likes

Yes, because the files themselves are stored indirectly.

I.e. what is stored in MData itself is just a file descriptor structure, which you can find here. The data_map_name field points to a special structure, a data map, which can be thought of as a private encryption key and also a pointer to a data location. This data structure is created by our self-encryption library, which splits the file into chunks and stores them as separate Immutable Data objects.
These chunks themselves are limited to roughly 1 MiB in size for now.

The trick is, as the file data can be split into multiple Immutable Data objects (the logic for it is defined here), the file size becomes virtually unlimited. However, it will be limited by the size of the data map, which itself is stored in an Immutable Data object, hence the max. size of a data map is 1 MiB.

Edit: also worth noting that of course the data map itself can be split into chunks, of course, but that can quickly become impractical as far as the CPU usage and the network bandwidth are concerned.

11 Likes

Fabulous explanation, thank you very much. :grinning:

This limit is not on the Alpha 2 network itself. See the Alpha 2 post:

When uploading files using Web Hosting Manager, the maximum file size is 20 MiB per file.

So it’s just a hard-coded limit for users who are uploading files using the Web Hosting Manager app. It doesn’t affect the SAFE API (Node.js API, Web API, etc.) :slightly_smiling_face:

6 Likes

Oh that’s interesting, what is the limit on immutable data?

As @nbaksalyar mentioned above:

2 Likes

Okay so at the moment, I am presuming that this limit may be removed at some point, the way to achieve the largest files on the network is through NFS (max size of the data map)?

Which limit? It’s still possible to upload files of virtually unlimited size, by splitting them into multiple chunks of 1 MiB each (this is done automatically by the self_encryption library).

We can recursively self encrypt, so a large data map can itself be encrypted into chunks and a smaller datamap.

3 Likes

Okay I see what you are saying. :grinning: So if my understanding is correct, something like Home - Documentation can write an almost arbitrary amount of data to the network? As the self_encryption library will chunk all the data in the background.

2 Likes

Yes, exactly :smiley:

2 Likes

Fantastic, thanks for your help! :grinning: Will be trying out files in the new year on the order of 1MiB to 75GiB, so should be interesting!

4 Likes

If you try this on the Alpha 2 network, you might run out of mutation requests though :stuck_out_tongue:

But you could try it on the mock network or set up your own test network.

6 Likes

Haha yes I would imagine that won’t be allowed :joy:

I’m hoping to try both those approaches out, should be fun!

5 Likes

@DaBrown95 if you succeed in this it would be great to have a simple ‘how to’ post to refer to as I’m sure this will get asked and asked again. Good luck!

2 Likes

I sure will, will be able to include links to my code too so hopefully should act as a good resource. :+1:

1 Like