Does the SAFE network currently implement any sort of compression for public, unencrypted data?

General Support
,
1

I thought this was a discussion worth having, as at the moment the implementation of the SAFE network browser is hiding whether any of the files received by the browser were compressed:

screen1
screen11021×211 9.78 KB

You can see that it sets the Size of each file received as 0 Bytes. If you hover over the size of a file, it shows you the resource size, but again, it’s hiding how much was actually sent over the network. Presumably there is some networking data which the browser needs to be made aware of and I would consider this a bug which needs fixing:

screen2

In terms of compression, the self_encryption libraries contain a reference to Brotli, however I~ can’t see this being used anywhere else, for instance:

Is the network currently any using compression, and if not, why? At network launch it’s my belief that CPU time will be cheap, but bandwidth will be hard to come by. We should be doing everything we can to reduce the size of resources sent over the network where we can, a good compression scheme (like Brotli) can reduce resource size by 60-70% (obviously I don’t need to tell you this, but I’m aware that none-technical people also read this forum so I’m adding it for posterity)

2 Likes
2

Aye, the browser itself doesn’t handle this at the moment. It’d likely just be a matter of adding sizing headers to the request to get this in there. Which I think is available on the immutable data objects.

That’s a separate point to compression, which I’m not aware of the current state in network. Maybe @lionel.faber you know the current state of things there?

3 Likes
3

self encrypt first stage is compress before encrypt and then lastly xor. The last stage is where we get security not from encryption, strangely enough. Encrypt is to randomize the output of compression, then xor with deterministic pad (only if you have the original file) is where the security comes in.

tl;dr self_encrypt does compress.

3 Likes
4

Does that only apply to the self_encrypt library? What’s the state of compression in the other libraries used by the network as a whole?

5

As David said, compression is done in self_encryption. The upper layers, SCL, safe-api, FFI etc. use the self_encryption functions to store data.

So for public, unencrypted data in case of ImmutableData the data is encrypted and compressed.
Immutable data (public and private) is always chunked and encrypted (if it’s greater than 1 MB).

For AData and MData there is no compression and encryption by default. The entries can be encrypted.

1 Like
6

Fantastic, in that case I think the only actionable work from this would be for @joshuef to add the correct headers in to the response object. Do you think we’ll be able to get the compressed size as well?

I would suggest adding:

content-encoding: br
content-length: N

Where N is the compressed size sent over the network, that seems to be how Chromium currently implements it for http

7

I dont think we’ll be able to get compressed size. Currently we store size in the ID, (which I believe we’re pulling just straight from the filesystem), so we can easily show that.

I’ll happily work this in with a PR as I’m currently fixing a bug in fetch there anyway.

1 Like
8

just a wee note. @Shane, content served isn’t brotli encoded at this point, as its all handled by the safe libs under the hood. :+1:

9

This topic was automatically closed after 60 days. New replies are no longer allowed.