RFC 41 – Low Level API

General RFCs
,
1

Discussion topic for RFC 41 – Low Level API

5 Likes
2

Nice. Will read. I mentioned on the other forum, but when y’all get to the launcher side of this can y’all take a peek at Discuss - 0028-launcher-low-level-api · Issue #115 · maidsafe/rfcs · GitHub? Thanks.

4 Likes
3

Sure:

Is this the normal operation or is this launcher specific? If it’s something already supported and is a “SAFE” concept in general, most of the questions below are moot since everyone pays the same cost.

These are restrictions imposed by the SAFE Network. safe_core already enforces it and vaults should do it if not already.

Can we get a real world use case here? Let’s pretend safecoin was implemented today and I wanted to store 5 GB.

How many chunks would that store? I forget whether it’s 3MB a piece or what.

Each ImmutData would be <= 1 MB so 5GB/1MB chunks.

How large would the data map be for a 5 GB store before it was re-stored as ID (there aren’t any details I can find on the format data maps might use to marshal into ID)?

I dont have a precise answer but this should give a rough estimation: DataMap is a Map of 64bytes chunk vs another 64bytes. So each row is 128 bytes. So it would need 1MB/128bytes number of rows to reach ~1MB size. Say you have 1GB = 1024MB file. It will get divided into 1 MB chunks and each row in DataMap will correspond to 1 MB of file. In total you have 1024 chunks == 1024 rows => 1024 *128 bytes = size of DataMap for 1 GB file.

How many recursions until it gets < 1MB and how many extra ID objects might this cost (in time and safecoin)? It’s worth comparing this expense vs if I hadn’t used the launcher and was willing to store a large data map myself in some other way.

There is no way to not use it as the size restriction is applied by the Network itself.

How can you tell the contents of the ID are just another data map vs the actual contents that may appear datamap-like? Will the the resulting data map somehow carry the number of levels of recursion or something?

This is completely handled by safe_core in implementation specific way. If safe_core gave you the ID it will give you the final data too. If you used some other means to get the ID then ofcourse safe_core would error out saying it couldn’t make sense of it.

Why 1 MB? I might prefer 100K just in case I wanted to store it in SD. Maybe configurable or is this giving apps that are using the launcher too much control? Then again, I guess I could store the 1MB back in ID and do the same recursion myself that you are doing.

These were numbers that were just picked and will be profiled. So say you have 10 MB file - the data map will point to 10 chunks as each will be max 1 MB. You can store the data-map itself in StructuredData as for 10 MB file it will be pretty small. Now instead if the max size of Immutable data were to be 100 KB as you say, 10 MB file would get divided into 1000 chunks and instead of 10 you would do 1000 gets to reconstruct it. So it’s just about striking a balance. But this is not something we have frozen and right now you can view it as a number we thought would be a good starting point. Might need to change it in future if there are good arguments for using any other value.

2 Likes
4

This being the tricky part here since these numbers will have to factor in a few variables while profiling their consequence. Few such as:

  • Data concentration in a given group. So if say size wasn’t cap’d at all and was just free-for-all, then famous SD could grow quite large in size putting more load in a given group during churn to efficiently relocate data

  • At the same time, too small a number could drastically increase the number of GETs being made across the network for a simple file which would affect not just performance of client but the network itself too maybe.

2 Likes
5

I think the struct_data_ is verbose. Maybe sd_ prefix doesn’t hide any meaning and would be worth changing to.

2 Likes
6

Launcher shall enforce a rule of separate secretbox::gen_key() for each app […] Though the keys are persistent, there is no way for Launcher to know if one app is mimicking another during the authentication process. So Launcher will ask user each time an app starts and tries to register with Launcher

Why doesn’t app use a random string like firefox.x2ll8jja.default? This is how Firefox create profiles. It’ll get rid of the annoying pop-ups. I understand this is handled on the launcher part, so there is no need to change this RFC in respect to the safe_core API. Better yet would be to a “subprofile” within the profile, like username and password, but any password would work and it’d give you a different profile.

1 Like
7

Not sure if this belongs here or into RFC 42, as I’d fix it in the user-facing part of the app, but RFC 41 is currently talking about it specifically, so I want to ask this here. I posted a follow-up there.

In the RFC 41 under Detailed Design it reads:

Currently one of the functionality of Launcher is to provide sandboxing. Apps which pass through the Launcher (which is the recommended approach because it is considered bad to give one’s credentials to every app) have access to data either within specific folder created for them or within SAFEDrive which is where common data is. No app is allowed to access data in a folder reserved for another app. However this guarantee will be broken once the low level API’s are exposed because apps will have freedom to create whatever data they want and wherever they want it on the network. Under the current implementation this would mean that private data stored by one app can be potentially compromised (accessed by another app). For e.g. say App-0 creates and stores StructuredData abc somewhere in the network. If App-1 uses a direct GET for abc there is no way Launcher knows this should not be allowed. Previously apps were only allowed to travel a directory hierarchy to get data and Launcher could assert it travelled only the permissible ones.

To get around this limitation, Launcher shall enforce a rule of separate secretbox::gen_key() for each app that registers successfully with it. All private data created on the network by the app will use this to encrypt/decrypt data. These keys will need to be persistent, so Launcher will write the details in its configuration file against the registered app. Which keys will be used will be determined by CipherOption below. Note that, new nonce shall be generated everytime encryption is used.

I am not sure I understand this second paragraph: get around what limitation. The Limitation of being sandboxed? Or to ensure that one App can’t write (accidentally) as the other on the same key?

Either way, do I understand correctly that then the id is globally unique? Or will it be namespaced for the same user somehow? Because all that reads that if someone (accidentally) wrote the userProfile-id, this would the network-wide globally unique. Which would be terrible as no other app could ever use that name again, or any other that has a potential collision. The only way you could half-way assure you aren’t overwriting other stuff apps would be to namespace (aka prefix) the key yourself and use a super-random (as in crypto-random UUID) key. It would also make collisions more likely the more popular the app gets.

1 Like
8

The reading part is prevented by having a separated encryption keys per app. An app will not be able to read data of another app. Modification of data is however possible because currently there is only one signing key and that’s all the vaults evaluate. It is explained here.

I don’t know what userProfile-id is - is that something frontend specific ? It’s not used in safe_core as far as i know.

Anyway, you cannot accidentally fetch and (attempt to) read and modify another app’s data unless you know the DataIdentifier, which barring DNS and now Email is random and not guessable. Only if an app made a deliberate attempt to snoop the activities of another app to figure it out can it get one. And even then the reads would fail as data would be encrypted by keys not accessible to this app.

9

Thanks @ustulation for the information.

I am clearly missing some information here: for the information the API is provided does end up with one specific (XOR)-Resource-Location in the network, correct? So, how is the information present (App-ID, resource-id, DataIdentifier etc) combined to create that location?

My (maybe wrong) assumption was that without app-sandboxing, two apps referring to the same resource ID (in the example named userProfile) would try to read and write the same exact resource in the network. I wasn’t too concerned with those failing to read (or write) but more with the problem that they pollute each others GLOBAL space and thus creating unexpected side-effects upon one another (even only be overwriting the others apps content). Similarly like any variable not prefixed with the var attribute in javascript pollutes the global namespace of all other functions – and thus creating terrible side-effects on one another. Something I’d really try to avoid.

You added DataIdentifier into that group of things now – something that at least on the LOW LEVEL API with the launcher, I have not provided (and no idea how I would to). Either way, if those act as globals, I’d find it rather problematic to just define general terms there, too. For the same clashing reasons. Is that identifier documentation somewhere I could understand its meaning and how it is used as part of the XOR location? It follow along the lines of why apple’s appstore and googles play store insinst on having app-ids in at .-notation with at least three entities (in all lowercase): otherwise Who gets to claim to be the global “email”-app?

10

The only namespacing in the network is provided by the virtue of data being of different types. So a XOR space X can be simultaneously occupied by an SD, a Pub AD, a Priv AD and an ImmutData. Apart from that there is no namespacing. The address (as far as the network is concerned) is completely based on the data itself - it has nothing to do with the creator/owner of the data. So ImmutData with name X or an SD with name Y will always be located with nodes closest to that address irrespective of who created them. If I have an SD with name Y and you try to PUT/POST/DELETE it, vaults will not allow as you don’t own it as ownership is tied with SD/AD. However for ImmutData X we both can do a PUT and although the network will charge us both, data will be de-duplicated and only one copy present - this is because ImmutData have no ownership tied to them.

One e.g. When nfs creates directory hierarchy, all the directory listings are SD’s. The id of the SD is crypto-random for each and thus gets created in random places in the network. However if it happend that you got the same id (and type-tag) which is already in use, then you will get a creation failure and will merely have to re-create the directory which will give a new random ID and will probably not clash.

For cases where the ID is deterministic however - e.g. in case of DNS / email etc. it is hash of certain user-input string, it is 1st come 1st serve basis. So currently if my DNS long-name is pepsi.co.in you cannot take that anymore. Same for emails - for emails it is true in clearnet too. For DNS, to me personally, it’s not as black-and-white. In clearnet domains were taken as the companies emerged - you would probably not have anticipated taking pepsi.com before it existed/became famous. But here the thing is, domains are already famous so you can take pepsi.com and advertise your local non-filtered cigarettes brand.

There is no global email app as such. It’s all first come first serve basis. An app can add prefixes. So if you entered email as ABC and the app was asdf it could create abc@asdf.com or something. But there is no prevention of another app taking that address space for another user already.

1 Like
11

There’s a blog post or two in this question/response… thanks guys!

2 Likes