Dumb DOM API questions

happybeing · July 18, 2017, 3:44pm

Starting from the DOM API docs I have some observations and questions that no doubt come from a lack of understanding as to what the API does and how to use it, but which might be useful as feedback for the documentation.

Much of this may be irrelevant and naive, or answered in the NodeJS API (but I’m ignoring it for now to help test these docs) in which case by all means say so and if you dare, help me understand better

Initialisation

the app init and authorisation process is complicated and it is not obvious what is happening or why so many API functions are required, and I wonder if if can be simplified. But perhaps understanding will reveal all…
what is a “session” and why do we (app devs) need to know about it?
could an app always have access to an unauthorised session without having to know about it (ie initialised on first use)? Or if the idea is that an app could have multiple sessions, could we change the model so that instead of safeApp.initialise() you roll the app and the session into one object, and if an app needs multiple sessions, provide a way for it to initialise multiple safeApp objects instead? So for vast majority of apps, you never need to do this and can eliminate the concept of sessions from the API, and can eliminate safeApp.connect and safeApp.connectAuthorised?
am I right in thinking that the own_container’s of different apps are differentiated by the combination of AppInfo properties (ie the values from all properties { id: 'net.maidsafe.test.webapp.id', name: 'WebApp Test', vendor: 'MaidSafe Ltd.'} in which case I suggest pointing this out for mentions of both AppInfo and own_container. If not, what distinguishes one own_container from another?
how as an app dev would I know which containers I can request and any conventions around their use?
what happens when requesting permissions for a container that doesn’t exist, and is there a way to test if a container exists and create it if not, or to enumerate containers?
to confirm: for unauthorised / read only access the app should use safeApp.initialise() and then safeApp.connect()
to confirm: for authorised / read/write access the app should use safeApp.initialise() and then safeApp.authorise() and then safeApp.connectAuthorised(), and then optionally safeApp.authoriseContainer() in order to expand permissions further.
the naming of appToken and authUri seem inconsistent with normal usage (no expert here, so I’m asking rather than sure about this). ‘token’ is I think usually something returned by a web authorisation API. The docs do say that appToken is a ‘handle’, so would if be clearer to change appToken to appHandle, and change authUri to authToken?

NFS <<<< NEW (EDIT 3)

[NEW from here until “Miscellaneous”]

To try and understand how to use NFS it was necessary to examine web_hosting_manager source, and I think I’ve learned the following, so I’ll summarise here things which I hope are true and think need to be made apparent in the docs (either in an overview or where appropriate in the relevant function descriptions). I also have some questions… naturally

to use NFS, first obtain a mutable data handle (mdHandle) for mutable data object of the relevant container, for example using safeApp.getContainer(appHandle, '_public')
then use the mdHandle to obtain an nfsHandle to access the NFS emulation for the given mutable data, by calling
safeMutableData.emulateAs(mdHandle, 'NFS')
to upload files with paths relative to the container:
first, store the file as immutable data and obtain a FileHandle for the file:
safeNfs.create(NFSHandle, content)
then include this file in the container’s mutable data:
safeNfs.insert(NFSHandle, FileHandle, 'safenetwork.ico')
Or, to place the file in a subdirectory of the container, include the full path relative to the container, as in:
safeNfs.insert(NFSHandle, FileHandle, ‘icons/safenetwork.ico’)```
Multiple subdirectories are allowed in a path, such as ‘icons/system/safenetwork.ico’ etc.
question: once the mutable data for the container is full does the NFS emulation silently create a chain of mutable data to handle an arbitrary number of files, and if not, how could this be handled? I’m seeing a lot of problems with it silently creating a daisy chain of mutable data (e.g. how to list the files on such a chain? and for large chains going through files becomes unmanagable etc.). So maybe what I should ask is what are the limitations on NFS emulation for a container?
suggestion: I understand that safeNfs.update() updates the content of a file at an already existing path, in which case I think the description “Replace a path with a new file. Directly commit to the network.” would be clearer as “Update the content of an existing file. Immediately commit to the network.”
question: how would you remove a file from a container, so its name (path) would not be present when calling safeMutableDataEntries.forEach() etc?
note: I don’t understand how versions are used. I haven’t looked into this yet, but it isn’t clear from the docs at this stage.

Miscellaneous

This is a placeholder for comments on other parts of the documentation or API.

I’m finding it necessary to look at both the NodeJS API and the code to try and understand the DOM API (so there will be comments about SAFE NodeJS here API/docs too - sorry )
… this also means that the DOM (and other) docs will benefit from expanded explanations of the concepts, data structures, their characteristics and typical uses, with reference to relevant parts of the corresponding API. Some of this can be separated in to an overview section, but I think the individual APIs could also have more guidance. For example: window.safeMutableData.newPublic() has a parameter name. The function description refers to this as an ‘address’ but without relating it to ‘name’ as in “Initiate a mutuable data at the given address with public access”. The parameters section describes name as “Xor name/address of the MutbleData [sic]” and I’m left wondering if I can use anything I like here, or if I have to construct some special thingy that is an XOR address, but there not enough information here to understand what qualifies and how what you choose here affects anything. Making a not very educated guess, I suspect it is unnecessarily confusing to mention ‘Xor’ in this context because I think it refers to the underlying implementation and is irrelevant to the app, but I may be wrong. Either way I suspect this and other areas of the documentation and/or API terminology could be clarified or simplified from the app dev’s point of view.
NodeJS: window.safeApp.getHomeContainer would be better called window.safeApp.getOwnContainer (or own_container renamed home_container for app authorisation etc. to show that they are referring to the same thing).

dirvine · July 19, 2017, 8:16pm

Just pinging @Krishna and @Shankar_S to chase this up I hope. Cheers Mark, excellent feedback.

bochaco · July 20, 2017, 6:35am

Thanks @happybeing, I agree this is excellent feedback. I’m partially replying here as I was working on the responses but didn’t finish with all. This is of course my personal point of view and we will be reviewing them with the whole team and merging it with any other feedback ofc:

That’s correct, it’s constructed with the AppInfo.id and AppInfo.scope, i.e. the container created can be found at apps/<AppInfo.id>[/AppInfo.scope]

You can use window.safeApp.getContainersNames to get the list of containers.

There should be an error returned with the promise being rejected. You can test if you have access to a container using window.safeApp.canAccessContainer, and getContainersNames to list them.

Correct.

I agree it’s a point to consider.

I think that shouldn’t happen, we should aim at enhancing the documentation for both APIs (and any future API and its docs.).
It is also true that making the documentation really good to the point that someone who doesn’t even know how the network works or what its internal data structures are (like the MutableData) takes a lot of effort since you need to add explanations of what everything is apart of explaining how to interact with it. So I think this is probably a long process. E.g. the own_container param can trigger long explanation of what it is, if it’s private or public data, its location and difference from other containers (if any), the impact in the behaviour of other API functions when it’s created or when it’s not, etc.
…and after writing this I saw it’s quite aligned with your next item

I’m always and particularly concerned about these type of things when working on any API, but as you can see the API does expose many internal concepts of the SAFE network, it’s not completely abstracted from them, so it can be as confusing for the non-educated user to have this matching between internal concepts with API, as it could be for educated users if we abstract the documentation and names in the API too much. I also believe and agree there could be some abstraction API layer which exposes just data structure types of functions where you don’t need to explain much of how they are stored internally by the network at all.

Yes, it makes sense to me to keep consistency as much as possible to avoid confusing the user (a dev user in this case).

bochaco · July 20, 2017, 9:17am

It has to do with the new authentication model, as you know there are two separate steps that an app has to perform:
1- get authorised by the user thru the authenticator
2- use the authorisation URI that was provided to actually connect to the network.

If the app is designed to store the auth URI locally (securely cached), it could use it to connect without needing the authenticator, this is the main reason you have at least to separate steps when invoking the API functions. I think the main purpose was to be able to have this decoupled to support mobiles so you don’t need to authenticator to be running in your phone every time you run your app, but you can also think of the IoT devices that need to be pre-provisioned with an auth URI so they can connect without the authenticator.
I guess you are imagining of having a function which does step 1 and 2 altogether, i.e. authoriseAndConnect, and maybe also which includes the step of initialising the library (initialise) too, all these options can be considered I guess, I think in such a case they shouldn’t be a replacement but just additional helper functions.

Unless I’m misunderstanding, there are no sessions as you are describing it, the initialise function basically just creates a safeApp instance which dynamically loads the safe_app library and it makes some initial setup like activating the logging.
When you call authorise you are not really creating a session but just getting an authorisation URI (or an auth token) that you can then use to connect. After you received the auth URI, there is no sessions neither with the network nor with the authenticator.
At this point, you can connect to the network using the auth URI and start interacting with it until the safeApp instance is deleted. After you are connected, you could re-connect using the same safeApp instance with the same auth URI, or a different auth URI, but you cannot have several connections/sessions with the same safeApp instance, so if an app needs multiple sessions it needs to create multiple safeApp instances (note the same auth URI can be used to connect with all of them, as log as the AppInfo is the same).

DavidMtl · July 20, 2017, 10:17am

Great feedback @happybeing. What is app-token for exactly, is it because it could be useful to run multiple instance of the safe app within a website? If not, I agree the initialise() step could be hidden and even simply done automatically for each tabs of the browser if it’s not too resource heavy.

If I may add a tiny feedback about the doc. The left sidebar is too narrow and we can’t read the whole function call without scrolling it horizontally. I suggest to make it wide enough to be able to see every function without scrolling.

bochaco · July 20, 2017, 10:27am

It’s the safeApp instance handler, you could have multiple safeApp instances in a single app if needed.

happybeing · July 20, 2017, 2:11pm

[@bochaco - now completed, you can look when ready. No hurry.]

Thanks Gabriel. It looks like a clean, well structured and powerful API. I can see a lot of thought has gone into it and enjoying learning about it… much to learn!

What is ‘scope’? Is this AppInfo.name + AppInfo.vendor or something else?

Marvelous, I must read more. Are there any pre-defined containers? I’m curious about the ones used in the docs (_public, _other etc.). We should explain if these are completely made up examples or suggested conventions or blah, and that the underscore is intended to imply X, etc. A link to a short section in an overview “More about Containers” perhaps. I like the approach of keeping words to a minimum but having links for help answer queries - what we have here is excellent reference info IMO.

I’m ok with this layer being the way it is, while expecting that we’ll also arrive at helper libraries that mask this to provide common readily understood design patters, more or less application oriented (blog, forum, database, filesystem, data archive etc.).

So its fine for me to see MutableData and Immutable data at this level. These are new concepts that we should expose because it is important to begin at a level that gives near full access to the underlying functionality, and then abstract for either simplification or specialist needs. Using the term ‘Xor’ didn’t seem necessary is all, so it is nit picking on my part.

I think we’re on the same page hooray!

I’ve since been vassilating between:

safeApp.getOwnContainer / options.own_container

and

safeApp.getAppContainer / options.app_container

But can’t decide which is most intuitive so I’ll leave it to others!

I’m thinking this could be more intuitive and it is why I was asking what a session was. From the API:

window.safeApp.connect

Create a new, unregistered session (read-only), e.g. useful for browsing web sites or just publicly available data.

As an app developer, what do I need to know/do to:

get a read/write app authorised and running
maintain that auth between sessions

So, not saying remove any of the features, but wondering if it can be more straightforward. For example, do we need init, then auth, then connect. Perhaps init can do all three, but provides a way to access the authUri (eg from the app handle?), and which is an optional parameter to safeApp.init() or for use with an alternative safeApp.initWithUri( authUri ), which the app can call if it has an authUri, before falling back to safeApp.init() if initWIthUri() fails. So could we have…

...
var appConfig = { /*blah*/ };
var appHandle = null;
if (authUri)
   safeApp.initialiseWithUri(authUri).then(
       => (initResponse){ appHandle = initResponse; });

if (!appHandle)
  safeApp.initialise( appConfig.appInfo, appConfig.appPermissions, appConfig.appOptions).then(
      => (authResponse){
         appHandle = authResponse.appHandle;
         authUri = authResponse.authUri;
      },
      => (err){ console.log("Unable to connect to SAFEnetwork due to ", err); } );

Or the authUri could be accessible from the appHandle so:

   authUri = safeApp.getAuthUri( appHandle );

Above, I’m rebelliously using appHandle rather than appToken here

I don’t think we lose any functionality with this, but we eliminate “connect()” (which becomes implicit in successful authorisation that we’re also connected), and an app that never stores authUri has only one function to call. So “App Zero” can do the lot in a single line of code.

Thanks for the informative and helpful responses, and your patience. I tend to try running before I can walk as you see I hope my follow up comments make sense but I’m not 100% yet, still groping around. But unfortunately not much time to play further and get properly up to speed. Will do more as and when. Thanks for the great work Gabriel and team. It is very much appreciated here.

happybeing · July 21, 2017, 8:25pm

I just added the following the OP:

NFS <<<< NEW (EDIT 3)

[NEW from here until “Miscellaneous”]

To try and understand how to use NFS it was necessary to examine web_hosting_manager source, and I think I’ve learned the following, so I’ll summarise here things which I hope are true and think need to be made apparent in the docs (either in an overview or where appropriate in the relevant function descriptions). I also have some questions… naturally

to use NFS, first obtain a mutable data handle (mdHandle) for mutable data object of the relevant container, for example using safeApp.getContainer(appHandle, '_public')
then use the mdHandle to obtain an nfsHandle to access the NFS emulation for the given mutable data, by calling
safeMutableData.emulateAs(mdHandle, 'NFS')
to upload files with paths relative to the container:
first, store the file as immutable data and obtain a FileHandle for the file:
safeNfs.create(NFSHandle, content)
then include this file in the container’s mutable data:
safeNfs.insert(NFSHandle, FileHandle, 'safenetwork.ico')
Or, to place the file in a subdirectory of the container, include the full path relative to the container, as in:
safeNfs.insert(NFSHandle, FileHandle, ‘icons/safenetwork.ico’)```
Multiple subdirectories are allowed in a path, such as ‘icons/system/safenetwork.ico’ etc.
question: once the mutable data for the container is full does the NFS emulation silently create a chain of mutable data to handle an arbitrary number of files, and if not, how could this be handled? I’m seeing a lot of problems with it silently creating a daisy chain of mutable data (e.g. how to list the files on such a chain? and for large chains going through files becomes unmanagable etc.). So maybe what I should ask is what are the limitations on NFS emulation for a container?
suggestion: I understand that safeNfs.update() updates the content of a file at an already existing path, in which case I think the description “Replace a path with a new file. Directly commit to the network.” would be clearer as “Update the content of an existing file. Immediately commit to the network.”
question: how would you remove a file from a container, so its name (path) would not be present when calling safeMutableDataEntries.forEach() etc?
note: I don’t understand how versions are used. I haven’t looked into this yet, but it isn’t clear from the docs at this stage.

bochaco · July 26, 2017, 2:06am

Yes, I agree, moreover, I think there are several other conventions we will need to document eventually, other examples are the type tags values or how the services names are being stored.

Regarding the proposals for the initialise functions, I like your proposal to simplify it, we will need to evaluate all that.

Also there is something worth mentioning, since it was an important aspect of the some design decisions made. The safe_app_nodejs binding API should ideally be just a NodeJS language binding for the API functions exposed by the underlying safe_app API in the safe_client_libs. The reasoning behind is that you don’t want to have redundant algorithms/implementation in several parts, i.e. if we implemented many helper functions in the safe_app_nodejs side, we would then need to replicate the same helper functionality in the other language bindings, like Java, Python, etc., which is not just additional effort but it also helps to have more bugs as you have to maintain them all and keep their implementation (ideally) in sync.
In an analogous way, you don’t want the DOM API to be so different from the others but just to be a binding to allow access to the safe_app_nodejs API from the DOM. Also, you would ideally want to be able to interact with both of them in a very similar way, since I’m sure many SAFE apps will be developed for the web but also provided as mobile/desktop versions which may use the NodeJs or Java bindings API, so you don’t want developers to have very different code (or interaction flow) for each version, all the contrary actually.

All this is just some background information in relation to why it may sound very simple to add some helper functions to the DOM or nodejs API but it needs a bit more of consideration.

Correct, although note that the entries in the MD need to follow the conventions that the NFS emulation follows. So if you have populated it with some entries which don’t follow it (i.e. using plain MD functions), it’s very likely you won’t be able to access those entries using the NFS emulation functions.

There are some enhancements/changes being implemented now, please take a look at this for some details: [MAID-2090] - JIRA

No this is not supported at the moment, and we have to think how to do so, and yes all those issues will need to be taken into consideration. I think this is not only at NFS emulation level, but perhaps even at the MD level. So current limitations are the same as for the MD at the moment, which if I’m right it’s up to 100 entries and up to 1Mb in size.

One of the enhancements being made is the addition of a delete function. Although note that since at the moment it’s not possible to do a hard-remove of a MD entry, its value is currently being cleared when you try to delete it from the MD, thus you would need to temporarily filter those entries with an empty value when iterating them with MD.forEach (this is exactly what we are currently doing in the sample apps, e.g.: https://github.com/maidsafe/safe_examples/blob/master/email_app/app/safenet_comm.js#L174)

Each entry in a MD has a version associated, which needs to be bumped up for every modification, this is also used by the network to consider a mutation valid (I think to prevent something like a replay attack or just delayed change requests reaching some node). So when you want to either remove or update an entry you need to provide the current version + 1. When you insert a new entry it will be inserted with version 0. The same is applicable to the MD permissions.
Now, I agree it would be nice to have this hidden behind the API so you don’t need to provide it when interacting with it, but it’s not clear and/or confirmed this will be the case as of yet.

bochaco · July 26, 2017, 9:52am

I just realised that the information in Jira is for the Rust code.

After reviewing this, all your assumptions were and will still be correct after the changes, the only difference is that the safeNfs.create changes internally (and it doesn’t create an ImmutableData anymore, and it now makes use of the self-encryption mechanism) but you still call to that function first and it will still return a FileHandle which you can use for the other operations, plus some new ones now being added.

happybeing · July 26, 2017, 4:25pm

I’m away from code for a bit but have read and am grateful for your very clear and complete responses. Thank you very much

Just a couple of comments for now then:

regarding versions and your response (quoted) you’ve given exactly what I needed, but I didn’t mean to suggest this should be hidden. In fact exposing this is potentially useful. One case is when updating data where changes can happen out of sync, where the application can detect this and recover. Another is, I think, for allowing stateless interaction, where the app wants to test if it holds the latest version or needs to refresh. So I think having access to a versioned interface might be better than not.
regarding the NFS content limit (100 entries of up to 1MB) I suggest that while this persists it would help to add a note to the relevant API methods (eg insert) because it will be very painful to discover later on!

I want to say again, I really appreciate the care of your responses. You don’t just answer what I explicitly ask, but take time to understand what else is useful to make use of the features I’m asking about. This is rare IME, and very helpful.

@Viv ^^

And no MaidSafeCoin changed hands

Joseph_Meagher · August 1, 2017, 8:08pm

Is it possible to append to mutable data or use versioning to get a specifc versions of the same publicly accessible mutable data handle with different keys and values yet?

bochaco · August 2, 2017, 4:47am

Hi @Joseph_Meagher, unless if I’m understanding what you are saying, it is not possible since the version doesn’t work as in a versioning control system (CVS, git, etc.), i.e. keeping the history of changes, it just keeps the latest version. I’m not sure what you exactly mean by appending in this context though.

Perhaps this helps to understand what the version is for.
Imagine you have the same SAFE app opened in two different devices (PCs, mobiles, etc.), and both instances are trying to modify the same entry in the same MutableData, at the exact same moment, but with different values.
They will both send the request to update it, with let’s say version = 1 (let’s assume this is the first modification being made to the entry after it was inserted with version == 0), one of the requests will reach a consensus first, it’s unknown which one ofc.
So one of the changes will be applied, but the other request should be rejected since it was meant to modify an entry with version 0 to a new value with version 1, but the version was already bumped by the first request to version 1, so this request becomes invalid and rejected to let the app act accordingly.

rob · August 2, 2017, 5:03am

David mentioned a little while ago that it would be possible to update a MD and optionally have the network keep a copy of the previous MD as a “receipt” which is in essence a previous version.

bochaco · August 2, 2017, 5:07am

Yes, you are right, I remember that, at the moment is not like that though.

Joseph_Meagher · August 2, 2017, 1:09pm

Hi @bochaco, What I meant is, is it possible to create a mutable data that everyone on the testnet can access and add data to but the previous data can not be taken away deleted or removed (a bit like using .push to add an item to an array)?

bochaco · August 2, 2017, 1:32pm

Yes, you need to set the permissions to allow insert to ANYONE, and keep the permissions of update and delete to be allowed only to the MD owner’s account/app.

Joseph_Meagher · August 2, 2017, 1:38pm

How do you set a permisson to allow ANYONE and reserve one for the person who made the mutable data originally?
Do you use setuserpermissions for anyone and insertpermissionset for just yourself, or is it the other way around?

bochaco · August 2, 2017, 2:11pm

The email app does that for the inbox:
https://github.com/maidsafe/safe_examples/blob/master/email_app/app/safenet_comm.js#L211

So anyone can append new emails to the inbox but only the owner can remove/update them, so it sets all permissions for the email app’s instance sign key using the quickSetup function on the MD, and it then sets the permissions for ANYONE (using null as the signKey, we will be exposing constants for these things) for insert.

draw · August 2, 2017, 2:21pm

Is it also possible to use such a Mutable data to count the number of unique accounts that have inserted something?
I guess not because then there must be some sort of ‘id’-field with each insertion with a value, unique to the ‘insert’-account, that can’t be set by the ‘insert’-account.