SAFE Network API - Getting started (draft)

Most of the common operating systems, be they mobile or desktop, have default folders like Gallery, Documents etc. One important reason behind these default folders is to have a standard place (based on the type of files to be stored) that other applications can easily collaborate with. For example, a camera application can capture images and save them to the Gallery folder, while an image editing application can look for files from the Gallery folder. Having default folders brings more organised content and also makes it easier to collaborate and share data with other applications.

Default Containers

The SAFE Network follows a similar pattern of providing default containers. Default containers can be shared by applications when the user grants the needed permission. The default containers are:

  • _documents - to store document related data
  • _downloads - to store downloaded content
  • _music - to store music files
  • _pictures - to store images
  • _videos - to store videos
  • _public - to store unencrypted data
  • _publicNames - to store public name/identity which can be looked up for public information

The data stored in the containers must be encrypted, except for the _public container.

In addition to the default containers, applications can create their own root container. Applications can store metadata related to the application or any other data based on the needs.

Data Types

The SAFE Network provides two data types to store and retrieve data. The amount of data that a user can store depends on the Safecoin balance that the user account has. During test networks, the user account limits are regulated based on the number of mutation requests permitted for the account.

Data deduplication is a unique feature of the SAFE Network which is achieved by the process of Self-Encryption. ImmutableData types are subjected to Self-Encryption and stored in the network to avoid data duplication. Binary data and other types of files can be good candidates for storing in the network as Immutable Data. Immutable Data types are cached by the clients and fetching the same can be quicker.

On the other hand, we need a defined structure to store and retrieve data which can be mutated. The second data type, MutableData facilitates in allowing data to be mutated based on the need. MutableData can be visualised as a simple key-value store where data can be stored, retrieved, updated and deleted. In addition, MutableData provides fine-grained access control.

The default containers are built using MutableData and the files are stored as ImmutableData chunks in the network. This is just another use case of using MutableData and ImmutableData.

Client Ecosystem

The authenticator is an important application that users need to create an account and get started on the SAFE Network. The authenticator is packaged with the SAFE Browser.

The SAFE Browser is used to surf safe:// websites and web applications, while the authenticator is used for creating the user account and managing the application authorisations. Applications must be authorised by the user to be able to connect to the SAFE Network on behalf of the user. The user can revoke the granted access at any time from the authenticator.

SAFE Application Development

Application development for the SAFE Network is not any different from the standard practices followed. Applications can easily integrate the safe_app library based on the platform the application is being built on. Node.js and Java are two platforms supported at present. We are eager to expand support to other platforms with time. Web applications can be built using the DOM API of the SAFE Browser.

Similar to the OAuth process, the application sends a request using the library for authorisation. When the authorisation is approved by the user, the application gets a token which is used to connect to the SAFE Network to build decentralised applications.

Node.js API

Npm Dependency

Include safe_app dependency in the package.json. safe_app is not released in the npm repository yet. Should be linked from the git repository for the time being.
Example: "safe-app": "git+https://github.com/maidsafe/safe_app_nodejs/"

The documentation for Node.js safe_app API is hosted here: Home - Documentation

Authorisation

The application must authorise with the authenticator with the needed access permissions. When the user approves the request, application specific encryption keys are generated. The application will be identified in the network using its keys. When the user grants or denies authorisation, the application will receive a URI. Applications can connect to the SAFE Network on behalf of the user by using the URI received.

The application can create its own container, and request access to default containers or other applicationsā€™ containers through the authorisation request.

READ, WRITE, UPDATE, DELETE, MANAGE permissions can be requested for every container.

Cipher Opts

The safe_app library provides crypto API functions. Likewise, CipherOpts is a type which specifies the type of cipher technique to be applied while storing the data in the network.

There are three types of CipherOpts:

  • Plain - Data will not be encrypted.
  • Symmetric - Data is encrypted with a symmetric key.
  • Asymmetric - Data is encrypted using a key pair.

Immutable Data

As the name suggest, the data saved as ImmutableData can not be mutated. Binary content can be written and read from the network using the ImmutableData API. ImmutableData API uses self-encryption to store the data in the network to avoid data deduplication. When the data is stored in the network, a XOR name is returned which is used to reference the data while reading it.

Mutable Data

MutableData can be compared to a simple key-value store. MutableData allows fine-grained access control for collaboration. Permission to insert, update, delete, manage can be requested or permitted for collaboration.

MutableData can be used in different flavours:

  • Public Mutable Data - Data is unencrypted.
  • Private Mutable Data - Data is encrypted.
  • Shared Mutable Data - Data is encrypted and can be used for collaborative purpose. E.g. an inbox where anyone can insert data or allow only whitelisted users to insert.

DOM API

A web application can communicate with the SAFE Network and Authenticator by interacting directly with the SAFE Browserā€™s DOM API, i.e. window.safe* functions.

This API is very similar to the Node.js API, the main difference is that the web application receives handles for each of the objects that are instantiated when interacting with the API, e.g. SAFEApp and MutableData instances. The web app is required to release the handles provided by calling a specific ā€˜freeā€™ function on each of the tokens received (please refer to the examples below for more details).

The full documentation of the SAFE Network DOM API is published here: http://docs.maidsafe.net/beaker-plugin-safe-app

Webapp Example #1

This example code demonstrates how a web app can connect to the authenticator to request for authorisation and receive the auth URI.

function SAFE_Tutorial_Example_1() {
  console.log('SAFE Example #1: Authorising web application from: ', window.location.href);
 
  let app = {
    name: 'Example safe-app web page #1',
    id: 'net.maidsafe.tutorial.webapp.example1',
    version: '0.1.0',
    vendor: 'MaidSafe Ltd.',
  }
 
  let access = {
    _public: ['Read']
  };
 
  window.safeApp.initialise(app)
    .then((appToken) => {
      console.log("Application Token received:", appToken);
      window.safeApp.authorise(appToken, access)
        .then((authURI) => {
          console.log("Application was authorised by user. Auth URI received:", authURI);
          window.safeApp.connectAuthorised(appToken, authURI)
            .then(_ => {
              console.log("Application is now registered in the network");
 
              // Make sure that SAFEApp instance is freed from memory.
              window.safeApp.free(appToken);
              console.log("SAFEApp instance freed.");
            });
        }, (err) => {
          console.warn("Application authorisation was rejected")
        });
    })
    .catch((err) => {
       console.error("Error from webapp: ", err);
    });
};

Webapp Example #2

This example code demonstrates how a web app can connect to the authenticator to request for authorisation and receive the auth URI, use the auth URI to then connect to the SAFE Network, and finally create a MutableData.

function SAFE_Tutorial_Example_2() {
  console.log('SAFE Example #2: Creating a MutableData');
 
  let appInfo = {
    name: 'Example safe-app web page #2',
    id: 'net.maidsafe.tutorial.webapp.example2',
    version: '0.1.0',
    vendor: 'MaidSafe Ltd.',
  }
 
  let access = {
    '_public': ['Read', 'Insert', 'Delete', 'ManagePermissions']
  };
 
  const typeTag = 15000;
  const MUTABLE_DATA_ENTRIES = { key1: 'value1', key2: 'value2' };
 
  window.safeApp.initialise(appInfo)
    .then((appToken) => window.safeApp.authorise(appToken, access)
      .then((authURI) => window.safeApp.connectAuthorised(appToken, authURI))
      .then(_ => window.safeMutableData.newRandomPublic(appToken, typeTag)
        .then((mdata) => {
          console.log('MutableData created, handle: ', mdata);
          return window.safeMutableData.quickSetup(mdata, MUTABLE_DATA_ENTRIES)
            .then(_ => {
              console.log('MutableData setup succeeded');
              return window.safeMutableData.get(mdata, 'key1')
                .then((val) => {
                  console.log('Value successfully retrieved: ', val.buf.toString());
 
                  // Make sure that SAFEApp instance is freed from memory.
                  window.safeApp.free(appToken);
                  console.log("SAFEApp instance freed.");
 
                  // Make sure that MutableData instance is freed from memory.
                  window.safeMutableData.free(mdata);
                  console.log("MutableData instance freed.");
                });
            });
        })
      ))
    .catch((err) => {
       console.error("Error from webapp: ", err);
    });
};

Webapp Example #3

This example code demonstrates how a web app can connect to the authenticator to request for authorisation and receive the auth URI, use the auth URI to then connect to the SAFE Network, after connected, it creates a MutableData, and finally it iterates over the entries, key and values of the created MutableData.

function SAFE_Tutorial_Example_3() {
  console.log('SAFE Example #3: Iterating over MutableData entries');

  let appInfo = {
    name: 'Example safe-app web page #3',
    id: 'net.maidsafe.tutorial.webapp.example3',
    version: '0.1.0',
    vendor: 'MaidSafe Ltd.',
  }

  let access = {
    '_public': ['Read', 'Insert', 'Delete', 'ManagePermissions']
  };

  const typeTag = 15000;
  const MUTABLE_DATA_ENTRIES = { key1: 'value1', key2: 'value2' };

  window.safeApp.initialise(appInfo)
    .then((appToken) => window.safeApp.authorise(appToken, access)
      .then((authURI) => window.safeApp.connectAuthorised(appToken, authURI))
      .then(_ => window.safeMutableData.newRandomPublic(appToken, typeTag))
      .then((mdata) => window.safeMutableData.quickSetup(mdata, MUTABLE_DATA_ENTRIES)
        .then(_ => window.safeMutableData.getKeys(mdata))
        .then((keys) => keys.map((key) => console.log("Key:", key.toString())))
        .then(_ => window.safeMutableData.getValues(mdata))
        .then((values) => values.map((value) => console.log("Value:", value.buf.toString())))
        .then(_ => window.safeMutableData.getEntries(mdata)
          .then((entries) => window.safeMutableDataEntries.forEach(entries,
            (key, value) => console.log("Entry: (", key.toString(), ",", value.buf.toString(), ")"))
          .then(() => {
            console.log("Entries iteration finished");

            // Make sure that MutableData Entries instance is freed from memory.
            window.safeMutableDataEntries.free(entries);
            console.log("MutableData Entries instance freed.");
          }))
        )
        .then(_ => {
          // Make sure that SAFEApp instance is freed from memory.
          window.safeApp.free(appToken);
          console.log("SAFEApp instance freed.");

          // Make sure that MutableData instance is freed from memory.
          window.safeMutableData.free(mdata);
          console.log("MutableData instance freed.");
        })
      )
    )
    .catch((err) => {
       console.error("Error from webapp: ", err);
    });
};
22 Likes

Will those folders and content be visible/accessible to the user?.. the test of which is can they delete them??.. if only for the thought that a buggy app might need that to occur.

[quote=ā€œmaidsafe, post:1, topic:726ā€]
ImmutableData ā€¦ to avoid data deduplication.[/quote]

Confused as thatā€™s suggested twice - would have expected it to be made to avoid duplicationā€¦

[quote=ā€œmaidsafe, post:1, topic:726ā€]
Applications can easily integrate the safe_app library based on the platform the application is being built.[/quote]

Previously there was allusion to some direct rust to rust optionā€¦ but not obvious what that might be.

:thumbsup:

1 Like

oops that was a typo. I just fixed it :slight_smile:

2 Likes

Good stuff, concise understandable, really useful to help get going I think.

This reads a bit like revoking the Authenticator, so maybeā€¦

From the Authenticator, the user can revoke permissions for any app at any time.

Looking guard to paying with this :slight_smile: Not happening soon though :frowning2: still working in DIY and trip preparation.

1 Like

Thank you, this is helpful, but I remain a little confused as to how apps will talk to the SafeNetwork.

Some clarification would be much appreciated:

  1. At some point, there was talk of an SDK rather than an API and this involved NodeJS (and I think React framework)? I was concerned as this seemed to be mandating dependencies and a specific framework, but didnā€™t get deep enough in my understanding to be sure.

  2. It seemed to me from reading that there wouldnā€™t be a REST API in future, but the docs for this are still easily findable and shows http requests and refers to the launcher. GitBook . Is any of this still relevant or not? (Iā€™m pretty sure the launcher is going, right?)

  3. We now have two alternatives, Node.js API and DOM API? Is that correct?
    Is Node.js API able to directly interface between an app and the SafeNetwork without SafeBrowser?

  4. How will native, compiled apps be authenticating and interfacing with SafeNetwork?

Many thanks to anyone with the time to jump in, Iā€™d like to have a bit more clarity before our meetup this evening.

Hi Peter,

Iā€™m not aware of a plan for a React framework at the moment, there is indeed a NodeJs library you can already start using for interacting with the network, which is the one referenced above: (Home - Documentation)

The plan is to not need a Launcher but just the Authenticator which is already embedded in the SAFE browser, so you shouldnā€™t count on this REST API.
However, I think that the information that the launcher was able to display is really useful to the user, so I think we can either embed it as part of the Authenticator in the browser, or perhaps update the Launcher to become just a monitoring/statistics tool which can be run separately like any other SAFE app.
On he other hand, if anyone needs a REST API, they can implement it on top of the safe_app_nodejs API.

Thatā€™s correct, but the JAVA library is also showing some good progress and soon will be also available.

This is already possible if the app already has the auth URI. After the app requested the Authenticator (which is embedded in the browser) to be authorised, it receives an auth URI which is then used to connect directly to the network. If the auth URI is securely stored by the app, it can keep connecting to the network without the need of the Authenticator, unless the app is revoked by the user, in which case it will it will need to request a new auth URI to the Auhtenticator.

The authorisation & connection flow would be the same as that of a NodeJS/WebApp/JAVA app. Now, a native app can interface with the safe_client_libs using the safe_app FFI interface, or using the lower level interface (safe_core) if itā€™s a Rust app.

1 Like

@Peter_Robertson, this is a draft of a diagram that will be part of the guide, and it should hopefully help to clarify the flow:

13 Likes

Indeed it does Gabriel, thank you very much.

Cheers

Peter

3 Likes

When I try executing SAFE_Tutorial_Example_3() using the 0.2.1 SAFE browser (mock routing), on Windows 10, the console just reads:

SAFE Example #3: Iterating over MutableData entries
undefined

It doesnā€™t display any of the keys or values. I had a similar problem with example #2 as well.

Hi @am2on, can you please confirm how you are running the examples, i.e. are you uploading it with the web hosting app first and then loading it on the browser? If so, can you please share the code exactly how you are uploading it, i.e. the whole HTML page?

I put the code within index.html, in between <script> tags and added a function call at the bottom so the function would run automatically. I uploaded each example to a separate service via the hosting manager. So, example 3ā€™s index.html file looked like this:

<h1>Example 3</h1>
<script>
/*
This example code demonstrates how a web app can connect to the authenticator
to request for authorisation and receive the auth URI, use the auth URI to then
connect to the SAFE Network, after connected, it creates a MutableData, and
finally it iterates over the entries, key and values of the created MutableData.
*/
function SAFE_Tutorial_Example_3() {
  console.log('SAFE Example #3: Iterating over MutableData entries');

  let appInfo = {
    name: 'Example safe-app web page #3',
    id: 'net.maidsafe.tutorial.webapp.example3',
    version: '0.1.0',
    vendor: 'MaidSafe Ltd.',
  }

  let access = {
    '_public': ['Read', 'Insert', 'Delete', 'ManagePermissions']
  };

  const typeTag = 15000;
  const MUTABLE_DATA_ENTRIES = { key1: 'value1', key2: 'value2' };

  window.safeApp.initialise(appInfo)
    .then((appToken) => window.safeApp.authorise(appToken, access)
      .then((authURI) => window.safeApp.connectAuthorised(appToken, authURI))
      .then(_ => window.safeMutableData.newRandomPublic(appToken, typeTag))
      .then((mdata) => window.safeMutableData.quickSetup(mdata, MUTABLE_DATA_ENTRIES)
        .then(_ => window.safeMutableData.getKeys(mdata)
          .then((keys) => window.safeMutableDataKeys.forEach(keys,
            (key) => console.log("Key:", key.toString()))
          .then(() => {
            console.log("Keys iteration finished");

            // Make sure that MutableData Keys instance is freed from memory.
            window.safeMutableDataKeys.free(keys);
            console.log("MutableData Keys instance freed.");
          }))
        )
        .then(_ => window.safeMutableData.getValues(mdata)
          .then((values) => window.safeMutableDataValues.forEach(values,
            (value) => console.log("Value:", value.buf.toString()))
          .then(() => {
            console.log("Values iteration finished")

            // Make sure that MutableData Values instance is freed from memory.
            window.safeMutableDataValues.free(values);
            console.log("MutableData Values instance freed.");
          }))
        )
        .then(_ => window.safeMutableData.getEntries(mdata)
          .then((entries) => window.safeMutableDataEntries.forEach(entries,
            (key, value) => console.log("Entry: (", key.toString(), ",", value.buf.toString(), ")"))
          .then(() => {
            console.log("Entries iteration finished");

            // Make sure that MutableData Entries instance is freed from memory.
            window.safeMutableDataEntries.free(entries);
            console.log("MutableData Entries instance freed.");
          }))
        )
        .then(_ => {
          // Make sure that SAFEApp instance is freed from memory.
          window.safeApp.free(appToken);
          console.log("SAFEApp instance freed.");

          // Make sure that MutableData instance is freed from memory.
          window.safeMutableData.free(mdata);
          console.log("MutableData instance freed.");
        })
      )
    )
    .catch((err) => {
       console.error("Error from webapp: ", err);
    });
};

SAFE_Tutorial_Example_3();
</script>

However, I tried it again just now after seeing the response, and now it works! I made no changes to the code or environment, so Iā€™m not sure what went wrong yesterday.

4 Likes

Thanks for the details, I just asked because people tend to open it from a file:// location which is not possible at the moment, the DOM API is only made available when loading it from a safe:// location.

Great! if it happens to you again please donā€™t hesitate to report it.

1 Like

@bochaco Gabriel, I though inline scripting (and styles) was blocked by the browser Content Security Policy. Is that no longer needed?

Great work on these example, they help a lot get started.

Hereā€™s my test site with all 3 examples if anyone needs them: safe://test.davidmtl/index.html

1 Like

Hi @happybeing, isnā€™t CSP applicable when the script is loaded from a different domain? I havenā€™t tried these type of things myself yet.

I think the problem is that if you allow inline scripting (or inline CSS) a script canā€™t be prevented from loading from another domain, so in order to implement a same origin policy, CSP rules normally block inline scripts & CSS. I suspect we donā€™t have any CSP rules being imposed by the browser at present, but have not checked that.

When testing websites with the early SAFE Beaker Browser inline scripts & CSS were disabled for same origin security, but I think this was imposed by the web proxy which has since been removed, so that may be why it is no longer active.

What Iā€™m not sure about is whether it is still needed or not. So thatā€™s really my question - have we left a security hole open, or is this no longer a risk?

I suspect it is a risk, but that it is (like the current Web) up to the website to provide appropriate CSP headers. However, my understanding is that the main browsers are taking on that responsibility by gradually imposing ever stricter CSP rules because so few websites are set up adequately in this regard. In which case we should probably have SAFE Browser do the same.

So my hunch is that we will need to close this loophole or we risk leaving users open to using websites and apps that are at risk from attacks such as XSS (eg inserting HTML into comments, forum posts etc).

4 Likes

Hi @bochaco, I started using an Ubuntu 17.04 VM for dev purposes, and got the same problem for the mock browser for Example 3.

The JS Console still gives no errors, nothing at all except for the first log message at the beginning, but this time i ran the browser executable from a terminal, and I saw this in the terminal after I authorized the example 3 app:

Mutation not authorised
thread 'App Event Loop' panicked at 'assertion failed: vault.authorise_mutation(dst, self.full_id.public_id().signing_public_key())', /home/travis/build/krishnaIndia/safe_core/safe_core/src/client/mock/routing.rs:854
note: Run with `RUST_BACKTRACE=1` for a backtrace.
thread 'App Event Loop' panicked at '

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!   unwrap! called on Result::Err                                              !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
/home/travis/build/krishnaIndia/safe_core/safe_core/src/client/mock/routing.rs:848,20 in safe_core::client::mock::routing

Err(
    "PoisonError { inner: .. }"
)

', /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/unwrap-1.1.0/src/lib.rs:67

I also discovered this error only occurs the first time the app asks for authorization. If I restart the browser and open the page without revoking authorization, the MD example starts working as intended, and there is no error in the terminal!

But if I revoke the appā€™s permissions before restarting the browser, and accept the permissions again, the problem happens again.

On my Win10 machine, where I originally encountered the problem, it works all the time now ā€“ even after revoking and reauthorizing. I was trying to run the mock browser & example from CMD, to see if the underlying error message was similar, but I canā€™t reproduce on this system anymore. Iā€™m not sure why.

Also, when I tested it (on the Ubuntu VM and the Windows host), Iā€™m running the JS from a separate file, so itā€™s not inline js anymore. I used exactly the same JS & index.html for both systems.

Hope this info is useful! :slight_smile:

1 Like

Hi @am2on, let me ask this first, are you building the browser yourself, and if so, note that we merged everything to the master branch last week, so are you building the master branch?

Also, if you are following the tutorial from this post, note that itā€™s a bit outdated now.

I was using the prebuilt browser binary on the release page (v0.2.1). I can try building the browser myself with the latest changes, and see if that improves things. Are there any up-to-date guides for building it with mock-routing? If not, what should I do differently from that post to get it to work? Thanks!

I am interested in binaries that would allow me to develop/test web apps on a non-connected laptop. My style of coding is very iterative and loading new files into the actual network every time I do a small edit is extremely painful.

I understand there is a mock-vault binary. Is this all that is needed, or is there a special mock version of the breaker browser needed as well? If I use the mock-vault, can I just copy the app files to a directory, or do I still need to ā€œuploadā€ them into the mock-vault (which would still present something of a bottleneck to iterative development)?

Ideally, I want to edit my javascript, click something (double clicking on a batch file would be OK), wait only a second or two, and be able to test the edits.

3 Likes