Java API support on desktop platforms

lionel.faber · January 17, 2019, 3:08pm

Recently, we released support for native Android development for SAFE using a new Java API and we’re excited to see applications from the community!

Java, being platform independent, will run on any platform and since the recent release targeted only Android an interesting question arises, “What about desktop?”

The Java API are indeed supported on desktop too. But however, we are looking into certain aspects of packaging (following maven standards) and some IPC requirements before official release.

Nonetheless, by following the instructions below you can get started with the Java API on desktop. Note that these are not yet finalized and are likely to change.

Building the Java libraries:

The java libs for desktop can be built from the safe_app_java repository. The desktop libraries can be built via the following commands:

On Windows, use gradlew.bat instead of gradlew

git clone https://github.com/maidsafe/safe_app_java
chmod +x gradlew
gradlew safe-app:download-nativelibs

This command will download all the native libraries that are required. You can now build the java libraries using the following command

gradlew :safe-app:build

This command will run the tests and build the java libraries and output 2 JAR files in the safe-app/build/libs folder:

safe-app-0.1.0-mock
safe-app-0.1.0-nonMock
safe-app-0.1.0 // Please ignore this one

These JAR files will contain the required classes and native libraries for the two variants (mock / non-mock). You can now include these libraries into your project and use the SAFE APIs
For more information about the APIs check out this Android tutorial and the API documentation.

Note: Both the JARs contain the same class files so adding both the libraries will throw an error.

Authentication with the SAFE browser

The application can authenticate to the SAFE network through the SAFE browser. You can create an auth request by following the authentication section of the tutorial. The only difference on desktop would be that the URL format to send the request to the authenticator would be:

safe-auth://<encoded_auth_request>

instead of

safe-auth://<app_id>/<encoded_auth_request>

On opening this URI the auth request will be sent to the authenticator in the SAFE browser.

After authentication, the browser will send the auth response back to the application through a custom URI registered for the application in the following format:

safe-<base_64_encoded_app_id>://<encoded_auth_response>

The application will need to register itself to open URLs of this type and use the auth response to connect to the network. This URI registration can be done across platforms using the system_uri crate.

With some experimentation I was able to create Java bindings for this crate using jnr-ffi as a proof of concept. The code is available in this repository. A lot more is to be done there in terms of error handling across the FFI but it’s a start You can download the built jars (with native libs) from the releases.

Example CLI application

To demonstrate the authentication process with the SAFE browser I have created a simple Java CLI application which uses the SAFE browser to authenticate and connect to the network. The code is available is this GitHub repository.

To run the application on the mock network:

Download the latest mock browser from the GitHub releases
Clone the repository
Create a libs folder in the repository’s root and copy the built safe-app-0.1.0-mock.jar file from the safe_app_java/safe-app/build/libs folder. You will also need the system_uri-0.0.1.jar file from this release.
Now run the following command:

gradle jar

This will generate a java-0.0.1.jar file in the build/libs folder. This executable JAR is packaged along with the safe-app and system-uri packages. Now open the downloaded mock browser, create an account and login. On executing the CLI application via:

java -jar <path/to/executable.jar>

an auth request will be sent to the SAFE browser and on authentication the response will be sent back to the CLI application.
Now that you are connected to the network you can perform network operations of your choice by referring the rest of the tutorial or the API docs

Good luck!

OS X Limitation:

Unfortunately, we cannot map an executable JAR to a custom URI on OS X. It needs to be packaged as an application with an Info.plist file. So this CLI is limited to Windows and Linux(for now). More details about this issue on this thread.

Traktion · January 18, 2019, 9:12am

Thanks, @lionel.faber - really useful stuff there. I wish I had had this on Tuesday when I was working on my app!

Are there plans to allow authentication directly (using safe_client_libs, I assume?) instead of via the browser? Could tokens be stored for repeat authorisation without having to re-authenticate each time? It would be good to see what has been authorised (outside of the browser) though.

For CLI apps, this would be essential in many cases, as using a browser may not be possible or desirable (IoT, services, etc).

lionel.faber · January 18, 2019, 12:01pm

For development you can authenticate in a number of ways:

Using the mock authenticator APIs - like in the example application
Using the createTestAppWithAccess() API - Link to docs This will take an auth-request and return a Session object

Yes. You can store the Auth response from the browser and use it to connect the next time. However, if the app is revoked from the Authenticator then the string is of no use.

For now authentication to Alpha-2 is limited to the Authenticator(browser and mobile application) for tier-1 platforms. Technically, we can authenticate and fetch the authenticated apps using the API exposed by the safe-authenticator crate but the libraries currently expose them only for mock routing.

davidpbrown · January 29, 2019, 11:10am

Mission: Have flu; kill bugs.

Below, then works on Linux; a few typos fixed and after a struggle with the unfamiliar becomes a bit forced and saw the sdk command line tools installed, where perhaps there is still a hack to avoid needing that:

Check java version > 7
$ java -version
java version “1.8.0_191”

sudo apt-get install gradle

hiccup tempts then faking the license =>

However, I still got a run of “Warning: License for package Android SDK Platform ## not accepted.”
and a suggested solution like “sudo sdkmanager --licenses”, wasn’t simple to action.

Download Android Studio Command line tools only from

unzip

cd ./tools/bin
./sdkmanager --licenses

answering yes to those.
is a route to accepting licenses but it had no effect.

So, I guessed the unzip wanted to be in ~/.android
moving the unzipped files and merge/copy over licenses to be within ~/.android

and trying the

$ ./gradlew safe-app:download-nativelibs

is successful.
as is then

$ ./gradlew :safe-app:pack

and then from https://github.com/maidsafe/safe_app_java - if you include the “./”

./gradlew :safe-app-android:download-nativelibs
./gradlew :safe-app-android:build
./gradlew :safe-app-android:javadoc

also work, though with a head full of glu, I have no idea what they do just yet

lionel.faber · January 29, 2019, 3:19pm

You can run the tasks using gradle or gradlew (gradlew.bat on windows).
gradlew is a gradle wrapper that will help you run gradle tasks without installing gradle. StackOverflow has a good comparison.

This was because the ANDROID_HOME environment variable was set to ~/.android. So moving the CLI tools to that directory did the trick

Indeed there is. The safe_app_java project is a multi module gradle project for both desktop (safe-app) and android (safe-app-android)
If you want to build the libraries only for desktop without downloading the Android SDKs you can remove :safe-app-android from the settings.gradle file and then build the desktop JARs.

For some clarity on what the gradle tasks do:

The safe_app_java project uses native code that is written in Rust. This native code is compiled for the supported platforms in their respective formats. The download-nativelibs task downloads these native libraries into the project directories.

The native libraries and the required Java classes can be packaged into JARs and AARs for desktop and mobile respectively. The safe-app:pack task will build the required JARs for desktop and the safe-app-android:build task will build the AARs for mobile.

The API documentation for the Java API is available at docs.maidsafe.net/safe_app_java. However if you prefer to build it locally the safe-app-android:javadoc task will generate the documentation into the safe-app-android/build/docs/javadoc directory.

Traktion · February 14, 2019, 6:06pm

Ok, my son was sick again, so I got another day to play with this! ha!

So, I’m using createTestAppWithAccess() now and it seems to be getting further. However, there are a few things that aren’t as I’m expecting.

I’m attempting to use the NFS library to create a file and then read back from it. Using either mock or non-mock native libraries, this doesn’t seem to work for me - it just throws an exception when I try to read the data.

I have setup the app using the following basic settings:

AppExchangeInfo appExchangeInfo = new AppExchangeInfo("id", "scope", "name", "vendor");

ContainerPermissions[] containerPermissionsArray = new ContainerPermissions[1];
ContainerPermissions containerPermissions = new ContainerPermissions("_public", ALL_PERMISSIONS);
containerPermissionsArray[0] = containerPermissions;

AuthReq authReq = new AuthReq(appExchangeInfo, true, containerPermissionsArray, 1, 1);

Session session = Session.createTestAppWithAccess(authReq).get();
session.setOnDisconnectListener(o -> LOGGER.debug("Disconnected from network"));

Then some simplified code to read/write:

MDataInfo containerMDI = session.getContainerMDataInfo("apps/id").get();
File newFile = new File();
session.nfs.insertFile(containerMDI, path.getFileName().toString(), newFile).get();

NativeHandle writeHandle = session.nfs.fileOpen(containerMDI, newFile, NFS.OpenMode.OVER_WRITE).get();
session.nfs.fileWrite(writeHandle, byteBuffer.array()).get();
session.nfs.fileClose(writeHandle).get();

NativeHandle readHandle = session.nfs.fileOpen(containerMDI, newFile, NFS.OpenMode.READ).get();
session.nfs.fileRead(readHandle, 0, bufferLength).get();
session.nfs.fileClose(readHandle).get();

The exception message I get is:

java.util.concurrent.ExecutionException: java.lang.Exception: Core error: Routing client error -> Requested data not found : -103

From debugging, it looks like newFile is never populated with anything, when I was assuming it would be. Moreover, I’m wondering how I can open a file from just a file name (which I suspect may require MDataEntries manipulation of some sort).

Finally, I can’t see any temporary files being stored in my tmp directory, which I was expecting to see. I’m wondering if the two could be related?

I’ll keep playing with it and I’m pleased to have ported some of my old REST client code to the new native library. It would be great to get something writing/reading though!

vigneshwara · February 15, 2019, 7:19am

hi, @Traktion. Hope your son recovers soon.

When you are inserting the file, it is empty. And then when you are writing the file, the file doesn’t get updated. So you need to write the file, close it and use the returned file object to insert the file.
I’ll show you an example.

MDataInfo mDataInfo = session.getContainerMDataInfo("apps/id").get();
File file = new File();
NativeHandle fileHandle = session.nfs.fileOpen(mDataInfo, file, NFS.OpenMode.OVER_WRITE).get();
byte[] fileContent = Helper.randomAlphaNumeric(LENGTH).getBytes();
session.nfs.fileWrite(fileHandle, fileContent).get();
file = session.nfs.fileClose(fileHandle).get();
session.nfs.insertFile(mDataInfo, "sample.txt", file);

On doing this, we inserting a file that is already populated.

For reading the file, we open it and store the contents in byte array.

fileHandle = session.nfs.fileOpen(mDataInfo, file, NFS.OpenMode.READ).get();
byte[] readData = session.nfs.fileRead(fileHandle, 0, 0).get();

Thanks for asking and we hope to help you more.

Traktion · February 15, 2019, 11:50am

Thanks for the examples! It would be great to have them at the top of the NFS javadoc, as it isn’t obvious how to sequence these steps.

Maybe my brain just read it backwards, but it felt like I needed to insert the file before I could write to it!

If I don’t have the object for the inserted file, is there an easy way to retrieve it based on file name? Do I use nfs.updateFile or some such?

lionel.faber · February 15, 2019, 1:10pm

This should do what you’re looking for

NFSFileMetadata metadata = session.nfs.getFileMetadata(containerMDI, "filename").get();
NativeHandle readHandle = session.nfs.fileOpen(containerMDI, metadata, NFS.OpenMode.READ).get();
byte[] content = session.nfs.fileRead(readHandle, 0, session.nfs.getSize(readHandle).get()).get();
System.out.println(new String(content));
session.nfs.fileClose(readHandle).get();

This is something we have in mind We limited the javadoc to the API definitions for initial release. We’ll definitely add examples in the versions to come! And of course, pull requests welcome

Traktion · February 15, 2019, 2:22pm

Thanks! I should have some good examples in github to link to soon too, with any luck!

Traktion · February 15, 2019, 5:30pm

Ok, tried that out and got read/write working well!

The next challenge is directories… any pointers as to how to create/delete directories?

happybeing · February 15, 2019, 5:38pm

Directories don’t exist as entities - they are only ‘implied’ and there is no API in SAFE that understands a directory. So it is just an application convention that ‘/’ characters in the key under which file is stored are interpreted as directory separators.

Hope that makes sense, it feels hard to put into words!

Traktion · February 15, 2019, 5:42pm

Ah, right - so if I can have a key which is “/directory1/directory2/file.txt”?

EDIT: Or is it more like having 3 entries like:

/directory1/
/directory1/directory2/
/directory1/directory2/file.txt

The old REST API did have a concept of directories, so I’m guessing that was just an extra layer of abstraction. Interesting!

happybeing · February 15, 2019, 6:20pm

The first form is correct. There is no way (convention) at the moment to have an empty directory.

lionel.faber · February 15, 2019, 6:35pm

Yes. Having the entire path as a key would be one way to go about it.

Another way could be to have the folder name as the key and the value could be serialized MDataInfo of a different mutable data. This other MData can hold all the files inside that directory
So for instance,

In the app’s own container(App’s root folder):

Key	Value
file1	dataMap
file2	dataMap
folder1	serializedMdInfo(name: folder1, typetag: 15234)
file3	dataMap

And in mutable data with name: folder1 and typetag: 15234

Key	Value
file4	dataMap
file5	dataMap

You can even have an empty folder, which will be a mutable data without any entries. You can add files(entries) later, whenever required

P.S.: This just an approach, there can be multiple other possibilities too

Traktion · February 15, 2019, 6:41pm

Is there a convention for an approach? It sounds like one is an object store (entire path as key) and the other is a more traditional file/container hierarchy file system.

lionel.faber · February 15, 2019, 6:46pm

I don’t believe there is a convention as such. Both approaches have their own pros and cons. However, I believe that the WHM and the browser go with the latter to upload/fetch website files.

happybeing · February 15, 2019, 7:08pm

This is news to me! I’ve even discussed on here (and I think raised an issue) about the inability to have an empty folder in SAFE NFS, and this wasn’t mentioned so I don’t think it is well understood, and I’ve not seen it documented anywhere.

If like to have this specified so we can build on the convention. It has implications for everyone, so it would be very helpful to have SAFE NFS specified down to this level.

It will make it easier to develop, and help achieve cross app compatibility if there is a clear definition of what is expected of apps that want to inter operate.

happybeing · February 23, 2019, 1:03pm

So the points I raised above don’t get forgotten I’ve opened an issue requesting clarifying documentation wrt to this and other aspects of SAFE NFS:

https://github.com/maidsafe/safe_client_libs/issues/744

lionel.faber · February 25, 2019, 9:13am

Sorry this took a while.

Over the weekend I played around with the NodeJS API to implement this approach for empty folders in NFS.

Here’s some brief context about the NFS implementation:

An NFS container is a mutable data that stores a list of entries where the key is the file name and the value is the location(dataMap) of a file which is stored as immutable data.

Since this container is a mutable data, we can perform general MData operations on it too. So, for an empty folder, we create a new mutable data instance, serialize it and store it as an entry with the folder name as key and the serialized instance as the value.

We can code the application to handle files and folders differently
Here’s a snippet using the NodeJS API:

// Initialize mutable data
  const typeTag = 15000;
  md = await safeApp.mutableData.newRandomPublic(typeTag);  
  await md.quickSetup({});

  // Emulate root folder as NFS and insert a file
  const nfs = await md.emulateAs('NFS');
  const content = '<html><body><h1>WebSite</h1></body></html>';
  const fileName = 'page1.html';
  const fileContext = await nfs.create(content);
  await nfs.insert(fileName, fileContext);

  // Initialize new mutable data. This will be an empty sub-folder
  let newMd = await safeApp.mutableData.newRandomPublic(typeTag);
  await newMd.quickSetup({});
  
  // Serialize the sub-folder info and store it in the root folder
  const folderName = "page2";
  const serialisedMD = await newMd.serialise();
  const mutation = await safeApp.mutableData.newMutation();
  await mutation.insert(folderName, serialisedMD);
  await md.applyEntriesMutation(mutation);
  
  // Fetch the sub-folder info and deserialize it to get the md object
  const entries = await md.getEntries();
  const value = await entries.get('page2');
  const emptyFolderMd = await safeApp.mutableData.fromSerial(value.buf);

  // Emulate the folder as NFS and add a file
  const cssFolder = await emptyFolderMd.emulateAs('NFS');
  const newFile = await nfs.create(content);
  await cssFolder.insert('page2.html', newFile);
  
  // Print the XOR URLs to view them in the MData viewer
  const rootInfo = await md.getNameAndTag();
  console.log("Root folder: " + rootInfo.xorUrl);
  const cssFolderInfo = await newMd.getNameAndTag();
  console.log("CSS folder info: " + cssFolderInfo.xorUrl);

I hope this helps!