SAFE Drive - help with testing

[isntism]

Ah right, the development branch, that was not mentioned in the github page, but as for using 400 puts, I may have mistyped it since it only used 4, though the extra folders did show up, ah well, its working.

Now to try and find ways to break it, shouldn’t be too hard at this stage.

edit:
moved a txt file over, took a bit of time but it showed up, opened it without trouble and closed it again, since the file is slightly over 1KB it used two puts, I was under the impression it was a MB, but that may be wrong, excellent work so far though!

[draw]

I’ll try to test it this evening. Which version of npm to use, or do you think that doesn’t matter?

[happybeing]

@draw

I’ll try to test it this evening. Which version of npm to use, or do you think that doesn’t matter?

Good question. I’m using npm v6.4.1 and node v8.11.3 but I don’t know what is or isn’t good. I’ve been tracking what Maidsafe are using but may be out of step.

[happybeing]

Great that you got going so quickly @isntism. You were too fast for me to mention the development branch sorry But yes, please use developmentand NOT master which is way, way behind.

@isntism

moved a txt file over, took a bit of time but it showed up, opened it without trouble and closed it again, since the file is slightly over 1KB it used two puts, I was under the impression it was a MB, but that may be wrong, excellent work so far though!

That sounds about right because a small file will require one PUT to update the directory, and another to put the data (which will all fit in the data map). @marcin wrote a good post which summarises what causes a PUT or a GET which helped me optimise the code. Here’s what he says about small files, but the post contains a lot more detail which is very useful for devs:

@marcin

The actual number of GETs and mutations depends on the size of the data. Any idata at least 3kb in size will be self-encrypted to at least 3 chunks. Smaller than that, and the entire idata is held in the datamap. Each chunk is a separate GET/mutation. The maximum size of a single chunk is 3mb. So keep in mind that getting a whole idata will almost always be more expensive than e.g. getting the size

Note that if you copy a file that is already on SAFE it should take only one PUT not matter how big it is because I just insert a pointer to its data map in the directory.

[isntism]

Ok, it seems that a lot of programs create a temporary file then delete the previous one and rename it, which does not seem to be working right now since I get “Error renaming temporary file: Operation not supported” when replacing it from my filebrowser, now you probably already know that and how to fix it, but I felt the need to say so just in case.

Is it possible to see other people’s public files yet, or even that git project you tested on, or is that still a wip?

[happybeing]

@isntism

Ok, it seems that a lot of programs create a temporary file then delete the previous one and rename it, which does not seem to be working right now since I get “Error renaming temporary file: Operation not supported” when replacing it from my filebrowser, now you probably already know that and how to fix it, but I felt the need to say so just in case.

Can you say which apps you tried, what you did, what happened?

I haven’t re-tested this but previously found that some worked and some didn’t. File renaming should work though, so you may be onto something useful there. More info please

Is it possible to see other people’s public files yet, or even that git project you tested on, or is that still a wip?

Everything I described works, er might work , so long as you are on developement branch. I haven’t tried publishing a git repo live myself, but John ran the test script on the live network and that tries everything that is needed AKAIK, so it should work.

[isntism]

That perticular error was thrown by ubuntu’s own fileroller as I copied a txt file and told it to overwrite the same-named file, I also tried saving an edited version using the default app for said txt file, which did cost 4 puts but didn’t actually change the file, so I ended up assuming that the same thing would happen for my other programs. (bionic beaver btw, in case that may end up being usefull)

Should I quickly fire ubuntu back up and try and run a debug version before going to bed, or can it wait till tomorrow?

[happybeing]

No hurry, but as much detail as you can will help me find these apps and try them myself. I’m not sure what you mean by “ubuntu’s own fileroller” (if you mean the file manager, I think that can vary with the Ubun version and desktop) and don’t know what “bionic beaver” does for example. I could search but it may save time if you can give a bit more detail.

So when you get time if you can describe the steps that are needed to cause a problem, and what that was it can save me time trying to replicate it that would be great. But it’s fine if not, every bit of feedback helps point me in the right direction.

Also the version of Ubuntu and which desktop you have might help here if you know them.

I assume you are doing everything on the live network, which is great because most of my testing is done on mock.

Thanks a million for stepping in and helping out

EDIT : I just saw “bionic beaver” flying past on twitter so that mystery has been resolved

[draw]

I’m at the point of ls ~/SAFE

I’m testing on Linux Mint 18.3. I had initially some compile and run errors. That seemed to be solved after I removed the old ~/.npm, yarn and ~/.cache dirs and/or the clean installation of npm v6.4.1 and node v8.11.3, like you described…

Remarks until now:

• it seems npm v6.4.1 + node v8.11.3 is not a default combination. If you install node(+npm): then it is 8.11.1 (which comes with npm v5.6.0) or the most recent stable: node v10.13.0 and npm v6.4.1. Maybe this doesn’t matter that much. I’ll try to test tomorrow if the default node/npm version combination also works. But it could be the cause of my initial problems to get it working.
• Do you also get a lot of warnings at the ‘npm install’ command in safenetwork.js like this one:?
  ./src/binding.cc:643:8: warning: ‘Nan::Maybe<bool> Nan::ForceSet(v8::Local<v8::Object>, v8::Local<v8::Value>, v8::Local<v8::Value>, v8::PropertyAttribute)’ is deprecated [-Wdeprecated-declarations]

• The ‘npm link’ for safenetwork.js needs to be executed as root, at least on my system. This is necessary to have create perm. for symlink /usr/local/lib/node_modules/safenetworkjs. → Maybe an idea to say ‘sudo npm link’ in the github readme.md?
• It seems to work, but I find it strange, that according to github of safenetwork-fuse, you have to do
  NODE_ENV=dev npm install
  But then connecting not to the mock, but to the alpha 2 network, it seems

@happybeing

• start Peruse browser
• login to SAFE Network

→ Is the ‘NODE_ENV=dev’ necessary in this case?
I also get a lot of ‘deprecated’ warnings during this npm install command.

• When I execute the find cmd in the ~/SAFE dir, the find (and fuse) hangs. I think on the mutable data entry, created by safe://to-do.

[happybeing]

@draw Re:
npm versions: that’s useful to know. Thanks for reporting back. I don’t recall why I have the versions I do, but I guess I updated one at some point and am stuck until I update again.

npm install warnings: I don’t recall exactly, but yes I probably get those too. I just haven’t done npm install for a while. I tend to ignore npm warnings (but not errors) until something doesn’t work and then I go back and look again.

npm link: I would say needing sudo is not great, not terrible either, but a system configuration issue. I don’t need that on Ubuntu so suspect Mint isn’t setting permissions up in line with expectations.

NODE_ENV=dev npm install: is a Maidsafe feature which will be (may be already) deprecated. All it does is make sure the debug SAFE libraries are installed. Without it you may only get the production libraries and would not be able to use the mock network. I’m still using the old 0.8 APIs so it is probably still required for now.

Is the ‘NODE_ENV=dev’ necessary in this case? When running SAFE Drive NODE_ENV=dev selects mock network (dev SAFE APIs), so you use it when you want mock rather than live network.

When I execute the find cmd in the ~/SAFE dir, the find (and fuse) hangs.

Bingo! I hadn’t thought to try find. Nice catch. Looks like something to add to the test script. I can do this, or if you fancy working with mock and updating the script to fall over on this, be my guest. Just let me know so we don’t both do it! If you would like to add an issue for this that would be good too.

Thank you for diving in Ward, I really appreciate it.

[draw]

About the sudo: that was more a remark to add the sudo to the github readme.md instructions, to make it clearer.
I’m not immediately planning to look too deep into the code/change the test script. But should I do that, I’ll let you know. If I have more time and desire, getting Python api working seems more interesting (technically) to me. Not a big fan of Javascript.
Does your solution use ‘caching’ atm? I do mean first saving it locally, and then syncing with the Safe Network at the background?
Because if I look at the Dropbox solution on Linux: that Dropbox folder is not a mount. What seems logical to me: you first write the changes to local disk and then sync with ‘the cloud’ afterwards.

[happybeing]

@draw

I’m not immediately planning to look too deep into the code/change the test script. But should I do that, I’ll let you know. If I have more time and desire, getting Python api working seems more interesting (technically) to me. Not a big fan of Javascript.

The script I’m referring to is a simple bash script so it is relatively easy to add something that will fall over for a particular condition once you’ve got a simple way to reproduce the problem. So in this case you might mount the ‘to-do’ files by accessing ~/SAFE/_webMounts/to-do and then run a find command that currently hangs. No worries if not, but I like to suggest ways people can do a bit to help just in case. If you prefer to give me the command that failed I’ll be happy to add it.

@draw

Does your solution use ‘caching’ atm? I do mean first saving it locally, and then syncing with the Safe Network at the background?

No, the caching is currently just on reading file attributes which FUSE does incessantly! I’ve no plans to cache writes, or file content at all but…

If you want to save locally and sync to SAFE that is possible outside of SAFE Drive with a simple script using rsync which can be automated with cron or run manually like a backup. At some point I guess we’ll put together tips and tricks or provide some scripts to do stuff like that. It is trivial to do.

[draw]

Ok, I’ll have a look at that bash script when I have the time.
Ps: the rsync option ‘–stats’ maybe useful, if writing/reading to SAFE Drive introduces extra delay…

[happybeing]

Thanks Ward, that would be great (and for the rsync tip).

When you look at the script, search for _webMounts and I think your test will fit in that area nicely.

[isntism]

Well that was annoying, my ubuntu glitched out and I had to poke it for two hours to get it to work, but finally I can acces the notes I wrote down yesterday.

First of all, bionic beaver is the version, easier to remember then 18.04 but it may have been smart to say the number instead, secondly, I indeed meant file manager, for some reason I had roller in my head.

I first used the old precompiled version, then compiled it in the stable branch and then in the development version, with each one I attempted to connect it to peruse, and all three seemed to work, though I may have mistyped SAFENETWORKJS_TESTS=testing seeing how it didn’t use anywhere near 400 puts.

Then I moved the .txt file I was writing notes into to SAFE/_public/tests/data1 and marveled as the console was filled with text at an unreadable rate and opened it into the text editor.

I edited said text file in the text editor and tried saving but got an error Cannot handle "file:" locations in write mode. Please check that you typed the location correctly and try again.

NOTE, I am redoing them to be sure this is what happened and suddenly the file seems to be empty, though the console also has not printed anything after socked disconnected, tried restarting the program and the file still is empty:scream:, still not getting large amounts of texts though, maybe I forgot a parameter

So I edited the original file and simple dragged it over and clicked the replace button you always get when transferring same-named files and get an error shortly after, with under more details finding Error renaming temporary file: Operation not supported

Hope that helps… somehow, and don’t forget to ask for specific things if you want them, I don’t know which logs and commands to use/send and I’ll try to get you whatever you need asap.

[happybeing]

This is great @isntism and definitely pointing me towards more useful tests so I can experiment to see if I can replicate this.

I’m not sure exactly which code you have been running or how, as what you report doesn’t match what I would expect.

What I need everyone to try atm is only the development branch, and running the script using the commands I mentioned above. If you are doing this, and the test script prints “ALL TESTS PASSED” you will definitely use 350 or so PUTs, so if you aren’t seeing that you must be doing something different. (Note: you need to click the refresh after each test to see PUTs used updated).

Do you know which editor you were using? I tried gedit and got a result a bit like you describe, but not the exact error message.

All good though, thank you. Oh, and there is no need to ‘compile’ anything as the command I suggested is running the script (bin.js) not the compiled build which will be in ./dist/prod or ./dist/mock

Keep me posted if you try anything or can provide more details. Fingers crossed your Ubuntu won’t flake out again!

[isntism]

This time I tried ./bin.js in the safenetwork-fuse folder and it succesfully connected to the network, but I don’t actually see it doing the test so I scrolled up and found SAFENETWORKJS_TESTS=testing DEBUG=safenetworkjs*,safe-fuse* node bin.js, I put that in as a single line and it started working, I saved everything it said and refreshed my account status on peruse, it only increased with a handfull again.

I’m assuming that you want it so I snipped out what I think is my account, but I’m not sure what’d be the best way of sending it.

Oh and yeah, according to the ubuntu software app it was indeed gedit.

[happybeing]

From what I understand you aren’t running the tests. What you are doing there is starting SAFE Drive. That doesn’t use any PUTS so the small decrease you see may be due to logging into your account.

To run the tests, you change directory to safenetwork-fuse/tests and run the test script with either ‘live’ or ‘mock’ as the parameter, depending on whether you are using the live or mock network.

But you may as well save your PUTs and do your own tests rather than repeating those, so we can find problems that are not picked up by those tests. So that means just ‘doing various stuff’ and reporting back whether things seem to work, and with details of anything that doesn’t. Essentially, try to break it by trying all sorts of things!

How that makes sense.

[isntism]

It does, but I wanted to get you something tangible instead of, what I feel like, a vague description of a problem somewhere in my system.

But ok, instead of running the test I’ll just try a bunch of things, write down what does and doesn’t work and report here in… a few days, assuming I don’t forget, though I’m fairly sure that maidsafe will grant more puts if we end up spending them on testing, this is an important project after all.

Thanks for the patience, and good luck finding the big bugs.

[happybeing]

@isntism

Thanks for the patience, and good luck finding the big bugs.

Thanks, and and no worries, just do what you feel like I say. That’s why I’m doing this after all

And yes, Maidsafe will give you more PUTS, but for live testing that gets tedious if you can only run the test script a couple of times. So live testing with the script can be saved until we’ve done all the grunt work with small tests. Where we find problems, I can add them to the script where feasible or we keep a note of how to reproduce them - creating an issue is ideal for this.

I can work on fixing them using mock, and then check it still works in the live network.

So you can test with live or mock. For most people it is easier to use the live network, and it is nice for me to see confirmation that people can use it, and find it works with the live network.

But really it is up to you.