Trying to understand store_blob

I’m reading store_blob because @joshuef mentioned the PUT command (I assume this is what the command does). And I can’t quite understand it, in particular, it seems to only store the data map to the network, but not the data itself? I haven’t looked into self-encryption yet, but I assume data map only stores info about where to find each piece of the data, but not the data itself?

Here’s the code for reference, in sn_client/src/client/blob_apis.rs:

pub async fn store_blob(&mut self, the_blob: Blob) -> Result<Blob, ClientError> {
    info!("Storing blob at given address: {:?}", the_blob.address());

    let is_pub = the_blob.is_pub();
    let data_map = self.generate_data_map(&the_blob).await?;

    let serialised_data_map = serialize(&data_map)?;
    let data_to_write_to_network =
        serialize(&DataTypeEncoding::Serialised(serialised_data_map))
            .map_err(ClientError::from)?;

    let blob_to_write = self
        .pack(self.public_key().await, data_to_write_to_network, is_pub)
        .await?;

    let cmd = DataCmd::Blob(BlobWrite::New(blob_to_write.clone()));

    self.pay_and_send_data_command(cmd).await?;

    Ok(blob_to_write)
}

Another minor question I have is, why so many serializations? E.g. first the data map is serialized, and then the serialized data map is wrapped and serialized again, and in pack() there is a third serialization. So it seems like each step requires serializing the previous data, and I’m a bit curious what makes this a necessity?

Again I thank everyone for their time reading my questions, I know the team is really busy now. But I guess reading the code is one of the better ways to understand the project, and I hope my inquiry is not too much burden on you guys.

4 Likes

Yes, that was a bug that is now resolved. The serialisations are overkill atm and we tend to do that for hashing etc. but it’s a clean up needing to be done if you fancy a shot? Nice detective work, the team had test fails and a couple of days bug-hunting to find we were not storing the chunks :wink:

1 Like

Thanks for the kind words. I’m rather ashamed that I didn’t bother updating my local repo before reading (good detective work should start with up-to-date info).

FTR, now that I checked, the bug was fixed back on Nov 28, and unfortunately I was reading Nov 26.

2 Likes

Does not take away the fact you found a bug very fast and it was a good one. Pretty impressive.

2 Likes

Honestly pure luck, as I mentioned Josh mentioned the PUT command so I looked into it lol. But I appreciate the encouragement, all the reason to keep reading and learning!

1 Like

I’m willing to help, but I think I would need guidance from you guys, because I’m totally new to the code and I’m not sure about the scope of this change and the exact goal of this task. But if you think my getting involved would make it more productive instead of counter-productive, then sure!

2 Likes

Definitely keep digging @treslumen, it’s all super helpful.

@lionel.faber maybe you can comment some one the current blob impl, as you’ve just reworked it?

1 Like