Not able to add or get a file greater than 256 KB

[Sameeksha Jain]

Describe the bug

let { Peer, BlockStore } = require('../js-dag-service/dist/index')
let { setupLibP2PHost } = require('../js-dag-service/dist/setup/index')
let { MemoryDatastore } = require('interface-datastore')
const fs = require("fs");
const store = new BlockStore(new MemoryDatastore())
const main = async () => {
    // Bring your own libp2p host....
    const host = await setupLibP2PHost()
    const lite = new Peer(store, host)
    await lite.start()
    let content = fs.createReadStream('./a.txt');
    var source = [
        {
            path: "bar",
            content: content,
        },
    ];
    var data = await lite.addFile(source, { format: "raw", hashAlg: "sha2-256" })
    const cid = data.cid
    const data2 = await lite.getFile(cid)
    fs.writeFile("b.txt", data2, "text", function (err) {
        if (err) {
            console.log(err);
        } else {
            console.log("The file was saved!");
        }
    });
    await lite.stop()
}
main()

file size
-rw-rw-r-- 1 8725848 Aug 26 17:17 a.txt
-rw-rw-r-- 1 1710 Aug 26 17:25 b.txt

To Reproduce
Steps to reproduce the behavior:

1. git clone js-dag-service and create a dist folder
2. run the above code like 'node index.js'

Expected behavior
A file b.txt should have the content of file a.txt

Desktop :

• OS: ubuntu 18.04.4

Nodejs
v12.18.3

Thank you for any help

[Irakli Gozalishvili]

@jainsameeksha729 sorry for not getting back to you sooner. At the moment lot of changes are happening on integrating some newer IPLD bits which needs to be resolved before I can investigate this more.

That said I think your problem was that you were specifying "raw" encoding which would attempt to create a single block (exceeding block size limit) out of the whole file instead of chucking it up as IPFS does with regular files. I believe that omitting that would have made this problem go away.

Once new IPLD pieces are integrated I'll get back to you on this to either validate my comment or investigate this issue further.