IPFS: How to add a file to an existing folder?

Question

Given a rather large folder, that has already been pushed to the network, and deleted locally. How would a file be added to that folder, without re-downloading the entire folder it?

Answer 1

You can only do it by using ipns after downloading it again with ipfs get, which should be fast if it's still pinned to your local storage:

(1) first add (i.e. re-add) your folder to ipfs recursively: ipfs add -r /path/to/folder. The second column of the last stdout line has the ipfs hash of the parent folder you just added. (The original files are still the same, so the hashes will be the same too.)

(2) then publish that hash: ipfs name publish /ipfs/<CURRENT_PARENTFOLDER_HASH>. This will return your peer ID, and you can share the link as /ipns/<PEER_ID>; repeat this step (ipfs name publish) whenever the folder contents (and therefore the parent folder hash) changes. The ipns object will then always point to the latest version of your folder.

(3) if you plan on sharing a lot, you can create a new keypair for each folder you share: ipfs key gen --type=rsa --size=2048 new-share-key … and then use that key (instead of your default key) to publish (and later republish) that folder: ipfs name publish --key=new-share-key /ipfs/<CURRENT_PARENTFOLDER_HASH>

See also the documentation here: https://docs.ipfs.io/reference/cli/#ipfs-name-publish

Answer 2

I'm a bit late to answer this, but I found the 2 existing answers a bit unclear.

Tl;Dr; Just commands and minimal info

If you want a thorough detailed explanation, scroll down to the section starting with The 2 keys to mutability.

If you just need the commands you should run, and barebones usage info so you know how to actually adjust the command for your use case, then read this TL;DR; section.

Use IPNS / DNSLink for references to IPFS objects that can be updated

IPNS

Create a key, back it up if using in production, then use ipfs name publish to change the object that your key currently points to. Access your key by prefixing /ipns/ to commands / URLs instead of /ipfs/.

ipfs key gen test
# backup your key if used in production
ipfs key export -o /home/somewhere/safe/test.key test
umount /ipns
ipfs name publish -k test QmWRsWoZjiandZUXLyczXSoWi84hXNHvBQ49BiQx9hPdjs
#    Published to k51qzi5uqu5dkqxbxeulacqmz5ekmopr3nsh9zmgve1dji0dccdy86uqyhq1m0: /ipfs/QmWRsWoZjiandZUXLyczXSoWi84hXNHvBQ49BiQx9hPdjs

ipfs ls /ipns/k51qzi5uqu5dkqxbxeulacqmz5ekmopr3nsh9zmgve1dji0dccdy86uqyhq1m0
# Qme85tx5Wnsjc5pZZs1JGogBNUVM2WThC18ERh6t2YFJSK 37 lorem.txt

ipfs name publish -k test QmaDDLFL3fM4sQkQfV82LdNqtNnyaeAmgC46Qc7FDQdkq8
#    Published to k51qzi5uqu5dkqxbxeulacqmz5ekmopr3nsh9zmgve1dji0dccdy86uqyhq1m0: /ipfs/QmaDDLFL3fM4sQkQfV82LdNqtNnyaeAmgC46Qc7FDQdkq8

# Since it's not a folder this time, we use 'ipfs cat' to read 
# it to the console, since we know the file was plain text.
ipfs cat /ipns/k51qzi5uqu5dkqxbxeulacqmz5ekmopr3nsh9zmgve1dji0dccdy86uqyhq1m0
# foo bar foo bar foo foo foo
# bar foo foo bar bar foo bar

DNSLink

Set a TXT record on _dnslink above the (sub)domain you want to use as an IPNS reference. Set the value to dnslink=/ipns/<id> or dnslink=/ipfs/<id> depending on whether you're pointing it at an IPFS object or an IPNS address, and replace <id> with the object ID / IPNS address you want to point it to.

          Domain:     privex.io
(Subdomain) Name:     _dnslink.test
     Record Type:     TXT
           Value:     dnslink=/ipns/k51qzi5uqu5dkqxbxeulacqmz5ekmopr3nsh9zmgve1dji0dccdy86uqyhq1m0
    TTL (expiry):     120 (seconds)

Just like normal IPNS, you should now be able to query it with IPFS CLI tools, or IPFS gateways by using /ipns/<your_domain> instead of /ipfs/<object_id>.

If we now cat /ipns/test.privex.io we can see it's working properly, pointing to the foo bar text file (no wrapped folder).

ipfs@privex ~ $ ipfs cat /ipns/test.privex.io
foo bar foo bar foo foo foo
bar foo foo bar bar foo bar

Add an existing IPFS object ID to another IPFS object (wrapped folder)

Using the following command, you can add an individual IPFS file, or an entire wrapped folder to an existing object using their respective object IDs, and the command will output a new object ID, referencing a new object that contains both the original folder data, and the new data that you wanted to add.

The syntax for the command is: ipfs object patch add-link [object-to-add-to] [name-of-newly-added-file-or-folder] [object-to-inject]

ipfs@privex:~$ ipfs object patch add-link QmXCfnzXHThHwaTvSSAKeErxK48XkyVoL6ZNEhkpKmZyW3 hello/foo.txt QmaDDLFL3fM4sQkQfV82LdNqtNnyaeAmgC46Qc7FDQdkq8
QmaWoYZnSXnKqzskrBwtmZPE74qKe4AF5YfwaY83nzeCCL

The 2 keys to mutability

1. Having an IPFS object ID that stays the same despite the content changing

Unfortunately, IPFS object IDs (the ones starting with Q) are immutable, meaning their contents cannot be altered in the future without getting a new ID, due to the fact an object ID is effectively a hash (usually a form of SHA256).

HOWEVER, both IPNS and DNSLink have a solution for this.

IPNS is "Interplantary Name System", which is strongly integrated into IPFS. It allows you to generate an address (public key) and a private key, similar to how Bitcoin and many other cryptocurrencies work. Using your private key, you can point your IPNS

First, you'll want to generate a key (note: you'll need a key per individual IPNS address you want)

ipfs@privex:~$ ipfs key gen test
k51qzi5uqu5dkqxbxeulacqmz5ekmopr3nsh9zmgve1dji0dccdy86uqyhq1m0

If you plan to use your IPNS address for something other than testing, you should export the private key and keep a copy of it somewhere safe. Note that the private key is a binary file, so if you want to store it somewhere that expects plain text, you can convert it into base64 like so: base64 test.key

ipfs key export -o /home/somewhere/safe/test.key test

Next we'll publish a random IPFS folder to the IPNS address, which contains one file (lorem.txt) with a few lines of lorem ipsum text. If you use the FUSE /ipns folder, you may need to unmount it before you're able to publish via IPNS:

ipfs@privex:~$ umount /ipns

ipfs@privex:~$ ipfs name publish -k test QmWRsWoZjiandZUXLyczXSoWi84hXNHvBQ49BiQx9hPdjs
    Published to k51qzi5uqu5dkqxbxeulacqmz5ekmopr3nsh9zmgve1dji0dccdy86uqyhq1m0: /ipfs/QmWRsWoZjiandZUXLyczXSoWi84hXNHvBQ49BiQx9hPdjs

ipfs@privex:~$ ipfs ls /ipns/k51qzi5uqu5dkqxbxeulacqmz5ekmopr3nsh9zmgve1dji0dccdy86uqyhq1m0
Qme85tx5Wnsjc5pZZs1JGogBNUVM2WThC18ERh6t2YFJSK 37 lorem.txt

That's just one example though - to prove that the IPNS address can actually be updated with different content, in this next example, I'll publish an individual text file directly to the IPNS address (not a wrapped folder).

# Publish the IPFS object 'QmaDDLFL3fM4sQkQfV82LdNqtNnyaeAmgC46Qc7FDQdkq8' 
# to our existing named key 'test'
ipfs@privex:~$ ipfs name publish -k test QmaDDLFL3fM4sQkQfV82LdNqtNnyaeAmgC46Qc7FDQdkq8

# Since it's not a folder this time, 'ipfs ls' won't return anything. 
# So instead, we use 'ipfs cat' to read it to the console, since we 
# know the file was plain text.
ipfs@privex:~$ ipfs cat /ipns/k51qzi5uqu5dkqxbxeulacqmz5ekmopr3nsh9zmgve1dji0dccdy86uqyhq1m0

foo bar foo bar foo foo foo
bar foo foo bar bar foo bar

DNSLink

DNSLink is a part of IPNS that allows for human readable IPNS addresses through the standard domain system (e.g. example.com).

Since the IPNS section was rather long, I'll keep this one short and sweet. If you want to know more about DNSLink, please visit dnslink.io.

First, either you already have a domain to use, or you acquire a domain from a registrar such as Namecheap.

Go to your domain record management panel - if you use Cloudflare, then they are your domain management panel. Add a TXT record for _dnslink.yourdomain.com or if you want to use a subdomain, _dnslink.mysub.yourdomain.com (on most registrars, you only enter the part before the domain you're managing, i.e. _dnslink or _dnslink.mysub).

In the value box, enter dnslink= followed by either /ipfs/ or /ipns/ depending on whether you want to use an IPFS object ID or an IPNS name address, then enter your object ID / IPNS name to the end.

For example, if you were pointing your domain to the IPNS address in the earlier example, you'd enter:

dnslink=/ipns/k51qzi5uqu5dkqxbxeulacqmz5ekmopr3nsh9zmgve1dji0dccdy86uqyhq1m0

Or if you wanted to point it to the example folder containing lorem.txt with a few lines of lorem ipsum, it would be

dnslink=/ipfs/QmWRsWoZjiandZUXLyczXSoWi84hXNHvBQ49BiQx9hPdjs

For example purposes, here's a summary of how I setup test.privex.io

          Domain:     privex.io
(Subdomain) Name:     _dnslink.test
     Record Type:     TXT
           Value:     dnslink=/ipns/k51qzi5uqu5dkqxbxeulacqmz5ekmopr3nsh9zmgve1dji0dccdy86uqyhq1m0
    TTL (expiry):     120 (seconds)

(note: most people are fine with "auto" TTL, or the somewhat standard 600 TTL. If you intend to change the DNSLink value regularly, or you're experimenting and likely updating it constantly, you may want a low TTL of 60 or even 30)

After setting it up, with the IPNS address still pointing at the raw foo bar text data, I used ipfs cat to read the data that the domain pointed to:

ipfs@privex:~$ ipfs cat /ipns/test.privex.io
foo bar foo bar foo foo foo
bar foo foo bar bar foo bar

2. Add existing IPFS objects to your object, without having to download/organise the object being added.

First we create the IPFS object - a wrapped folder containing hello/lorem.txt - which has the object ID QmXCfnzXHThHwaTvSSAKeErxK48XkyVoL6ZNEhkpKmZyW3

ipfs@privex:~$ mkdir hello
ipfs@privex:~$ echo -e "lorem ipsum dolor\nlorem ipsum dolor\n" > hello/lorem.txt

ipfs@privex:~$ ipfs add -p -r -w hello
added Qme85tx5Wnsjc5pZZs1JGogBNUVM2WThC18ERh6t2YFJSK hello/lorem.txt
added QmWRsWoZjiandZUXLyczXSoWi84hXNHvBQ49BiQx9hPdjs hello
added QmXCfnzXHThHwaTvSSAKeErxK48XkyVoL6ZNEhkpKmZyW3
 37 B / 37 B [=======================================================================] 100.00%

ipfs@privex:~$ ipfs ls QmXCfnzXHThHwaTvSSAKeErxK48XkyVoL6ZNEhkpKmZyW3
QmWRsWoZjiandZUXLyczXSoWi84hXNHvBQ49BiQx9hPdjs - hello/

ipfs@privex:~$ ipfs ls QmXCfnzXHThHwaTvSSAKeErxK48XkyVoL6ZNEhkpKmZyW3/hello
Qme85tx5Wnsjc5pZZs1JGogBNUVM2WThC18ERh6t2YFJSK 37 lorem.txt

Next, for the sake of creating an example external object ID that isn't part of the original wrapped folder, I created foo.txt containg a couple of lines of random foo bar text, and uploaded it to IPFS on it's own. It's object ID is QmaDDLFL3fM4sQkQfV82LdNqtNnyaeAmgC46Qc7FDQdkq8

ipfs@privex:~$ echo -e "foo bar foo bar foo foo foo\nbar foo foo bar bar foo bar\n" > foo.txt

ipfs@privex:~$ ipfs add foo.txt
added QmaDDLFL3fM4sQkQfV82LdNqtNnyaeAmgC46Qc7FDQdkq8 foo.txt
 57 B / 57 B [======================================================================] 100.00%

Finally, we use ipfs object patch add-link to add the foo.txt object (QmaDDLFL3fM4sQkQfV82LdNqtNnyaeAmgC46Qc7FDQdkq8) I created before, inside of the hello/ folder of the original wrapped folder I created (QmXCfnzXHThHwaTvSSAKeErxK48XkyVoL6ZNEhkpKmZyW3).

The syntax for the command is: ipfs object patch add-link [object-to-add-to] [name-of-newly-added-file-or-folder] [object-to-inject]

ipfs@privex:~$ ipfs object patch add-link QmXCfnzXHThHwaTvSSAKeErxK48XkyVoL6ZNEhkpKmZyW3 hello/foo.txt QmaDDLFL3fM4sQkQfV82LdNqtNnyaeAmgC46Qc7FDQdkq8
QmaWoYZnSXnKqzskrBwtmZPE74qKe4AF5YfwaY83nzeCCL

It outputs a new object ID QmaWoYZnSXnKqzskrBwtmZPE74qKe4AF5YfwaY83nzeCCL which is the ID of the newly created object that contains both hello/lorem.txt from the original, and hello/foo.txt which was injected later on.

NOTE: This command ALSO works when adding entire wrapped folders to another wrapped folder, however, be careful to avoid double nesting. e.g. you have Qxxxx/hello/world and Qyyyy/lorem/ipsum - if you add Qyyyy to Qxxxx specifying the name lorem - it will be added as Qzzzz/lorem/lorem/ipsum

If we now do ipfs ls on the new object ID, we can see that the hello/ sub-folder contains BOTH foo.txt and lorem.txt - confirming that foo.txt was successfully injected into the duplicate, without needing to download both the original and foo.txt - then organising them properly in a folder before uploading.

ipfs@privex:~$ ipfs ls QmaWoYZnSXnKqzskrBwtmZPE74qKe4AF5YfwaY83nzeCCL
QmbU3BwdMarL8n6KCzVdYqMh6HEjCv6pLJQZhoVGWZ5bWW - hello/

ipfs@privex:~$ ipfs ls QmaWoYZnSXnKqzskrBwtmZPE74qKe4AF5YfwaY83nzeCCL/hello
QmaDDLFL3fM4sQkQfV82LdNqtNnyaeAmgC46Qc7FDQdkq8 57 foo.txt
Qme85tx5Wnsjc5pZZs1JGogBNUVM2WThC18ERh6t2YFJSK 37 lorem.txt

Summary

As explained in the first section, IPFS object IDs are immutable, thus while it's possible to merge existing objects on IPFS, it still results in a new object ID.

BUT, by using IPNS key addresses and/or DNSLink, you can have a mutable (editable) reference that points to any IPFS object, and can be updated to point to a new object ID on-demand, e.g. whenever you update the contents of an existing object, or if you decide you simply want your IPNS key/domain to point at something completely different, you're free to do so :)