Arch Linux

RubenKelevra

Member

Registered: 2013-03-01

Posts: 41

New Distributed Cluster Mirror via IPNS (Interplanetary File System)

Hey Guys!

This was requested like back in 2015 in the Forum and in a ticket in the IPFS-project. I gave it a go and set-up a fast server in Germany and wrote the scripting necessary to do the incremental updates.

IPFS can run as a cluster, so one or more servers can write to a cluster, while everyone can set up a "follower" of it, and hold another copy of the data, get updates as a stream & reprovide it to the IPFS-Network.

If you want to use it as your primary update server, feel free to do so. You need the IPFS-Daemon setup and running as a service on your system. (Details to do this in the wiki)

Afterwards you can add this as your first server entry (change the port if you setup your IPFS web gateway differently):

# IPFS
Server = http://127.0.0.1:8080/ipns/pkg.pacman.store/arch/$arch/default/$repo

If you run more than one computer in your network, you can use IPFS to share the data locally, you've already downloaded. Just turn mDNS-discovery on, and IPFS will automatically connect the computer and transfer the data locally if possible.

By default IPFS will hold 10 GB in its cache, you can change this number in the IPFS settings if you feel like that's too much or too little for your needs. The cache is garbage collected automatically.

Any additional information can be found in the git on github under pacman.store.

Last edited by RubenKelevra (2020-03-09 23:27:39)

RedArcher

Member

From: South Africa

Registered: 2009-09-03

Posts: 88

Re: New Distributed Cluster Mirror via IPNS (Interplanetary File System)

This solves so much to me.

Thank you for establishing an IPFS accessible mirror.
However, I'd be delighted if you would kindly arrange a step by step guide to clone or imitate your mirror.

Thank you!

RubenKelevra

Member

Registered: 2013-03-01

Posts: 41

Re: New Distributed Cluster Mirror via IPNS (Interplanetary File System)

Hey @RedArcher,

glad to hear!

So you can run a local copy of the mirror easily by "following" the cluster. You need ipfs on a machine, for example via my aur package:

https://aur.archlinux.org/packages/go-ipfs-git

After installation there's a config in /var/lib/ipfs/, modify Datastore.StorageMax to be above 315 GB. Documentation:

https://github.com/ipfs/go-ipfs/blob/ma … storagemax

You need to start/enable the service ipfs  (it will save it's data under /var/lib/ipfs/ by default). Check that it has been fully started.

Then you need my AUR package for the cluster:

https://aur.archlinux.org/packages/ipfs-cluster-bin

And then you need to start the provided binary ipfs-cluster-follow with the following parameter (just create yourself a service file or start it in a screen):

```
ipfs-cluster-follow pkg.pacman.store run --init cluster.pkg.pacman.store
```

This will connect to the rest of the cluster and fetch the newest version if the mirror to your local ipfs client and update it, as soon as new data is available. The build-in garbage collection of ipfs is already on by default if you use my go-ipfs-git package. Else you need to modify the service file to include "--enable-gc" if you don't do this IPFS will never reclaim any diskspace and just keep filling your disk until it's full.

Since all ipfs clients do connect to eachother on your local network via mDNS you should see very fast transfer speeds, after the cluster has been mirrored.