We have an Underlay IPFS cluster! It lives at underlay.store and exposes two subdomains:

  1. gateway.underlay.store is for fetching files from IPFS. This includes files that are pinned to the cluster (which will be fast to serve) but also works as a gateway to any file that anyone has on IPFS, anywhere (which will be a little slower to serve, but at least you can get them all in the same place). This subdomain is public! We want anyone to be able to use our gateway.

  2. cluster.underlay.store is the API endpoint for adding and removing files from the cluster. This subdomain is private and not open to the Internet.

The underlay.store domain was registered on name.com in December 2018. It uses Cloudflare’s nameservers.

Architecture

The cluster is a group of EC2 instances all running 1) an IPFS daemon and 2) an IPFS Cluster daemon. IPFS Cluster is project maintained by Protocol Labs that lets groups of IPFS nodes share responsibility for storing a collective set of files (“distributed pinset orchestration”) by running a distributed consensus algorithm (Raft).

Each cluster peer is a t3.medium instance attached to a 512GB external EBS volume mounted at /dev/sdb. The user data script for each instance that handles installing and configuring IPFS and IPFS Cluster, and bootstrapping into consensus with the rest of the cluster lives at this GitHub Gist.

Gateway

The gateway has a CNAME record pointing to an AWS application load balancer that forwards HTTP and HTTPS traffic to port 8080 of its target group, which includes all of the cluster peers. This means that requests to gateway.underlay.store are forwarded one of the peers, and AWS decides which.

curl https://gateway.underlay.store/ipfs/QmFoo…

Note the /ipfs/ at the start of the path; it’s necessary.

Cloudflare is configured to cache all pages under the gateway subdomain indefinitely.

Cluster

A cluster subdomain is a separate application load balancer that exposes APIs for adding and removing files (and configuring the cluster itself). There are two sets of APIs on two respective ports:

  1. The load balancer forwards port 5001 to port 9095 of its target group, which is the IPFS API Proxy endpoint of the cluster. This API endpoint perfectly mimics the regular IPFS HTTP API documented here, except it applies pin operations to the entire cluster instead of just the single node. You can read more about the proxy API here.

  2. The load balancer also forwards port 9095 to port 9095 of its target group, which is the HTTP API for IPFS Cluster itself. This includes equivalents for adding and removing files, but also other cluster-specific endpoints like adding and removing peers and generating health report graphs. These routes are documented here.

The upshot is that there’s a single endpoint cluster.underlay.store that can behave exactly as if it’s a single IPFS node (right down to the API!) but is actually a collection of peers in consensus, all handling a share of the traffic.

curl -F file=@file.pdf http://cluster.underlay.store:5001/api/v0/add
curl http://cluster.underlay.store:5001/api/v0/pin/rm?arg=Qmfoo…

Libraries

Protocol Labs maintains JavaScript and Go libraries that interface with the HTTP API to make your life easier:

const IpfsClient = require('ipfs-http-client')

// The client assumes port 5001 by default.
const ipfs = IpfsClient("cluster.underlay.store")

// Use the JavaScript interface from the spec!
// https://github.com/ipfs/interface-ipfs-core/blob/master/SPEC/FILES.md
ipfs.add(...)
ipfs.get(...)

Security

The two load balancers on AWS are configured with security groups cluster and gateway. gateway accepts incoming traffic from anywhere on ports 80 and 443 (although Cloudflare is configured to upgrade every http connection to https on gateway.underlay.store), while cluster restricts incoming traffic to the priorart-file-parser server’s security group. It’s important to protect it from the broader internet because then anyone can make us download and store stuff.

But if you just want to test it out, add your IP to the cluster security group and refresh https://cluster.underlay.store:5001/api/v0/id a bunch of times! It should roughly alternate between the peer ids of the cluster peers.

Pinned Assets

Here are some useful things that are pinned to the cluster: