Dumpster

One day during class, I’ve been asked by the professor to send to the other students a few documents consisting in a couple of PDFs and DOCXs. The overall dimension of the files exceeded 25MB, so emailing them wasn’t an option. I’d deleted my Dropbox account a few days ago, so I thought about putting them on Mega - while I was doing this I realized that the process was somehow long and cumbersome for what I needed:

• Open the Mega website
• Wait for it to load the locally installed extension
• Login to it
• Upload my files
• Get a sharing link
• Get a sharing key
• Forward the links & the keys to my classmates via IM or Email.

I decided that I didn’t like this method, and begun searching for alternatives. A quick throwaway service that I found was TinyUpload; it was comfy to use on desktop, but a real pain to use from mobile.

I then decided it was time to roll my own storage engine, fully customized and tailored to my needs, to solve this problem once for all.

Stack & Code

As of all my latest projects, I quickly yeoman-ized a Node.JS template and removed all the bells and whistles that I didn’t need.

I started to build a modular engine centered on easiness of use and platform compatibility, giving particular attention to the API aspect of it. One of my main concerns was making it suitable for a CLI interface, so that I could store files without ever leaving my terminal if needed - something that I thought would have been handy in the future especially for sharing code snippets.

After having fiddled around for some time with the file upload procedure - that proved itself to be a little more tricky than what I was expecting - I had my first working prototype: it was a simple POST upload endpoint backed by the popular Express framework, that made use of the multer middleware to handle big files and multipart-encoded uploads; GETting another API endpoint with a filename in the query would let you download it. As simple as it gets.

You can see how it looked like by browsing in the initial commits of Dumpster’s GitHub repo.

The substantial code bits follows, as one of my first rough proofs-of-concept:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

var multer = require('multer');
var express = require("express");
var app = express();
var uploadPath = 'uploads/';
var domainUrl = 'http://your.domain.name/' + uploadPath; // <-- Fill in

// ...

var uploadedFileURL = null;
var upload = multer({
  dest: uploadPath,
  rename: function(fieldname, filename) {
    return filename + Date.now();
  },
  limits: {
    files: 1, // Max 1 file per upload
    fields: 0, // No extra POST fields
    fieldSize: 52428800, // 50MB
    fileSize: 52428800 // 50MB
  },
  onFileUploadStart: function(file) {
    console.log("Accepting upload: " + file.originalname + " (" + file.size + " bytes, "+ file.mimetype + ")");
    uploadedFileURL = null;
  },
  onFileUploadComplete: function(file) {
    console.log("File '" + file.originalname + "' uploaded to " + file.path);
    uploadedFileURL = domainUrl + path.basename(file.path);
  }
});

var postUpload = function (req, res, next) {
    res.end("OK - " + uploadedFileURL + "\n");
}

app.post('/api/upload', [upload, postUpload]);

app.get('/uploads/*', function(req, res) {
  res.sendFile(__dirname + req.url);
});

// ...

Worried by the misuse the lack of authentication could have led to, I pondered what my auth options were beside “standard” usernames and password. I wanted Dumpster to be a dead-simple service, and I remembered about the Yubikey I had recently bought. “Perfect timing.” - I thought.

I forked Adam Baldwin’s Yubikey library and adapted it a little to suit my needs more comfortably, using Yubico’s API to validate my uploads. I included basic support for user differentiation in case in the future I had to give access to other people using different OTP tokens.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

var yubikey = require('./yubikey.js'); // https://github.com/evilpacket/node-yubikey
yubikey.apiId = 0; // <-- Fill in
yubikey.apiKey = ''; // <-- Fill in

// ...

var authUpload = function (req, res, next) {
  var user = req.query.user;
  var token = req.query.token;
  if (user == "yourUser") { // <-- Fill in
    yubikey.verify(token, function(isValid) {
      if (isValid) {
        console.log("Valid token: " + token);
        next();
      } else {
        console.log("Invalid token: " + token);
        res.status(401).send("AUTH ERROR - Invalid token\n");
      }
    });
  } else {
        console.log("Invalid user: " + user);
        res.status(401).send("AUTH ERROR - Invalid user\n");
  }
}

// ...

app.post('/api/upload', [authUpload, upload, postUpload]);

// ...

Fast-forward to the current status

All this crappy code has evolved nicely in time, while I kept adding features to the project and making it an overall decent and more efficient platform - as you can see from the latest releases of Dumpster. Eventually I even packaged it into a Docker container and published it to the Hub.

I’ve made the authentication better and simpler, with support for multiple users and tokens. I’ve added automatic deletion of stored files after a given time, with a persistency layer (implemented via LevelDB) to avoid mess when restarting the server. I threw checksums in the mix, using HTTP headers to tell the client if the file the server got actually matched by the bit the one that was intended to be uploaded.

At some later stage I’ve also added a WebUI when I grew tired of using this terminal client I wrote:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

#!/bin/bash
set -e -o pipefail
dumpster_url="your.dumpster.host"

# https://github.com/themattrix/bash-concurrent
source ~/bin/bash-concurrent/concurrent.lib.sh

path() {
    if [[ -z "$1" ]]; then
        echo "-> No file specified!" >&3
        exit 1
    fi

    if [[ ! -f "$1" ]]; then
        echo "-> Specified file does not exist!" >&3
        exit 1
    fi
}

token() {
    if [[ -z "$2" ]]; then
        echo "-> No OTP code given!" >&3
        exit 1
    fi
}

md5() {
    echo -n "$(/usr/sbin/md5sum "$1" | awk '{ print $1 }')" > /tmp/dumpster.md5
}

upload() {
    # https://github.com/jkbrzt/httpie
    result=$(/usr/sbin/http --timeout 300 -f POST "http://${dumpster_url}/api/upload?md5=$(cat /tmp/dumpster.md5)&del=3d&token=${2}" file@"${1}")
    echo "-> $result" >&3
}

cleanup() {
    rm /tmp/dumpster.md5
}

concurrent \
    - "Path verification" path "$@" \
    - "Token verification" token "$@" \
    - "MD5 calculation" md5 "$@" \
    - "Upload" upload "$@" \
    --sequential

cleanup

A listing of the almost-complete stack used can be viewed at StackShare, also embedded below here.

As of today Dumpster is still happily running on a VPS, lending me a hand whenever I need to quickly share some files - be it from the terminal of from a web browser.

All in all it might not be the best software I’ve ever written, but I’ve had a lot of fun during the development process and learned cool new tricks; but the most important thing, however, is that I built something that I actually use and improve on a daily basis, and not the umpteenth proof-of-concept to be forgotten in a few weeks. I highly value this, as I think of it as an important corner stone in a developer’s growth.

Learn more about Dumpster by taking a look at its GitHub repository, at its online demo, or at its Codacy code quality overview.