#!/bin/sh
# Shellballs & Self-Whatevering
#
# This blog post is an executable sh(1) program, which you can run as:
#   sh post.txt
#
# These days, the absolutely predominant way to install packages on unix-like
# systems is via a package manager, which provides (and imposes) a standard
# notion of a "package" and gives the sysadmin tools to work with and maintain
# them. The rise of package managers in general, and also standards for how
# software in source form is distributed, have sent the technique I'm about to
# describe firmly into the dustbin of history. However, it still does have
# occasional uses, and provides a neat little window into one of the things you
# wouldn't necessarily assume the shell was able to do.
#
# So, let's imagine you are (for some unwholesome reason) a vendor of
# proprietary software shipping binaries that run on Unix-ish systems. Your
# software presumably has some dependencies, without which it won't even start
# running - for historical accuracy, let's say something like Motif. You could
# simply link statically against this library or include a copy of it in your
# app; modern software happily does this and big programs routinely include
# dozens or even hundreds of other libraries, since disk space these days is
# virtually free relative to the size of code. However, in the Old Days, this
# would have both made your program use a good deal more disk space and network
# transfer. Worse than that, you had much higher odds that the version of Motif
# (or whatever) installed locally had fixes or behavior specific to that system
# that an upstream version you packaged might be lacking. Ow.
#
# So, of course, you can dynamically link. The problem with *that* is that if
# you try to dynamically link against some other library and it isn't present,
# the program loader will tell the user something cryptic on stderr instead of
# what you want to say, which is "install Motif please".
#
# Or: suppose that you are shipping a binary patch for an existing piece of
# software.  You're going to ship it via email (naturally) so it needs to be
# email-clean, even though it probably contains some binary. You could include a
# base64ed binary patch and instructions for how to unbase64 and apply it, but
# wouldn't it be easier if your patch file could simply be an executable the
# user could run which was *itself* email-clean?
#
# Or: you are shipping a large tarball containing three separate configurations
# of the same software, and you want the user to be prompted to choose and
# extract only one of those.
#
# The central trick here is as follows: a shell program doesn't have to be the
# only thing in its file! It's totally valid to have a shell script with garbage
# (from the shell's perspective) after it as long as the script stops running
# before hitting the garbage, or skips over it. This is very weird from the
# perspective of most compiled languages or even from a language like Python,
# where this program doesn't run at all:
#
#   sys.exit(0)
#   abcdef(
#
# ... because the Python interpreter tries to parse the entire file before
# executing it. The shell interpreter has no such pretensions, and happily runs:
#
#   exit 0
#   (
#
# We can use this to our advantage! Our shell script can have a code part,
# which is a valid shell script, and as long as the code part exits before
# trying to run the data part, everything will be fine. In fact, we can
# guarantee that by always having the code part end with exactly 'exit 0' - and
# if we always base64 the data part, we're guaranteed that that string can never
# appear in the data, so it's very simple to detect where the divide is.
#
# Let's do an example. We're going to make a self-extracting tarball that both
# checks its own integrity and decrypts itself when the user runs it. How are we
# going to do that? Well, we'll need both a decryption tool and some way to
# check integrity. The regular unix tool set offers plenty of ways to do the
# latter but few for the former, so we'll actually bundle our own. We'll have
# *two* "data parts", one for the decryption program and one for the tarball
# itself. In fact, we'll define a format here that supports as many data parts
# as we need with names for them: they start with 'Section: whatever' and end
# with a blank line.
#
# First, we'll need a couple of functions. Obviously if this was a Real Shell
# Program this would be called dpn() or something, but this is a blog post, so.
# data_part_named(foo) extracts the contents of the data part named foo to a
# temporary file and outputs the name of the temporary file.

data_part_named() {
	p=$(mktemp)
	# To pull out a named part, use sed(1) to delete all the lines up to and
	# including the section header, and then all the lines from the first
	# blank line to the end.
	sed --posix -e "1,/^Section: $1\$/d" \
                    -e "/^\$/,\$d" \
                     < "$0" > "$p"
	echo "$p"
}

# This function takes the pathname of a base64-encoded payload file and the
# pathname of an md5sum file to check it against. It validates the md5sum
# against the decoded file, then removes the checksum file. In a real shellball
# you'd want some error handling here in case the checksum fails. You would
# also, of course, not use md5sum.

check_integrity() {
	base64 -d "$1" | md5sum -c "$2"
	rm "$2"
}

# This function decodes the decryptor program (included in the shellball) with
# base64, uncompresses it, makes it runnable, then prompts the user for where to
# decrypt the payload file to and runs the decryptor on the base64-decoded
# payload. After that it cleans up all the other temporary files this tool made.
#
# In this case, the decryptor just[1] xors the input with 0x42, but a real one
# would prompt for a key and use a real encryption algorithm. Also, the handling
# of temporary files here is deliberately pretty sloppy to keep it simpler to
# read.

decrypt() {
	base64 -d < "$2" | gunzip > "$2.run"
	chmod +x "$2.run"
	read -p "where to: " outpath
	base64 -d < "$1" | "$2.run" "$key" > "$outpath"
	rm "$1" "$2" "$2.run"
}

# Pull the encrypted tar file and the checksum file out of the shellball. Check
# that the tar file isn't damaged by validating the checksum - in a real
# shellball the tar file would be the biggest part and the most likely to get
# damaged.

encrypted_tar_path=$(data_part_named "tarball")
checksum_path=$(data_part_named "checksum")

check_integrity "$encrypted_tar_path" "$checksum_path"

# Now, pull the decryptor out and run it on the encrypted tar file.

decryptor_path=$(data_part_named "decryptor")
decrypt "$encrypted_tar_path" "$decryptor_path"

# Done! So, here's what happens when you run this script:
# 1. The encrypted tarball and checksum sections are pulled out into temporary
#    files
# 2. The encrypted tarball is verified with the given checksum
# 3. The decryptor is pulled out into a temporary file and made runnable
# 4. The decryptor is run on the encrypted tarball to produce the decrypted
#    tarball
# 5. All the temporary files are cleaned up

# A couple of other unconnected notes:

# I created this file by hand, but there is actually a tool called makeself(1)
# that produces self-extracting "shellballs" like this. It supports a whole
# bunch of niceties, but basically you do something like:
#
#   makeself /path/to/package package.run "Your Package" ./setup
#
# and makeself arranges for the setup script to be run after the package is
# validated, maybe decrypted, and unpacked.

# This technique relies on embedding an encoded tarball inside a shell script,
# but if you wanted to do so, you could produce a tarball that was *itself* a
# valid shell script. This is possible because the very first field in the
# tarball is a filename, which is essentially arbitrary text. Constructing a tar
# file header that is both a valid shell script and that tar will ignore is a
# subject for a future blog post, but I believe it's possible :)

# That's it for now! Thanks for reading y'all :)
 
exit 0

Section: checksum
90c9d0a328304ff61cd80a7c83a3ef93 -

Section: decryptor
H4sIAEGI4GEAA6t39XFjYmRkgAEmBjsGME/AAcx3gIpHCMKVAMUsgOocGFgZWMBqWRiQgQMKvQHK
g9EMAhAKpI8ViQ+zD0a7Q4Xd0fSNglEwCkbBKBgFo2AUjIJRMApGwSgYBaOAdODR/IbDo/OZx/FD
oB65x/EDYMHjx0EUP6tH64ES6S4ZlYbPTh0yKkBZiKLjjGDZ1xeAIjZgJk7zIeMDH/+ji3NDZflg
/X4oEICK26GJS4HlGBHjBFAgChNXQBVXxCGugUOcISU1uaiyoEQvnyEvtaKEISU/L5UhPj6puDi+
uCSxqIQhPjUlsSQRSOWlMDDoFVfmliQmAemSIgidAWOVgHRTDqQZIGHHBhOAj49AKHc09YxY+ExY
zPWA6j8A5TMzQMZvOKB8CSjNCZVDBxzQcSBdAvYL4tBvCtWvSEA/ALq49vSYEgAA

Section: tarball
XclKQhLKoiNCQa+QHw2AclZEop+QHTNQckPBmqieTjy+Vo8EWXEQ9CevNrqpL4m6cmY6B5amPyz4
8fx/3tAL9yqU99/bqj760WbzHQc4hU2VYsvRCoZgp8CFwJ0EHsra5VOzcYz2kVjZdwbTkDg/aLWR
vH135v5v9t4RJKroYJs9TLllAU3mp4OcfIHElvan3iSVme8FaAg45Oj3zo+cNtcl2By/XS6bJNvx
wysxWJ8Nq+Bzd1nu6LWyVqggEc3G7SaFCJe7UawlFmSFPphlh+aNx3UbNRoLlYHuj2CR2E/ucHRm
9PunZYvoFo61/raPqqfLOlWdLhjOv+nLSaWsMhV37OWZqalyK+zlP2i2uipI6YSgKnKzL3Izdpow
F9H9XiE83jIHl1f0jWX5OS9NvrQVQEJCQkJCQkJCQkJCQkJCQkJCPZNVMNypDEJqQkI=

Section: footnotes
[1]: In the original version of this post, the decryptor was a C program,
     dynamically linked against libc to keep size down. This led to it not
     working on systems with an older glibc than I have or a different libc
     altogether. Sadly, statically linking that (trivial) C program would
     have produced a huge post, so I rewrote the decryptor in assembly; the
     encrypted tarball contains the source of the assembly version. The assembly
     source, if you happen not to be on an AMD64 Linux machine like me, is
     attached below as another section.

     Thanks to tunas of tilde.town for reporting that :)

Section: decryptor source
# build: as -o decrypt.o decrypt.s ; ld -o decrypt decrypt.o
# run: ./decrypt < in > out

.global _start
_start:
        sub $8, %rsp
        mov %rsp, %rsi
        mov $1, %rdx
next:
        # read(0, %rsp, 1)
        mov $0, %rax
        mov $0, %rdi
        syscall
        test %rax, %rax
        jz done

        # *%rsp ^= 0x42
        mov (%rsp), %bl
        xor $0x42, %bl
        mov %bl, (%rsp)

        # write(1, %rsp, 1)
        mov $1, %rax
        mov $1, %rdi
        syscall
        jmp next

done:
        # exit
        mov $60, %rax
        syscall