Fun with File Systems

Imagine you have a data logging application, that writes data to disk continuously. Since the application is not very stable, you want it to write out the data in small files, so it does not loose too much data, if the application crashes. This creates some need to find a good trade-off between file size and file system in order to avoid wasting too much disk space with file system overhead.

An approach to measure file system overhead and to explore the design space of different file systems and file sizes quickly is as follows:

• Create a ramdisk and in this ramdisk create a bulk file of given size (using dd)

• For all combinations of file size and file system:

• Format the bulk file with a desired file system (using mkfs) and mount it.
• Continuously write files with a fixed size to the mounted bulk file until an exception occurs and record how many files could be written to mounted bulk file (using some script).

Operations to the mounted bulk file are very fast, since the bulk file resides in a ramdisk. An experiment using this approach was conducted for a bulk file of 1 GiB. Considered file systems were ntfs, exfat, vfat, ext2,ext3 and ext4. File sizes were varied from 1 byte to 2²⁰ bytes. A plot summarizing the relative file system overhead for different file sizes and file systems is shown below:

From this figure it can be seen that file system overhead is excessive for small file sizes. ext2, ext3 and ext4 behave almost identical in terms of overhead. Minimal overhead in this experiment is observed for vfat at a file size of 65536 bytes per file. Strangely exfat is always outperformed by ntfs.

The scripts that were used to conduct this experiment can be downloaded here.

Aquarium Surveillance

Surveilling fish in an aquarium with a simple webcam is a fun activity for young and old. The following post describes a simple software setup that was used to continuously upload live still images of a small aquarium.

Nothing more is required than an aquarium, a webcam and a Linux computer with Internet connection. Optionally the aquarium may be substituted by a hamster cage. ;)

Probably ffmpeg could be used as an alternative to stream live video. However it was found that ffmpeg crashes rather often and that the encoding of popular video formats is CPU intense, which is especially infeasible on older computers. Another alternative for live video would be to use mjpegstreamer, which is especially resource preserving. Unfortunately the project seems to be inactive now and no 64-bit binaries are provided.

Therefore it was concluded that the best compromise would be to stream still images at the maximum resolution of the webcam and a refresh rate of a couple seconds. Many programs are suitable to acquire images. In this case fswebcam was chosen, because it is small and simple and can be found in the Debian/Ubuntu repositories.

Watching the fish should also have a social component. Therefore an Xajax sample program was used to provide a simple comment function ("graffiti wall"). The sample program can be tested and downloaded here.

The following packages need to be conveniently installed from the software repositories if not already present:

sudo apt-get install apache2 php5 fswebcam

Normally the images acquired with fswebcam would be written to disk continuously. To avoid letting the hard disk suffer too much, a ramdisk is created. This can easily be done by adding the following line to /etc/fstab:
#ramdisk for aquarium photos
ramfs  /var/www/aquarium/aqua_ramdisk ramfs defaults 0 0

You can download the script and the modified xajax web program here.

The webcam.sh script needs to registered for automatic startup. A description how to do this can be found here. The web program can simply be copied to the apache web folder eg. /var/www/aquarium.