Managing temporary files can be a daunting task, often leading to cluttered directories and reduced system efficiency. In this blog post, we will explore effective strategies to organize and handle temporary files, ensuring a smoother workflow and improved system performance.

What is a Temporary File?

A temporary file is a file you create, generate, or download for immediate use, but you’re unsure if you’ll need it in the future. For example, you might write a simple bash script to upload something to S3. You need the script now to upload the file, but you might not need it later. It’s not part of your project, and you don’t want to include it just yet—you just want to experiment and test things.

Another example is downloading an image from the internet just to open and play with it a bit. You might not be sure if you’ll need the image later.

These are just a couple of examples. If you work with computers daily, you likely need to use temporary files several times a day.

The Mess with Temporary Files

The problem with temporary files is the mess they create. Typically, these files fall into two major categories. The first category includes files downloaded from the internet, often stored in the “Downloads” directory. The second category includes files we create ourselves and store in the easiest-to-find folder—the “Desktop”. Some people use more than these two folders, attempting to organize them in a more structured layout. However, due to their nature—the uncertainty of their future use—we can’t place them in the right location from the beginning, which indirectly causes a mess over time.

Ultimately, we might end up with a “Downloads” directory filled with hundreds of files, some in use and some not. We then need to take the time to sort them out or risk losing something important by deleting everything.

A messy directory structure means more time spent finding things, more time finishing work, the occasional loss of important files, and the annoyance of running out of disk space and having to clear it.

Introducing the Temp Folder

When I was a young programmer, I found myself using the “Downloads” and “Desktop” folders excessively. This created problems because it took me time to find things and work efficiently. I tried to be more organized by naming those files logically, but it was difficult due to the reasons mentioned above.

Eventually, I came up with a simple solution, which I want to share. I created a new folder called “Temp” to serve as the root directory for temporary files. No one, including myself, creates files directly in this folder. Instead, it contains subdirectories, each named with the current date and time, such as “2006-01-02-150400”. Whenever I need to write something to a temporary file, it goes into a folder with the current date and time under the “Temp” directory.

Of course, it’s tedious to write the current date and time every time I need a temporary file, so I created a bash function to handle this for me:

export ROOT_TEMP_DIR=~/temp

function temp() {
    temp_dir="$ROOT_TEMP_DIR/$(date "+%Y-%m-%d-%H%M%S")"
    mkdir -p "$temp_dir"
    cd "$temp_dir"
}

(Yes, this solution is appropriate only for those who use the command line daily. If you’re afraid of the command line, you shouldn’t be reading my blog—sorry.)

Once you have created this function, include it in your .bashrc (or the appropriate shell configuration file for your environment). From then on, whenever you need a temporary location to store a file, simply type temp in the terminal, and you will be directed to the designated folder.

Additionally, I use a convenient alias that works on Linux:

function cpath() {
    pwd | copy
}

The cpath (Copy Path) function copies the current directory path to the clipboard. By running temp followed by cpath, you will have the temporary location copied to your clipboard. You can then switch to your browser using alt-tab and paste the new location with ctrl+v for easy access to your newly downloaded file.

Time-Based Cleanup

When the “Temp” directory becomes cluttered or disk space is running low, it’s time for a cleanup. This can be done by removing older directories first, such as those older than six months. It’s crucial to ensure that no important files are stored in this directory. After a set period, you should have a clear plan for the destination of each temporary file. If a file remains in the “Temp” directory for more than six months, it can be safely deleted. If not, you’re misusing the mechanism by not relocating these files as intended.

Here is a script to clean the temp folder:

function clean_temp() {
    current_date=$(date +%s)
    cd $ROOT_TEMP_DIR

    for dir in */; do
        dir=${dir%/}
        dir_date=${dir:0:10}
        dir_date_seconds=$(date -d "$dir_date" +%s)
        difference=$(( (current_date - dir_date_seconds) / (60*60*24*30) ))

        if [ $difference -ge 6 ]; then
            echo "Deleting directory: $dir"
            rm -rf "$dir"
        fi
    done
}

Include this function in your .bashrc to make it available in every shell session.

Temp Aliases

To further enhance the temp function, we can add support for an “alias” parameter, which appends an alias to the directory name. For example, “2006-01-02-150400-<some_alias>”. This alias should be a word or phrase that helps you quickly identify the purpose of the folder. This feature is especially useful if you create multiple temp folders in a single day and do not remember the exact time each was created. The following function enables you to navigate to a temp folder with a matching alias:

function cdtemp() {
    alias=$1
    cd "$(find "$ROOT_TEMP_DIR" -maxdepth 1 -name "*$alias*" | sort | tail -1)"
}

Add this function to your .bashrc, and from any shell, you can type:

temp uploader-script

# Perform any tasks, including moving away from the temp directory

cdtemp uploader-script # Return to the temp directory

Temp Partition

The “Temp” folder served me well for several years, but I eventually realized that cleaning it based on date wasn’t necessary. I typically wiped it with rm -rf *, which can be slow due to the file system’s internal mechanisms for creating and maintaining inodes.

This process was time-consuming, and I wanted to optimize it. Additionally, running intensive benchmarks and heavy operations in the temp directory caused fragmentation and slowness to the whole system.

To address these issues, I created a dedicated 100GB partition and mounted it to the ROOT_TEMP_DIR. Once a year, I simply format this partition, which eliminates all fragmentation and leftover files. This setup provides a dedicated space for conducting benchmarks and disk-intensive tasks without impacting the rest of the system.