Managing Temporary Files: A Professional Approach

7 min read

#Linux #Bash

Part 6 of 6 in Running Linux, My Way

Temporary files turn into a mess if you don’t have a system for them. Here’s the setup I use to keep them organized, and why it works.

What is a Temporary File? #

A temporary file is something you create, generate, or download for immediate use, but you’re not sure if you’ll need it later. For example, say you write a quick bash script to upload something to S3. You need it right now, but maybe not tomorrow. It’s not part of your project, and you don’t want to commit to that yet. You just want to experiment.

Another example is downloading an image from the internet just to open it and poke around a bit. You’re not sure if you’ll need it later either.

These are just a couple of examples. If you work with computers daily, you probably use temporary files several times a day.

The Mess with Temporary Files #

The problem with temporary files is the mess they create. They usually fall into two categories. The first is stuff downloaded from the internet, which ends up in the “Downloads” folder. The second is stuff you create yourself, which ends up on the “Desktop” because that’s the easiest place to find later. Some people use more folders and try to organize things better. But you can’t file these away properly from the start, because the whole point of a temp file is that you don’t know yet if you’ll need it. That uncertainty is exactly what causes the mess over time.

Go look at an actual Downloads folder that hasn’t been cleaned in a year. You’ll find the same fossils in every single one: forty screenshots named “Screenshot 2023-04-12 at 11.42.03 AM.png”, a PDF called “invoice_final_FINAL_v2.pdf” that is neither final nor the final version, three installers for the same app because you forgot you already had it, and at least one “New Folder” that somehow still has a file extension. Desktops are worse. People treat the Desktop like a junk drawer with icons on it: forty of them scattered around with no order, half named “New Folder (2)” because apparently renaming things is optional.

A completely scientific survey of the average Downloads folderA bar chart of what actually fills a typical Downloads folder: dozens of screenshots and images, a few empty new folders, one file that claims to be final, and only two files anyone actually still needs.a completely scientific survey of the average Downloads folderscreenshots47IMG_XXXX.jpg89"New Folder" (also empty)12invoice_final_FINAL_v2.pdf1things you actually needed2
Peer-reviewed methodology: I looked at my own Downloads folder and felt shame.

Eventually you’re staring at hundreds of files with no idea which two or three you actually still need, trying to decide between spending twenty minutes sorting through them or just nuking the whole folder and hoping for the best.

A messy folder structure costs you time: time spent finding things, time spent finishing work, the occasional lost file, and the annoyance of running out of disk space and having to clean it up.

Introducing the Temp Folder #

When I was a young programmer, I found myself using the “Downloads” and “Desktop” folders excessively. This created problems because it took me time to find things and work efficiently. I tried to be more organized by naming those files logically, but it was difficult for the reasons above.

Eventually, I came up with a simple solution, which I want to share. I created a new folder called “Temp” to serve as the root directory for temporary files. No one, including myself, creates files directly in this folder. Instead, it contains subdirectories, each named with the current date and time, such as “2006-01-02-150400”. Whenever I need to write something to a temporary file, it goes into a folder with the current date and time under the “Temp” directory.

Everything temporary lives under one root, sorted by when it was madeA single ~/temp root folder with four subfolders, each named after the date and time it was created, so nothing is ever created directly in the root.one root, sorted by when you made it~/temp2024-01-15-0930222024-02-03-1410552024-02-19-1022332024-03-08-160511
Nothing lives in the root. Everything lives in a dated folder, so you always know when you made it.

Of course, it’s tedious to type the current date and time every time I need a temporary file, so I wrote a bash function to handle it:

export ROOT_TEMP_DIR=~/temp

function temp() {
    temp_dir="$ROOT_TEMP_DIR/$(date "+%Y-%m-%d-%H%M%S")"
    mkdir -p "$temp_dir"
    cd "$temp_dir"
}

(Yes, this only works if you live in the command line. If you’re afraid of it, you’re on the wrong blog. Sorry.)

Once you have this function, put it in your .bashrc (or the equivalent config file for your shell). From then on, whenever you need a temporary location for a file, just type temp in the terminal and you’ll land in the right folder.

I also use a small alias that works on Linux:

function cpath() {
    pwd | copy
}

The cpath (copy path) function copies the current directory path to the clipboard. Run temp followed by cpath, and you’ll have the temp location copied and ready to paste. Switch to your browser with alt-tab, paste the path with ctrl+v, and you’ve got quick access to your newly downloaded file.

Time-Based Cleanup #

When the “Temp” folder gets cluttered or disk space runs low, it’s time to clean up. The simplest approach: delete anything older than six months. Just make sure nothing important is sitting in there. By the six-month mark, you should already know where each file actually belongs. If it’s still in Temp after that, it’s safe to delete. If it’s something you still need, that’s on you for not moving it out.

Folders older than six months get deleted automaticallyA timeline of temp folders by age. The three folders older than six months are marked for deletion with a dashed outline. The three folders younger than six months are kept with a solid outline.anything older than six months gets deleted, no questions asked6 months agooldertoday
No warning, no confirmation prompt. Six months up means it's gone.

Here is a script to clean the temp folder:

function clean_temp() {
    current_date=$(date +%s)
    cd $ROOT_TEMP_DIR

    for dir in */; do
        dir=${dir%/}
        dir_date=${dir:0:10}
        dir_date_seconds=$(date -d "$dir_date" +%s)
        difference=$(( (current_date - dir_date_seconds) / (60*60*24*30) ))

        if [ $difference -ge 6 ]; then
            echo "Deleting directory: $dir"
            rm -rf "$dir"
        fi
    done
}

Add this function to your .bashrc to make it available in every shell session.

Temp Aliases #

To make the temp function more useful, I added support for an optional alias that gets appended to the directory name, for example “2006-01-02-150400-<some_alias>”. The alias should be a word or phrase that helps you remember what the folder was for. This is especially handy if you create several temp folders in one day and don’t remember the exact time each one was made. The following function lets you jump back to a temp folder by its alias:

function cdtemp() {
    alias=$1
    cd "$(find "$ROOT_TEMP_DIR" -maxdepth 1 -name "*$alias*" | sort | tail -1)"
}

Add this function to your .bashrc, and from any shell you can do:

temp uploader-script

# Perform any tasks, including moving away from the temp directory

cdtemp uploader-script # Return to the temp directory

Temp Partition #

The “Temp” folder served me well for several years, but I eventually realized cleaning it based on date wasn’t necessary. I typically wiped it with rm -rf *, which can be slow because of how the filesystem manages inodes.

This was time-consuming, and I wanted to fix it. On top of that, running heavy benchmarks and disk-intensive tasks in the temp directory caused fragmentation and slowed down the whole system.

So I created a dedicated 100GB partition and mounted it at ROOT_TEMP_DIR. Once a year, I format that partition, which wipes out all fragmentation and leftover files in one shot. It gives me a dedicated space for benchmarks and disk-heavy work without dragging down the rest of the system.