This is a collection of every code change I contributed across companies, client work, and other projects.
Build a simplest signal of my technical work.
This pulse is far from perfect. There's a fundamental limitation of what you can get from a simple sparkline. Whatever you choose to keep in it, what you strip out, what you add. It's mostly real, but I also added some estimations, and I extrapolated some data. Paradoxically, If I didn't do those things that make the data "not real", the data would cease to be a realistic representation of reality.
Version control makes looking at historical contributions very simple. However, a sparkline of my commits wouldn't be very useful. Companies use different development paradigms, and I want this signal to normalize all of those into a singular meaning.
Therefore, the final sparkline attempts to roughly capture some of the following:
What this signal fails to capture
I'm dealing with sensitive data here. So, before concluding projects, I manually created anonymized artifacts that only store event metadata and salted hashes of identifiable data (such as authors). This pipeline very intentionally excludes all code, author information and even commit messages. This prevents any mistakes in the pipeline from accidentally publishing any sensitive data anywhere.
The same rules apply to code that I actively manage. Even when I have the code, this pipeline never reaches any code, or personally identifiable information.
#!/usr/bin/env bash
#
# Turn ANY git repository into an anonymized pool of commit events. The built
# artifact contains no code, no commit messages, no names (branches), no paths,
# while keeping known authors identifiable through salted hashes.
#
# ./build-git-repo-artifact.sh /path/to/repo > events.csv
#
# The salt is random per run and never stored. Set SALT=... to keep author ids
# consistent across repos, then discard it. Needs only git and openssl; runs
# on bash 3.2 (stock macOS).
#
# This script has been built for my build pulse project
# https://ondrejbarta.com/build-pulse
set -euo pipefail
repo="${1:-.}"
salt="${SALT:-$(openssl rand -hex 32)}"
FIELD_SEPARATOR=$'\x1f'
anon() {
hash=$(printf '%s' "$1" | openssl dgst -sha256 -hmac "$salt" -r)
echo "${hash:0:16}"
}
echo "sha,parents,author,author_time,is_merge"
# `|| [ -n "$sha" ]` keeps the final record - git's output has no trailing
# newline, and read alone would drop the root commit.
git -C "$repo" log --all --pretty=format:'%H%x1f%P%x1f%ae%x1f%at' |
while IFS="$FIELD_SEPARATOR" read -r sha parents author_email author_time ||
[ -n "$sha" ]; do
hashed_parents=''
for parent in $parents; do
hashed_parents="$hashed_parents${hashed_parents:+ }$(anon "$parent")"
done
is_merge=false
case "$parents" in *' '*) is_merge=true ;; esac
echo "$(anon "$sha"),$hashed_parents,$(anon "$author_email"),$author_time,$is_merge"
doneThe pool it produces looks like this:
| sha | parents | author | author_time | is_merge |
|---|---|---|---|---|
| 31e03a50da220e20 | 0005425c743aec0f | b91e11e5c36dd808 | 1785076107 | false |
| 0005425c743aec0f | f3b0f512ce0c7acf | b91e11e5c36dd808 | 1785072081 | false |
| f3b0f512ce0c7acf | f8ac6c88450bec93 | b91e11e5c36dd808 | 1785071803 | false |
| … | ||||
For the builder pulse, I decided to work with the following sources:
I also considered others, that didn't end up being useful.
The result is the data you see on your screen. This is the total of activity in all the source repositories and files.
As you can see, we're far from done. When dealing with high volume of unstructured data, it gets messy. There are often either gaps, or spikes that are unaccounted for. In the end, you have to understand all your data in order to extract the correct meaning. E.g. is a spike real, or was there a change in how the data was collected or what it represents?
Right now, the data contains all commits across all the repositories I've ever touched. First obvious problem is that not all commits are mine. Thorough my tenure, I used various accounts, so first I needed to manually dig through the data, and identify all the users that were mine.
At this point, commit authors in my dataset have been hashed with custom salt, so at this stage I only need to prepare a list of my own account hashes to match to my commits. Once I had a list of my accounts hashes, it's easy to write a query and remove the ones that are not mine.
In the highlighted portion of the data, you see a sizeable spike. That's a good indication that the data isn't quite clean yet. At that time, the repository was brand new (with a small team at Bitwala we just started building the product from scratch) and it wasn't set up to squash commits.
Easy fix is to look at the merge events, and only account for that during this period, which extends from the creation of this repository to the last non-squash merge commit.
This is a very simple data pipeline. And I know the data intimately.
Commits are hardly a good indicator of quality of contributions. I distinctly remember a week that I spent on massive refactoring, which culminated in a single pull request. That work was disproportionately more important than some 10 other commits that I made in a single day, fixing typos or making variable names clearer.
To normalize the importance of the low commit days, I have decided to apply a log scale. It's not a perfect solution, but periods of high commit rates would normalize over the days of low commit frequency.
Ultimately, I'm a believer in smaller commits. They're easier to review, they are typically better at isolating logic. One gigantic benefit of smaller commits is that, over time, this practice will modularize your thinking. A small self-contained change will ensure the codebase modularity almost automatically, because on each small change, your CI will (hopefully) ensure that trunk is still a working codebase.
My intention was to generate a succinct pulse of my work, and I believe I got as close as I could with the build pulse. The data is now properly clustered, clean, normalized, and actually represent my build history well enough to be a good indicator.