gource-graph-sept-2017

Rockstor Git Gource – project visualization

Gource is Git shiny.

Gource is an open source GPLv3 licensed OpenGL visualiser of Git/SVN/Mecurial, and Bazaar repositories over time and so serves well to indicate a project’s growth / activity.

gource git snapshot showing many names
Family Code Feeding

Wikipedia’s “A picture is worth a thousand words” page has the following:

The actual Chinese expression “Hearing something a hundred times isn’t better than seeing it once” (, p bǎi wén bù rú yī jiàn) is sometimes introduced as an equivalent, as Watts‘s “One showing is worth a hundred sayings”.[6] This was published as early as 1966 discussing persuasion and selling in a book on engineering design.[7]

Rockstor’s 5 git repositories visualised in just over 1.5 minutes.

The essence of Gource’s use, at least on a linux desktop, is to initially build from the code and then point the resulting binary at a copy of one’s repository. The wrinkle here is that Rockstor, akin to many projects, consists of multiple repositories:

  • rockstor-doc – our docs in their original restructured text format.
  • rockstor-core – the main code repository.
  • rockon-registry – our docker based ‘rock-on’ plugins repo.
  • rockstor-iso – code / artifacts concerning the creation of our stable release iso images.
  • rockstor-jslibs – the collection of js libraries we use to do what js does best.

But fear not as the now 29 strong developers of Gource have us covered in their appropriately named: Visualizing-Multiple-Repositories.

“Sometimes it may be interesting to show the history of multiple projects in the same Gource animation.”

Thanks people.

This mini HowTo is essentially a re-telling of that page as applied to Rockstor’s 5 repositories. The hope is that this ‘telling’ might aid other multi-repo projects and save us all some time; at least on mass.

Building Gource

As of 8th September 2017 v0.47 was released which is the version I used here. Be sure to visit their releases page to check on availability of any newer releases. As always it’s best to resource the original text on install matters: please favour the projects own INSTALL doc over what I state here. But by way of completeness I’ll indicate how it worked for me:

Using a Fedora26 desktop:

sudo dnf install git freetype-devel pcre-devel glew-devel SDL2-devel SDL2_image-devel boost-filesystem boost-devel glm-devel
mkdir ~/Downloads/gource
cd ~/Downloads/gource
wget https://github.com/acaudwell/Gource/releases/download/gource-0.47/gource-0.47.tar.gz
tar xvf gource-0.47.tar.gz
cd ~/Downloads/gource/gource-0.47
./configure
make

We refrain from ‘make install’ to keep all our gource activities confined to our Downloads dir. (snap / flatpak suggestions anyone?)

Now we jump up a directory and grab a local copy of Rockstor’s repos:

cd ~/Downloads/gource/
git clone https://github.com/rockstor/rockstor-core.git
git clone https://github.com/rockstor/rockstor-doc.git
git clone https://github.com/rockstor/rockon-registry.git
git clone https://github.com/rockstor/rockstor-jslibs.git
git clone https://github.com/rockstor/rockstor-iso.git

then we resource Gource’s ability to create a custom-log of activity spanning all the above repos. For this we jump back into our gource build directory:

cd ~/Downloads/gource/gource-0.47

and point the binary at each of the rockstor repos in turn with switches requesting our needed custom logs.

./gource --output-custom-log ../doc-log.txt ../rockstor-doc/
./gource --output-custom-log ../core-log.txt ../rockstor-core/
./gource --output-custom-log ../rockon-log.txt ../rockon-registry/
./gource --output-custom-log ../iso-log.txt ../rockstor-iso/
./gource --output-custom-log ../jslibs-log.txt ../rockstor-jslibs/

Then back up to our new logs:

cd ~/Downloads/gource/

Next we make our repos more distinct by putting them on separate branches via multiple stream edits (sed):

sed -i -r "s#(.+)\|#\1|/doc-repo#" doc-log.txt
sed -i -r "s#(.+)\|#\1|/core-repo#" core-log.txt
sed -i -r "s#(.+)\|#\1|/rockon-repo#" rockon-log.txt
sed -i -r "s#(.+)\|#\1|/iso-repo#" iso-log.txt
sed -i -r "s#(.+)\|#\1|/jslibs-repo#" jslibs-log.txt

Now we combine our custom activity logs in a single “all-rockstor-repos-log.txt” thus:

cat doc-log.txt core-log.txt rockon-log.txt iso-log.txt jslibs-log.txt | sort -n > all-rockstor-repos-log.txt

And finally, at least for now, we can move back to our binary dir:

cd ~/Downloads/gource/gource-0.47/

Gource Plays Rockstor Repos

Rockstor dev at 20 days a second = 1m 37s:

(Less than fair as major contributions / contributors flash by but serves as a quick test run.)

./gource -1280x720 --seconds-per-day 0.05 --highlight-users --highlight-colour E76545 --hide filenames --hide-root --multi-sampling ../all-rockstor-repos-log.txt

And if you liked that, try the slightly more sane speed variant of:

Rockstor dev at 5 days a second = 5m 43s:

./gource -1280x720 --seconds-per-day 0.2 --highlight-users --highlight-colour E76545 --hide filenames --hide-root --multi-sampling ../all-rockstor-repos-log.txt

Note the following abstract of the interactive controls:

  • (TAB) key to switch user highlight
  • (Space bar) pause / resume the animation
  • (V) center camera on above user (if selected) – bit much on the brain
  • (F12) Screenshot
  • (Alt+Enter) Fullscreen toggle – a winner this one
  • (+-) Adjust simulation speed
  • (ESC) Quit

There’s also a nice selection of mouse interactions such as jumping to any time point; although this does reset the graph. While paused one can inspect the details of individual files and users, and dragging with the left mouse button manually controls the camera. Centre button toggles tracking / entire tree view.

Building the Bling Vid

To capture the raw video we use Gource’s “-o” switch.

Entire dev history in 1.5m (earlier video):

./gource -1280x720 --seconds-per-day 0.05 --highlight-users --highlight-colour E76545 --hide filenames --hide-root --multi-sampling ../all-rockstor-repos-log.txt -o ../rockstor-dev.ppm

We then encode to x264 via ffmpeg from Negativo17’s multimedia repo: https://negativo17.org/handbrake/ (Elrepo dependency) Thanks.

sudo dnf install ffmpeg

ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i ../rockstor-dev.ppm -vcodec libx264 -preset medium -pix_fmt yuv420p -crf 20 -threads 0 -bf 0 ../rockstor-dev.mkv

And 16.2 GB becomes 33 MB.

Entire dev history in just under 6 minutes (slightly less manic):

./gource -1280x720 --seconds-per-day 0.2 --highlight-users --highlight-colour E76545 --hide filenames --hide-root --multi-sampling ../all-rockstor-repos-log.txt -o ../rockstor-dev-long.ppm

ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i ../rockstor-dev-long.ppm -vcodec libx264 -preset medium -pix_fmt yuv420p -crf 20 -threads 0 -bf 0 ../rockstor-dev-long.mkv

57 GB via a similar ffmpeg command (differing file names) becomes 60 MB: bargain.

And finally the last 90 days at the rather more sedate 1 second/day = 1m 31s:

./gource -1280x720 --seconds-per-day 1 --start-date $(date -I --date='-90 days') --highlight-users --highlight-colour E76545 --hide filenames --hide-root --multi-sampling ../all-rockstor-repos-log.txt -o ../rockstor-dev-last-90days.ppm

ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i ../rockstor-dev-last-90days.ppm -vcodec libx264 -preset medium -pix_fmt yuv420p -crf 20 -threads 0 -bf 0 ../rockstor-dev-last-90days.mkv

Where 15 GB via ffmpeg becomes 9 MB.

Note that as with the mouse ‘time jumps’ in the interactive mode covered earlier, using the “–start-date” option, as we have just done, results in no previous activity being displayed. That is, only files changed after the given date, and their respective repositories, will appear in the resulting Gource graph.

And thus we have our thousand words.

 

Read More

Why NUT

Expectation doesn’t always match actuality, which is a shame, but sometimes we can do something about it. This is a tiny tale of my attempt to do just that.

ups examples

Early in 2015 I was on the lookout for a Linux based Network Attached Storage (NAS) system that had the more enterprise goodness I had encountered with my ventures into ZFS via FreeNAS i.e. Copy on Write (CoW), data checksums, snapshots etc., and happened across Rockstor; but at the time it had no graphical Uninterruptible Power Supply (UPS) configuration. This for me was a show stopper (story to follow), my expectation was to adopt my dream NAS setup that ran on my favourite Operating System (OS) and to have an all Linux setup once again. Actuality was falling short.

Sure I was impressed with FreeNAS but I’m a Linux guy and having been an all Linux setup for 15 years at least, it grated to be using NanoBSD for my NAS. Nothing against FreeNAS but my mainstay OS of choice was set and I had grown very familiar with it; the BSD’s for all their splendour were just not my cup of tea and it sat uneasy that my darling OS was just not serving me in every way I wanted. It had been years since I’d looked at Linux NAS ditributions so surely something had happened. It was at this point that I first came across Rockstor. This thing was neat, focused, to the point, and most encouragingly developed completely in the open. Sure it was still quite young at less than two and a half years but the pace of development was fast. That’s great I thought, if it has what I need then marvellous but alas it was short of one key feature. It had no graphical way to configure Network UPS Tools (NUT), a UPS tools system, and for me this was just not the ticket. Given that Rockstor is essentially a full CentOS with a WebUI bolted on to deal with the btrfs / NAS / sharing stuff I knew I could simply do a text based NUT config but that wasn’t what I wanted. That wasn’t an appliance. I had already spent many years doing everything with my teeth even down to the early days of mode lines in X windows and reserving memory just to get 3D going, and regular kernel recompilations or CUPS recompilations just to be able to print. My Linux should be better than that, it should by now be for the people; all the people. With all the goodness of a modern file system.

So I wondered on by. But the whole Rockstor dabble had left a lasting impression and I just couldn’t shake the fact that I was just not happy running a NanoBSD based system when my passions and interests were Linux based. It also greatly excited me that I might once again become an all Linux setup. I enjoy the Linux ethos and I believe the nature of the licence to be key: i.e. Apple’s advancements in BSD that no one else has. Alas; sometimes expectation just doesn’t match actuality.

But hold on I thought, I can’t let a little thing like a missing feature stand in my way. I began making tentative inroads into Rockstor’s community forum and the GitHub hosted code to see what I might do to make a difference. There were some unusual elements to this NAS distro, they had obviously made significant efforts towards usability, you know where things make sense due to design; an all too sparse quality in many OSs. I slowly and surely became convinced that this was an OS NAS OS (open source NAS operating system) of the future, in the making. And I wanted to be part of that future.

Oh and what Brett said.

Sometimes one just has to adapt to a situation and sometimes one can adapt the situation.

series of serial adapters

Adaptation by adapter. Not all FT232R’s are equal; a null modem wired usb to serial adapter adapted for UPS testing. Sorted.

Part of the future

At that time the forum was run on something decidedly inferior to Discourse and wouldn’t even render in my browser of choice, this wasn’t good. So I dove into the git repository and browsed the issues and pull requests to try and gain a deeper understanding. In my meandering I saw one or two tiny fixes I could make and on a whim started submitting the smallest of pull requests. Whilst exploring the feature set further I received a friendly error message that something had gone wrong. Nicely caught and no horrible crashes just a neat dialog encouraging me to send an automatically prepared zip of logs to the developers. Elegant I thought. Not just a long wait in the nothingness of an unresponsive system but true assistance and guidance. I was further impressed. So I downloaded the log zip and not wanting to bother the developers with my silly problem I took a peak. What I saw was something crafted, full on Python with exceptions caught everywhere, not just another PHP monster but a proper structured entity with objects and fancy ways and means. Anyway, the logs pretty much pinpointed the problem but I just wasn’t familiar enough with the code to tackle this myself so in the interests of living up to my ideals I posted an issue.

The response to my issue was surprisingly fast and entirely jovial; I had (in internet terms) met Suman, the project lead. A fix was committed and in no time at all my silly little issue was sorted, and this was good. The joys of a rapid development cycle.

Soon thereafter the forum software was thankfully switched to the most excellent open source discourse and was becoming alive with the growing interest in this newfangled btrfs NAS solution. I was, perhaps unwisely, awarded the privilege of forum maintainer; a pretty friendly place so no worries there. By then I had also submitted some doc tweaks and additions. To my surprise I found the [docs][10] to be programmatically generated from .rst files, a mark up format know as reStructeredText that through the magic of Sphinx ends up as html for the Rockstor home page docs section. This was just getting better and better, and I had started to make a difference. At about the same time I ventured an email declaring my interest in adding NUT and was pleasantly surprised by the encouraging return. They had no plans to implement this feature “just yet” but they were all to happy to help in whatever way they could. So why NUT; or maybe why not NUT.

 

Why not NUT

So why not NUT, why bother, well indeed; in the context of no doubt well meaning mice this is a question that answers itself. But stories are nice so here goes.

Over time where I live and work the electric had been a little iffy, the occasional trip out but nothing too serious. I had as a matter of course implemented a number of UPS’s and hand configured NUT to deal with these little outages but more was to come.

One morning I awoke to a calmness, this can’t be bad I thought, but there was a down side. The entire electric had failed and this time there was no just flipping it back on and going about my business no no. This time it was serious. It turned out that we could no longer make any sockets live; the Residual Current Device (RCD) refusing to enact our expectation. An investigation was in order.

The investigation

Most people who have dealings with computers have had mouse problems at one time or another, these days the problems tend not to be physical but electrical; given the prevalence of optical mice. It turned out that my electrical problem was both physical and mouse related.

electrical mouse problem

The mouse problem diagnosed; definitely both physical and electrical and behind a drywall at the back of a cupboard. Inconvenience incarnate.

So without ripping apart my whole (rented) accommodation I can only assume that other as yet undiscovered gems such as the above are lying in wait. What’s in your walls?

After many hours of diagnostics and much frustration I had found at least one reason for our previously flaky and now critical electric situation. Needless to say my NUT interest grew and I began as a matter of course to make all systems aware of their pending doom; or at least when it might be a good idea to shut down. NUT, due to it’s network nature, is a perfect fit.

To cut a long story short I ended up submitting (on my 47th birthday) my first non trivial pull request to Rockstor core. A fortnight of advice reviews and tweaks later and Rockstor now has graphical NUT configuration (in beta). I for one am chuffed. And yes wouldn’t it be great to have a nice UPS data page or a whizzy additional widget on Rockstor’s dashboard but until that time, successive approximation and development in the open and all, I offer up some “desktop” shiny.

Some shiny

Given no fancy technology is complete without fancy telemetry; I present walNUT.

walNUT gnome extension

walNUT the Gnome Shell Extension to monitor one’s UPS, thanks Daniele.

And for those of a KDE persuasion there is KNutClient, thanks Daniel.

Note the similarity in the names; funny that. Graphical NUT clients are also available for other popular platforms.

What are these things but what we make them.

 

 

 

 

Read More