gource-graph-sept-2017

Rockstor Git Gource – project visualization

Gource is Git shiny.

Gource is an open source GPLv3 licensed OpenGL visualiser of Git/SVN/Mecurial, and Bazaar repositories over time and so serves well to indicate a project’s growth / activity.

gource git snapshot showing many names
Family Code Feeding

Wikipedia’s “A picture is worth a thousand words” page has the following:

The actual Chinese expression “Hearing something a hundred times isn’t better than seeing it once” (, p bǎi wén bù rú yī jiàn) is sometimes introduced as an equivalent, as Watts‘s “One showing is worth a hundred sayings”.[6] This was published as early as 1966 discussing persuasion and selling in a book on engineering design.[7]

Rockstor’s 5 git repositories visualised in just over 1.5 minutes.

The essence of Gource’s use, at least on a linux desktop, is to initially build from the code and then point the resulting binary at a copy of one’s repository. The wrinkle here is that Rockstor, akin to many projects, consists of multiple repositories:

  • rockstor-doc – our docs in their original restructured text format.
  • rockstor-core – the main code repository.
  • rockon-registry – our docker based ‘rock-on’ plugins repo.
  • rockstor-iso – code / artifacts concerning the creation of our stable release iso images.
  • rockstor-jslibs – the collection of js libraries we use to do what js does best.

But fear not as the now 29 strong developers of Gource have us covered in their appropriately named: Visualizing-Multiple-Repositories.

“Sometimes it may be interesting to show the history of multiple projects in the same Gource animation.”

Thanks people.

This mini HowTo is essentially a re-telling of that page as applied to Rockstor’s 5 repositories. The hope is that this ‘telling’ might aid other multi-repo projects and save us all some time; at least on mass.

Building Gource

As of 8th September 2017 v0.47 was released which is the version I used here. Be sure to visit their releases page to check on availability of any newer releases. As always it’s best to resource the original text on install matters: please favour the projects own INSTALL doc over what I state here. But by way of completeness I’ll indicate how it worked for me:

Using a Fedora26 desktop:

sudo dnf install git freetype-devel pcre-devel glew-devel SDL2-devel SDL2_image-devel boost-filesystem boost-devel glm-devel
mkdir ~/Downloads/gource
cd ~/Downloads/gource
wget https://github.com/acaudwell/Gource/releases/download/gource-0.47/gource-0.47.tar.gz
tar xvf gource-0.47.tar.gz
cd ~/Downloads/gource/gource-0.47
./configure
make

We refrain from ‘make install’ to keep all our gource activities confined to our Downloads dir. (snap / flatpak suggestions anyone?)

Now we jump up a directory and grab a local copy of Rockstor’s repos:

cd ~/Downloads/gource/
git clone https://github.com/rockstor/rockstor-core.git
git clone https://github.com/rockstor/rockstor-doc.git
git clone https://github.com/rockstor/rockon-registry.git
git clone https://github.com/rockstor/rockstor-jslibs.git
git clone https://github.com/rockstor/rockstor-iso.git

then we resource Gource’s ability to create a custom-log of activity spanning all the above repos. For this we jump back into our gource build directory:

cd ~/Downloads/gource/gource-0.47

and point the binary at each of the rockstor repos in turn with switches requesting our needed custom logs.

./gource --output-custom-log ../doc-log.txt ../rockstor-doc/
./gource --output-custom-log ../core-log.txt ../rockstor-core/
./gource --output-custom-log ../rockon-log.txt ../rockon-registry/
./gource --output-custom-log ../iso-log.txt ../rockstor-iso/
./gource --output-custom-log ../jslibs-log.txt ../rockstor-jslibs/

Then back up to our new logs:

cd ~/Downloads/gource/

Next we make our repos more distinct by putting them on separate branches via multiple stream edits (sed):

sed -i -r "s#(.+)\|#\1|/doc-repo#" doc-log.txt
sed -i -r "s#(.+)\|#\1|/core-repo#" core-log.txt
sed -i -r "s#(.+)\|#\1|/rockon-repo#" rockon-log.txt
sed -i -r "s#(.+)\|#\1|/iso-repo#" iso-log.txt
sed -i -r "s#(.+)\|#\1|/jslibs-repo#" jslibs-log.txt

Now we combine our custom activity logs in a single “all-rockstor-repos-log.txt” thus:

cat doc-log.txt core-log.txt rockon-log.txt iso-log.txt jslibs-log.txt | sort -n > all-rockstor-repos-log.txt

And finally, at least for now, we can move back to our binary dir:

cd ~/Downloads/gource/gource-0.47/

Gource Plays Rockstor Repos

Rockstor dev at 20 days a second = 1m 37s:

(Less than fair as major contributions / contributors flash by but serves as a quick test run.)

./gource -1280x720 --seconds-per-day 0.05 --highlight-users --highlight-colour E76545 --hide filenames --hide-root --multi-sampling ../all-rockstor-repos-log.txt

And if you liked that, try the slightly more sane speed variant of:

Rockstor dev at 5 days a second = 5m 43s:

./gource -1280x720 --seconds-per-day 0.2 --highlight-users --highlight-colour E76545 --hide filenames --hide-root --multi-sampling ../all-rockstor-repos-log.txt

Note the following abstract of the interactive controls:

  • (TAB) key to switch user highlight
  • (Space bar) pause / resume the animation
  • (V) center camera on above user (if selected) – bit much on the brain
  • (F12) Screenshot
  • (Alt+Enter) Fullscreen toggle – a winner this one
  • (+-) Adjust simulation speed
  • (ESC) Quit

There’s also a nice selection of mouse interactions such as jumping to any time point; although this does reset the graph. While paused one can inspect the details of individual files and users, and dragging with the left mouse button manually controls the camera. Centre button toggles tracking / entire tree view.

Building the Bling Vid

To capture the raw video we use Gource’s “-o” switch.

Entire dev history in 1.5m (earlier video):

./gource -1280x720 --seconds-per-day 0.05 --highlight-users --highlight-colour E76545 --hide filenames --hide-root --multi-sampling ../all-rockstor-repos-log.txt -o ../rockstor-dev.ppm

We then encode to x264 via ffmpeg from Negativo17’s multimedia repo: https://negativo17.org/handbrake/ (Elrepo dependency) Thanks.

sudo dnf install ffmpeg

ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i ../rockstor-dev.ppm -vcodec libx264 -preset medium -pix_fmt yuv420p -crf 20 -threads 0 -bf 0 ../rockstor-dev.mkv

And 16.2 GB becomes 33 MB.

Entire dev history in just under 6 minutes (slightly less manic):

./gource -1280x720 --seconds-per-day 0.2 --highlight-users --highlight-colour E76545 --hide filenames --hide-root --multi-sampling ../all-rockstor-repos-log.txt -o ../rockstor-dev-long.ppm

ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i ../rockstor-dev-long.ppm -vcodec libx264 -preset medium -pix_fmt yuv420p -crf 20 -threads 0 -bf 0 ../rockstor-dev-long.mkv

57 GB via a similar ffmpeg command (differing file names) becomes 60 MB: bargain.

And finally the last 90 days at the rather more sedate 1 second/day = 1m 31s:

./gource -1280x720 --seconds-per-day 1 --start-date $(date -I --date='-90 days') --highlight-users --highlight-colour E76545 --hide filenames --hide-root --multi-sampling ../all-rockstor-repos-log.txt -o ../rockstor-dev-last-90days.ppm

ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i ../rockstor-dev-last-90days.ppm -vcodec libx264 -preset medium -pix_fmt yuv420p -crf 20 -threads 0 -bf 0 ../rockstor-dev-last-90days.mkv

Where 15 GB via ffmpeg becomes 9 MB.

Note that as with the mouse ‘time jumps’ in the interactive mode covered earlier, using the “–start-date” option, as we have just done, results in no previous activity being displayed. That is, only files changed after the given date, and their respective repositories, will appear in the resulting Gource graph.

And thus we have our thousand words.

 

Read More

3.8-1 update. BTRFS Data import, virtio support and more testing

Rockstor 3.8-1 update is now available. Support has been added for a much anticipated and important feature to automatically import BTRFS data(Pools, Shares and Snapshots) on drives from a different Rockstor machine or a previous install. More functional tests have been added and KVM support is enhanced by fixing the issue with disk scan and Virtio.

This is the list of changes in this release:

1. Added support to Auto import BTRFS data from previous install or a different Rockstor system. Issue #534

2. Added more functional tests and improved test coverage. Issue #676

3. Enhanced disk scan logic to better support KVM. Special thanks to Phil Guyton for this. Issue #673

As always, update your Rockstor installation via web-ui. If you are new to Rockstor, watch the short video that demonstrates this process.

Read More

8TB Rockstor DIY NAS

chenbro_all_partschenbro_openedchenbro_closed

I’ve built several Rockstor NAS servers since I’ve been developing Rockstor. Rockstor 3 was released in early september of this year and we are glad to see more users every day since. All the while, my Rockstor NAS boxes on old dell and hp boxes with different amounts of accumulated dust are chugging along.

They work great, one of them even powers our live demo.

It’s time to build something new primarily to give our super-home and small business users a reference when building their own Rockstor NAS box. After researching various motherboard and cpu options, I’ve decided to keep things simple. The goal is not to build the cheapest system, but a decent one with parts that have credibility in the DIY NAS community. So I based my build on this technutz post which also gives us the opportunity to compare Rockstor with FreeNAS, the OS choice in that article.

Here is the list of parts.

Motherboard+CPU combo: ECS Elitegroup NM70-I Celeron 1037
[EDIT: the above motherboard is discontinued. Here’s the nextgen version of it. NM70-I2]
Memory: Crucial 8GB Single DDR3 1333 MT/s
Case+PSU: Chenbro SR30169 Tower Case SR30169T2-250
4 x 2TB NAS hard drives: WD Red 2 TB NAS Hard Drive
16GB USB stick: SanDisk Cruzer 16GB USB 2.0 Flash Drive

Note about RAM: The motherboard takes 1333Mz DDR3 RAM, so make sure you get the right speed. It failed to recognize a faster(1600 MT/s) SO-DIMM. With the correct RAM, it took about 30 minutes to assemble the box and it was ready for the Rockstor install.

BIOS notes: I went over every screen in the BIOS and changed a couple of settings. The important one is to enable AHCI mode in SATA configuration which lets you hotswap drives on the running system.

Rockstor requires one full disk drive for the operating system. So the choice is either to use one of the SATA drives or boot from a USB stick. There are mixed opinions about running Linux from USB. There two clear disadvantages: (1) OS runs slowly from USB compared to SATA (2) USB sticks incrementally deteriorate or suddenly crash one day. I’d like to experiment and find out how it works and in the worst case scenario, I’ll just reinstall Rockstor on a new USB stick. My actual data on the SATA drives will still be untouched.

I plugged the network cable, power chord, keyboard, mouse, external cdrom and started Rockstor install. It was a bit slow installing to the USB, but no other problems were encountered.

Note about USB bootup: I noticed that sometimes, upon reboot the system doesn’t recognize the usb stick. Just remove it, plug it back in and reboot. This worked every time I saw this problem. Also, reboots are rare and this did not happen on every reboot. so, small pain for extra hard drive space.

Rockstor has been running great on this little box. It’s the smallest and quietest 8TB Rockstor NAS box I know. As part of Rockstor testing, I am sure to stress test this box, gather performance metrics etc.. I’ll share my findings in future posts. Please submit your comments and feel free to email us: support@rockstor.com

Read More