Like woolly socks for your brain, you almost feel warmer inside when you stare at an information radiator.
There’s something about capturing the information that can bring a sense of calm. Even if the information paints a bad picture, at least it’s a picture you can see! You have reduced your known-unknowns.
Last year, our company made the move to Agile. The very word will conjure a variety of images for different people. For me, this sums it up:
But that aside, one of the pillars of Agile is about refining commitment and to do this you need to measure all the things. A vital part of understanding your throughput (or velocity) is having past information to hand. As a team I think it’s fair to say that our methods were a touch lo-Fi:
Yes, that is some scribblings on a whiteboard. However, it wasn’t the Art Attack nature of it that caused us to rethink; it was the fact that we have a cross-geo team. Short of setting up a permanent webcam a la the infamous Trojan Room coffee pot, our colleagues in India were not going to feel any the warmer for this information radiator.
Out with the old n’ busted; in with the new shiny.
We have now replaced Scribbles on a whiteboard with an interactive, browser-based, dashboard:
What you display is up to you, but we want to focus as a team on reducing two metrics: the number of open tickets and the number of open pull requests.
Now we have these numbers glowering at us every time we stand up from our desks on a 42″ screen hanging from the ceiling. But, and just as importantly for our cross-geo colleagues, this dashboard can be accessed from anywhere on our intranet and displayed in a browser.
It’s a major win over Scribbles since it’s also quite interactive. If you hover over a section of a graph it will break down the points for you (as you can see in the bottom part of the screenshot). It’s also set up so that you can click on one of the numbers and it will take you to JIRA or Github for the open ticket numbers and pull requests respectively.
How does it all fit together?
The infrastructure for this dashboard is pretty simple. We basically have a NUC (think Raspberry Pi) running Fedora strapped to the back of a TV which runs the following:
- Some scripts to get the numbers;
- InfluxDB to store the numbers;
- Grafana to visualise the numbers in a feature-rich dashboard;
- Firefox in fullscreen mode to show the dashboard.
InfluxDB
InfluxDB is an open-source (✓), schema-less (✓), locally-hosted (✓), time-series database. It just worked out of the box after installation. They have packages for many distributions on their download page including for Mac. For us, installation was as simple as:
{!{code class="language-sh"}!}czoyODQ6XCI8c3BhbiBjbGFzcz1cIm52XCI+JCA8L3NwYW4+d2dldCBodHRwczovL3MzLmFtYXpvbmF3cy5jb20vaW5mbHV4ZGIvaW5mbHtbJiomXX11eGRiLTAuOS40LjItMS54ODZfNjQucnBtDQo8c3BhbiBjbGFzcz1cIm52XCI+JCA8L3NwYW4+c3VkbyB5dW0gbG9jYWxpbnN0YWxsIGl7WyYqJl19bmZsdXhkYi0wLjkuNC4yLTEueDg2XzY0LnJwbQ0KPHNwYW4gY2xhc3M9XCJudlwiPiQgPC9zcGFuPnN1ZG8gY2hrY29uZmlnIGluZmx1e1smKiZdfXhkYiBvbg0KPHNwYW4gY2xhc3M9XCJudlwiPiQgPC9zcGFuPnN1ZG8gc2VydmljZSBpbmZsdXhkYiBzdGFydA0KXCI7e1smKiZdfQ=={!{/code}!}
Now we have the influxd daemon running. We can start throwing data at it. Here we create a separate database, but you could just as easily use the default one. Then we just write the data we want to store—no need for a schema, it’s more of a JFDI database:
{!{code class="language-sh"}!}czo1ODc6XCI8c3BhbiBjbGFzcz1cIm52XCI+JCA8L3NwYW4+L29wdC9pbmZsdXhkYi9pbmZsdXgNCkNvbm5lY3RlZCB0byBodHRwOi8vbHtbJiomXX1vY2FsaG9zdDo4MDg2IHZlcnNpb24gMC45LjQuMQ0KSW5mbHV4REIgc2hlbGwgMC45LjQuMQ0KJmd0OyBDUkVBVEUgZGF0YWJhc2Uge1smKiZdfTxzcGFuIGNsYXNzPVwibmJcIj50ZXN0PC9zcGFuPg0KJmd0OyBVU0UgPHNwYW4gY2xhc3M9XCJuYlwiPnRlc3Q8L3NwYW4+DQpVc2luZyBke1smKiZdfWF0YWJhc2UgPHNwYW4gY2xhc3M9XCJuYlwiPnRlc3Q8L3NwYW4+DQomZ3Q7IElOU0VSVCBjcHUsaG9zdDxzcGFuIGNsYXNzPVwib1wiPj08e1smKiZdfS9zcGFuPnNlcnZlckEscmVnaW9uPHNwYW4gY2xhc3M9XCJvXCI+PTwvc3Bhbj51c193ZXN0IDxzcGFuIGNsYXNzPVwibnZcIj52YWx1ZTwve1smKiZdfXNwYW4+PHNwYW4gY2xhc3M9XCJvXCI+PTwvc3Bhbj4wLjY0DQomZ3Q7IFNFTEVDVCAqIEZST00gY3B1DQpuYW1lOiBjcHUNCi0tLS0tLXtbJiomXX0tLS0NCjxzcGFuIGNsYXNzPVwibmJcIj50aW1lICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvc3Bhbj5ob3N0ICAgIHJlZ2lvbiB7WyYqJl19IHZhbHVlDQoyMDE1LTEwLTEzVDE0OjQ5OjE1LjUzMzU5OTgzNVogIHNlcnZlckEgdXNfd2VzdCAwLjY0DQpcIjt7WyYqJl19{!{/code}!}
It’s that simple. Queries are made made using InfluxQL which is straight-forward. InfluxDB’s Getting Started page is good place to start. They also draw up a comparison to SQL for those with an SQL background.
It also configures a web interface out of the box so you can make queries by going to localhost:8083 in your browser and, more importantly for us, it exposes an API over HTTP on port 8086 which we’ll use to record data from our scripts.
Scripts to write to the database
Getting the data you want is up to you. Being an open-source project, one of the things we wanted to record was the number of open pull requests to the repositories we maintain as a team. To do this we wrote a python script to find this information using Github’s REST API. Once we had our data that we wanted to record it was a simple HTTP POST to put the data into InfluxDB:
{!{code class="language-python"}!}czoyODE0OlwiPHNwYW4gY2xhc3M9XCJuXCI+aW5mbHV4X3VyaTwvc3Bhbj4gPHNwYW4gY2xhc3M9XCJvXCI+PTwvc3Bhbj4gPHNwYW4gY2xhe1smKiZdfXNzPVwic1wiPlwiaHR0cDovL2xvY2FsaG9zdDo4MDg2L3dyaXRlP2RiPWluZm9yYWRcIjwvc3Bhbj4NCjxzcGFuIGNsYXNzPVwiblwiPnRzdGF7WyYqJl19bXA8L3NwYW4+IDxzcGFuIGNsYXNzPVwib1wiPj08L3NwYW4+IDxzcGFuIGNsYXNzPVwibmJcIj5pbnQ8L3NwYW4+PHNwYW4gY2xhc3M9XCJwe1smKiZdfVwiPig8L3NwYW4+PHNwYW4gY2xhc3M9XCJuXCI+dGltZTwvc3Bhbj48c3BhbiBjbGFzcz1cIm9cIj4uPC9zcGFuPjxzcGFuIGNsYXNzPVwiblwie1smKiZdfT50aW1lPC9zcGFuPjxzcGFuIGNsYXNzPVwicFwiPigpKTwvc3Bhbj4gPHNwYW4gY2xhc3M9XCJvXCI+Kjwvc3Bhbj4gPHNwYW4gY2xhc3M9e1smKiZdfVwibWlcIj4xMDwvc3Bhbj48c3BhbiBjbGFzcz1cIm9cIj4qKjwvc3Bhbj48c3BhbiBjbGFzcz1cIm1pXCI+OTwvc3Bhbj4NCjxzcGFuIGNsYXN7WyYqJl19cz1cImtcIj50cnk8L3NwYW4+PHNwYW4gY2xhc3M9XCJwXCI+Ojwvc3Bhbj4NCiAgICA8c3BhbiBjbGFzcz1cImtcIj5mb3I8L3NwYW4+IDxzcHtbJiomXX1hbiBjbGFzcz1cInBcIj4oPC9zcGFuPjxzcGFuIGNsYXNzPVwiblwiPnJlcG88L3NwYW4+PHNwYW4gY2xhc3M9XCJwXCI+LDwvc3Bhbj4gPHNwe1smKiZdfWFuIGNsYXNzPVwiblwiPmNvdW50PC9zcGFuPjxzcGFuIGNsYXNzPVwicFwiPik8L3NwYW4+IDxzcGFuIGNsYXNzPVwib3dcIj5pbjwvc3Bhbj57WyYqJl19IDxzcGFuIGNsYXNzPVwiblwiPmNvdW50czwvc3Bhbj48c3BhbiBjbGFzcz1cIm9cIj4uPC9zcGFuPjxzcGFuIGNsYXNzPVwiblwiPml0ZXJpdHtbJiomXX1lbXM8L3NwYW4+PHNwYW4gY2xhc3M9XCJwXCI+KCk6PC9zcGFuPg0KICAgICAgICA8c3BhbiBjbGFzcz1cIm5cIj5kYXRhPC9zcGFuPiA8c3tbJiomXX1wYW4gY2xhc3M9XCJvXCI+PTwvc3Bhbj4gPHNwYW4gY2xhc3M9XCJzXCI+XCJvcGVuX3B1bGxfcmVxdWVzdHMscmVwbz08L3NwYW4+PHNwYW57WyYqJl19IGNsYXNzPVwic2lcIj4lczwvc3Bhbj48c3BhbiBjbGFzcz1cInNcIj4gdmFsdWU9PC9zcGFuPjxzcGFuIGNsYXNzPVwic2lcIj4lZDwvc3BhbntbJiomXX0+IDxzcGFuIGNsYXNzPVwic2lcIj4lZDwvc3Bhbj48c3BhbiBjbGFzcz1cInNcIj5cIjwvc3Bhbj4gPHNwYW4gY2xhc3M9XCJvXCI+JTwvc3BhbntbJiomXX0+IDxzcGFuIGNsYXNzPVwicFwiPig8L3NwYW4+PHNwYW4gY2xhc3M9XCJuXCI+cmVwbzwvc3Bhbj48c3BhbiBjbGFzcz1cInBcIj4sPC9zcGFue1smKiZdfT4gPHNwYW4gY2xhc3M9XCJuXCI+Y291bnQ8L3NwYW4+PHNwYW4gY2xhc3M9XCJwXCI+LDwvc3Bhbj4NCiAgICAgICAgICAgICAgICAgICAge1smKiZdfSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxzcGFuIGNsYXNzPVwiblwiPnRzdGFtcDwvc3Bhbj48c3BhbiBjbHtbJiomXX1hc3M9XCJwXCI+KTwvc3Bhbj4NCiAgICAgICAgPHNwYW4gY2xhc3M9XCJuXCI+cmVzcDwvc3Bhbj4gPHNwYW4gY2xhc3M9XCJvXCI+PTwvc3Bhe1smKiZdfW4+IDxzcGFuIGNsYXNzPVwiblwiPnJlcXVlc3RzPC9zcGFuPjxzcGFuIGNsYXNzPVwib1wiPi48L3NwYW4+PHNwYW4gY2xhc3M9XCJuXCI+cG97WyYqJl19c3Q8L3NwYW4+PHNwYW4gY2xhc3M9XCJwXCI+KDwvc3Bhbj48c3BhbiBjbGFzcz1cIm5cIj5pbmZsdXhfdXJpPC9zcGFuPjxzcGFuIGNsYXN7WyYqJl19cz1cInBcIj4sPC9zcGFuPiA8c3BhbiBjbGFzcz1cIm5cIj5kYXRhPC9zcGFuPjxzcGFuIGNsYXNzPVwib1wiPj08L3NwYW4+PHNwYW4gY2xhc3tbJiomXX1zPVwiblwiPmRhdGE8L3NwYW4+PHNwYW4gY2xhc3M9XCJwXCI+KTwvc3Bhbj4NCiAgICA8c3BhbiBjbGFzcz1cIm5cIj5kYXRhPC9zcGFuPiA8e1smKiZdfXNwYW4gY2xhc3M9XCJvXCI+PTwvc3Bhbj4gPHNwYW4gY2xhc3M9XCJzXCI+XCJ0b3RhbF9vcGVuX3B1bGxfcmVxdWVzdHMgdmFsdWU9PC9zcHtbJiomXX1hbj48c3BhbiBjbGFzcz1cInNpXCI+JWQ8L3NwYW4+PHNwYW4gY2xhc3M9XCJzXCI+XCI8L3NwYW4+IDxzcGFuIGNsYXNzPVwib1wiPiU8L3NwYXtbJiomXX1uPiA8c3BhbiBjbGFzcz1cIm5iXCI+bGVuPC9zcGFuPjxzcGFuIGNsYXNzPVwicFwiPig8L3NwYW4+PHNwYW4gY2xhc3M9XCJuXCI+cHVsbF9ye1smKiZdfWVxczwvc3Bhbj48c3BhbiBjbGFzcz1cInBcIj4pPC9zcGFuPg0KICAgIDxzcGFuIGNsYXNzPVwiblwiPnJlcXVlc3RzPC9zcGFuPjxzcGFue1smKiZdfSBjbGFzcz1cIm9cIj4uPC9zcGFuPjxzcGFuIGNsYXNzPVwiblwiPnBvc3Q8L3NwYW4+PHNwYW4gY2xhc3M9XCJwXCI+KDwvc3Bhbj48c3BhbiB7WyYqJl19Y2xhc3M9XCJuXCI+aW5mbHV4X3VyaTwvc3Bhbj48c3BhbiBjbGFzcz1cInBcIj4sPC9zcGFuPiA8c3BhbiBjbGFzcz1cIm5cIj5kYXRhPC9zcHtbJiomXX1hbj48c3BhbiBjbGFzcz1cIm9cIj49PC9zcGFuPjxzcGFuIGNsYXNzPVwiblwiPmRhdGE8L3NwYW4+PHNwYW4gY2xhc3M9XCJwXCI+KTwvc3Bhe1smKiZdfW4+DQo8c3BhbiBjbGFzcz1cImtcIj5leGNlcHQ8L3NwYW4+IDxzcGFuIGNsYXNzPVwiblwiPnJlcXVlc3RzPC9zcGFuPjxzcGFuIGNsYXNze1smKiZdfT1cIm9cIj4uPC9zcGFuPjxzcGFuIGNsYXNzPVwiblwiPmV4Y2VwdGlvbnM8L3NwYW4+PHNwYW4gY2xhc3M9XCJvXCI+Ljwvc3Bhbj48c3BhbiB7WyYqJl19Y2xhc3M9XCJuXCI+Q29ubmVjdGlvbkVycm9yPC9zcGFuPjxzcGFuIGNsYXNzPVwicFwiPjo8L3NwYW4+DQogICAgPHNwYW4gY2xhc3M9XCJue1smKiZdfVwiPnN5czwvc3Bhbj48c3BhbiBjbGFzcz1cIm9cIj4uPC9zcGFuPjxzcGFuIGNsYXNzPVwiblwiPnN0ZGVycjwvc3Bhbj48c3BhbiBjbGFzc3tbJiomXX09XCJvXCI+Ljwvc3Bhbj48c3BhbiBjbGFzcz1cIm5cIj53cml0ZTwvc3Bhbj48c3BhbiBjbGFzcz1cInBcIj4oPC9zcGFuPjxzcGFuIGNsYXNze1smKiZdfT1cInNcIj5cImVycm9yOiBDb25uZWN0aW9uIHRvIGxvY2FsIGluZmx1eGRiIGZhaWxlZFwiPC9zcGFuPjxzcGFuIGNsYXNzPVwicFwiPik8L3N7WyYqJl19cGFuPg0KICAgIDxzcGFuIGNsYXNzPVwiblwiPnN5czwvc3Bhbj48c3BhbiBjbGFzcz1cIm9cIj4uPC9zcGFuPjxzcGFuIGNsYXNzPVwiblwiPntbJiomXX1leGl0PC9zcGFuPjxzcGFuIGNsYXNzPVwicFwiPig8L3NwYW4+PHNwYW4gY2xhc3M9XCJtaVwiPjU8L3NwYW4+PHNwYW4gY2xhc3M9XCJwXCI+e1smKiZdfSk8L3NwYW4+DQpcIjt7WyYqJl19{!{/code}!}
Configuring Grafana
Once Grafana is installed, there isn’t much configuration to do. By default it binds to port 3000 but this can be changed either by changing the http_port option in /etc/grafana/grafana.ini or by forwarding all requests on port 80 to port 3000:
{!{code class="language-sh"}!}czoxMDQ6XCJzdWRvIGlwdGFibGVzIC10IG5hdCAtQSBQUkVST1VUSU5HIC1wIHRjcCAtLWRwb3J0IDxzcGFuIGNsYXNzPVwibVwiPjgwPHtbJiomXX0vc3Bhbj4gLWogUkVESVJFQ1QgLS10by1wb3J0IDMwMDANClwiO3tbJiomXX0={!{/code}!}
Grafana supports authenticated access and is configured with basic auth out of the box. It also has support for LDAP but we wanted to disable all authentication since it is only running internally. To do this, just un-comment the option in the config file:
{!{code class="language-diff"}!}czoxMDk6XCJbYXV0aC5hbm9ueW1vdXNdDQojIGVuYWJsZSBhbm9ueW1vdXMgYWNjZXNzDQo7PHNwYW4gY2xhc3M9XCJnZFwiPmVuYWJsZXtbJiomXX1kID0gdHJ1ZSAgJmx0Oy0tIHJlbW92ZSB0aGUgXCc7XCc8L3NwYW4+DQpcIjt7WyYqJl19{!{/code}!}
Next you’ll need to add the local InfluxDB as a datasource from which you can query in Grafana:
Then it’s just a question of adding graphs using Grafana’s web interface. It echos the query language that you’d use in directly with InfluxDB:
And there you go… “Job’s a good’n” as they say:
