Why I built Spectrum

2015-12-15T15:37:55.107484-06:00 2015-12-15T17:28:31.772202-06:00

Debugging Sucks.

Full stop. Sure on occasion, you feel a sense of pride and accomplishment. Figuring out the solution to that elusive Heisenbug in your multi-threaded micro-service makes you feel like SuperProgrammer. 

The majority of the time the whole process is less than ideal. 

I'm an efficiency nerd. I hate wasting my time, my team's time or our customer's time. So I spend a considerable amount of energy trying to find optimal solutions to all sorts of problems. REVSYS is well known for being at the forefront of scaling and performance of technology. Fewer people know we're also pretty damn good at process optimization. 

Obviously not creating bugs in the first place is the ideal, albeit unlikely, scenario. Since we all agree bugs are inevitable, what's the next best thing we can do to reduce the time spent bug hunting? 

Things I tried

Over my many years of experience, I've tried all sorts of things. By walking you through all of the tools and techniques I've used will help explain why Spectrum is built the way it is. 

Debuggers

Debugging tools for your language of choice aren't bad. They take a bit to get used to the specific commands, but after that, it's fairly smooth sailing. 

Except it's often hard to get your break points just so that they trip when you want them and NOT trip when you don't. Ever had to hit 'c' (for continue) 99 times in a 100 iteration loop to even get to where your bug happens? Now do that 10 times fast... Still pretty sub-optimal. 

To me, debuggers are really only useful if you are having trouble understanding the program flow. Usually, I understand the flow just fine, the bug is some bit of data being off and I can't see where it is happening.

Print statements

I'm a print statement sort of debugger. I can get a quick view of the data as we loop over it. Sure I get a barrage of 100 lines of output but I can ignore the first 99 and just look at the last one. 

Better, but not perfect. I still have to take a moment to put the print statement in. Often having to put in a few to see all the bits of data I need to inspect. And THEN I have to remove it all, so it doesn't print out random crap all the time. 

Probably only to put half of those back in again tomorrow or next week for the next bug. Wash that time over your whole team and it’s still pretty sub-optimal use of our precious time. 

Logging

So the ops/sysadmin part of my brain kicked in and I reached for logging. This is exactly what the DEBUG syslog level is for, right? 

Ops isn’t impressed with the scant logging anyway in the first few versions. So win win right? 

Sorta. Logging is the right answer, it's our developer tools for consuming these logs that are broken. Take this usual example. 

Tailing a file isn't awesome. Tailing a firehose of logging data REALLY isn't awesome. So you limit things to an INFO level. As part of your day, you end up generating an error. Looking at your logs you have a rough idea of the problem, but the crucial piece of info is only logged if you're at a DEBUG level. So you reconfigure. And then go through all the steps necessary to recreate your bug. Fix it, but then DEBUG is printing out too much noise. So... you reconfigure back to INFO. Wash. Rinse. Repeat. 

SaaS Logging Services

So why not use a fancy logging service in the cloud? These are great for ops teams and actually pretty acceptable for devs. Turn on the firehose and point it at your service. With a few seconds of delay, you can tail only the log information you want to see. 

It doesn’t seem like much of a delay, but I found it oddly distracting. Waiting for 3 to 10 seconds for the output to round trip back to me was enough time for my brain to wander off the problem. Moving to using a local instance of something like Kibana and the ELK stack can remove most of that delay if you’re looking for a DIY solution. 

However, these tool’s primary focus is not exactly our use case. Operations needs alerting, dashboards to see trends, relevance based searching inside a time frame, and, of course, archiving. 

Developers need precisely none of these features on their local machine. 

Enter Spectrum

What I really wanted was a very specific experience.  I wanted to be able to:

  • Spew a ton of logs constantly.  Turning up the verbosity to 11.
  • See the logs that are generated as quickly as possible, ideally in real time.
  • On a very fine-grained level be able to show or hide away different aspects of the logging data depending quickly and easily.
  • As a consultant, I needed to be able to do all of this without imposing my tool on the developers or operations staff I was working with.

This lead me to develop the initial prototypes of what became Spectrum. If you prefer video there is a quick demo on our homepage

Spectrum can take logs in a variety of ways and gives you the ability to filter at 3 different levels.  First we have the Stream which is the stream of logs from your app.  You can have one or more streams of these types:

  • Standard REST API
  • syslog
  • JSON over UDP
  • JSON over Websockets
  • Tailing a local file (or remote over SSH)

Each of these streams, if possible, segments the logs even further by level and sublevel.  If you’re familiar with standard syslog you can think of these as the facility and priority and when using the syslog stream type this is exactly what they are.  Here is a simple example of the JSON Spectrum might receive:

{
  "id": "7f496b8aa36e11e59b64600308902986",
  "level": "DEBUG",
  "sublevel": "app-logs",
  "message": "blah blah debugging blah",
  "timestamp": "2015-12-15 14:57:48.191647",
}

All Spectrum really needs is a unique ID, level, sublevel, timestamp and the message you want to show. However, you can add in any additional information you want and it will be displayed in an expanded view when that entry is clicked. For example, this is what my Python logging actually sent to spectrum:

{
  "filename": "demo.py",
  "function": "main",
  "id": "7f496b8aa36e11e59b64600308902986",
  "level": "DEBUG",
  "line": 17,
  "message": "blah blah debugging blah",
  "path": "demo.py",
  "process": 54549,
  "process_name": "MainProcess",
  "sublevel": "app-logs",
  "timestamp": "2015-12-15 14:57:48.191647",
  "traceback": null
}

As you can see it included basically everything the Python Logging system makes available. 

So let’s create a little example project to show you how Spectrum will be useful to you.  Let’s assume we’re building a little web app, in this case using Django, PostgreSQL, and Redis. 

We can setup a REST API stream on port 9000 and configure most of Python and Django to send any and all logs there using spectrum-python.  We can setup another REST API stream on port 9001 for our own app specific logging.  And a File stream to tail the logs our Redis server is spitting out in case we want to look at those. After I've spewed some logs to Spectrum our UI would look something like this:

 Why Spectrum - Main UI

On the left we have our Streams, Levels, and Sublevels which we can collapse if necessary.  The little red and green arrows indicate whether that stream/level/sublevel combination is currently being included, or shown, on the right or if it is being hidden. 

If we zoom in a bit we can see these controls more closely and also some of the more advanced options around colors when we right click a level:

Why Spectrum - Left Sidebar Zoom

While we're looking at screenshots, here is an example of a single entry after clicking on it to see the expanded information:

Why Spectrum - Expanded View

Daily Workflow

Now that we’ve got our logs all segmented between some of the more background noisy things and our core apps we can start working. 

While I’m working on this code Spectrum is just running in the background happily consuming the logs I’m mostly ignoring.  The moment I get confused about something or need to dive deeper is where Spectrum comes in handy.  

If I just encountered a bug for the first time, first I’ll probably show all stream/level/sublevel combinations I think might contain useful information. However, if there isn’t enough data sprinkling a few more debug level logging statements in the right places will ensure that both for this bug and all future bugs around that bit of code I have enough information to fix it.

Feature Highlights

In case you’re wondering if Spectrum has that one feature you want I’ll list out all of the bits I haven’t already highlighted.

  • Actively tails your logs as they come in. Scrolling up or hitting the space bar stops tailing so you can look at something.  Simply hit the spacebar again to resume tailing.
  • Color coding your logs by stream, level, or sublevel.
  • Clicking an individual log entry shows and expanded view.  For REST/JSON logs this shows the original JSON received by Spectrum which may contain much more information than the simple message string which is normally displayed.
  • You can choose to show or hide the date, time, level, and/or sublevels so you can focus on the data that matters to you.
  • The JSON expected by Spectrum is trivially simple.  This helps to ensure developers using any language or platform can easily integrate their logs to Spectrum.
  • Quickly show all logs from all streams.  Or can quickly show or hide all logs on a stream by stream basis.
  • You can also Solo a stream, level, or sublevel which hides all other logs except for those you want to highlight at this particular moment.
  • Hide away the left navbar of streams, levels, and sublevels so they don’t take up screen real estate when you aren’t adjusting things.
  • Quickly clear all logs between debugging iterations.
  • Setup multiple projects with different pre-defined configurations that match your different development projects so you don’t have to reconfigure when jumping between them.

Questions or Comments?

Hopefully this has given you some insight into the why of Spectrum and can see how it can drastically improve the speed at which you develop and debug code. 

If you have any suggestions for features or improvements please don’t hesitate to reach out to us. 

Signup for Early Access Beta!

If you think Spectrum is a tool you'd enjoy using we will soon be opening up an early access Beta program.  If you're interested signup for our mailing list and we'll be in touch as soon as Spectrum is ready.