We should've stopped--but let's keep going and see what happens.

Thoughts on Streaming Signal Services, Pt. 1

So, let’s think about the next interesting form of data processing for a bit.

We’ve done RDBMSes to death; lots of very clever people have done some excellent work to make it easy to query related data. NoSQL exists for varying degrees of easily-horizontally scalable data, and also for certain types of exotic data query (think something like Neo4J and graph DBs).

So, we’re going to be dealing with a lot of real-time events and data, right? All the loud money is toting “Internet of Things” and whatnot, massive sensor networks creating lots and lots of bytes of data and streaming it in to some system to digest and analyze.

These data streams can be visualized as flowing from a sensor, to a recorder, to a transform network, to a display device. There’s sort of two workflows you want to accomplish with this: real-time transformation and visualization, and also batch processing rifling through all the available data for trends.

A common problem is also that you’ve got a lot of cases where the same conceptual data source is moving across several data sensors—the usual mistake (at least, it’s a mistake in my opinion) is that people tend to lump the handling of the “what data source generated this data” along with the “this sensor generated this data, from some source” problems.

Really, you just want a very (simple!) service to pull records corresponding to one or more sensors, and then-a separate service for saying “Oh, this data source? Yeah, was tracked by thus-and-such sensors at the requested time…go bug them”.

First Impressions on Octopress

So, I’m starting over and going to try actually blogging—goal is a simple once-a-week, and if the chain builds, maybe we’ll see what happens.

First thing: making a new post is annoying: rake new_post is cool and all, but you really need to modify the Rakefile to automatically fire up a text editor once it’s been run, like so:

# at the end of writing the file in the new_post task:
Kernel.system('vim', filename)

Et voila, a text editor appears once you make a new post!

Next thing to worry about is deployment—the smart thing to do would be throwing this up on Github pages or Cloudflare, but let’s be honest: that’s not really my style.

Sooooo, I think the answer is a dumb little cronjob to pull the repo every half day and update.

Or, y’know, just manually update the damned thing until I’ve been hit with some divine inspiration.


Derp derp derp I should probably put a bit to commit to Github after I’m done writing—now wouldn’t that be something.


This is the first post on the new Octopress.