July 29, 2011

Oh The Joy!

I JUST got my new toys in!
check it out:

the BUB2, a jeelink and a jeenode! Yiiiipiiiie! can't wait!
I'll let you know what I do with it.

July 28, 2011

How SMALL can you get?

As part of a project of my own, I have to fit my server (UnboundID Directory Server to be precise) on a small machine. How small? Well that's exactly the question I'm trying to get an answer to so I can figure out what to buy for that particular project. Options are many, from fitPC to trimslice and atom-based netbooks, but the question is how small can I make the server and still be able to have meaningful use of it. The experiment needs to be able to store data from 14 sensors every 16ms. This is a total throughput of inbound data of 875 writes per second. More importantly than the sheer throughput -which will be coming from an external microcontroller in bursts- the server needs to have small response time in order to be able to process all the writes before the next burst in order to keep up with the constant flow of sensor data.

Fiddling with it a bit today I managed to fit the server in a 48MB footprint on a test laptop -which is one of the candidates as it makes my life easier on the battery side of things, provided it will be able to sustain the cold temperatures, more on that in a later post- and I am happy to report that I could comfortably achieve over 1,600 writes per second. Note that while I was doing these test, not only was I running DS in a very small heap, but the laptop is also running the UnboundID Synchronization Server in an equally small JVM and order to investigate the performance, I'm also running netbeans, jvisualvm and some other tools (Apache Derby, FirebirdSQL and JEdit to be precise). So you can see we're not nearly in a realistic environment but it does mean that there is a lot MORE stress at the moment on the machine, which is good.
So 1,600 writes per second on this extremely small and contrived setup is good, right? Not quite, the response times are hovering at 1.2ms, which is too much for the application I need DS for. So I throttled it down a bit to see what I could get without pushing the server to the limit, here's what I got :
Column 1: recent throughput (writes per second)
Column 2: recent response time (in millisecond)
Column 3: recent error rate
Column 4: average throughput (writes per second)
Column 5: average response time (in millisecond)

So there you go, ~0.8ms write response time on an old laptop that looks like a good candidate.

I pushed even further the experiment by attaching the sensor board and looking at the REAL traffic going in to the DS and the response times further fell to around ~0.7 which gives me a 0.3ms window of headroom.

Great, I might be able to do some extra crunching "in flight".

July 26, 2011

Engineering: It's an addiction...

I have sort of an addiction to "doing" stuff, solving problems in particular is a thing that I really get a kick out of. It was bug that I got a long time when my grand-father would (try to) teach me how to correctly bend wood beams to make the structure of wood boats. I found it fascinating as it takes a bit of understanding of the basic "fabric" quality of wood and how the fibers are disposed in the beam to bend, what temperature it requires and such. An absolutely fascinating art.
Later, my college buddy Matthieu Bourgeois totally infected me with the chess bug. I never quite got even nearly good at it but I did read a lot of books he recommended and found the mechanics, tactics and even philosophies behind it truly mesmerizing. 
More recently, I had the privilege to work a little bit with Kohsuke Kawaguchi (of Hudson and now Jenkins fame) who is an engineer in a league of his own. He decided to create an eXtreme Feedback Device for his Hudson builds and I soon followed suit with an XFD of my own. That introduced me to 2 things:
  - the USB bit Whacker which is an ingenious and cheap little device making it easy to go from the logical world of software engineering to the "magical" world of physical computing.
  - sparkfun.com where he had sourced the bit whacker. And THAT was the beginning of an eye opening journey for me into the world of electronics which had to that point seemed to be way over my head.

Being such a hacker by nature, I soon started to abuse the bit whacker to do other things than simple XFD. I created a "weather station" for OpenDS, the project I was officially working on at the time. I will try to post some pictures and description of this weather station as my original blog just got wiped by Oracle (nice,btw). 

I then got so into it that I bought a couple of picaxes because it's a microcontroller that's easy to start with AND it's roughly $8 with taxes, so no harm if I can't wrap my head around making it work. With the first picaxe, I made a water game for my kids that sprays water from underneath my deck.There are four rows of sprayer heads, 4 heads per zone. I'll post a video of it running tomorrow. All that to say that this was really the start of a great journey.
I soon wanted to do more advanced stuff and looked into arduino. The only issue is that wireless is a pain, most options being way too expensive for the occasional DIYer.

Enters JeeLabs. Once again I run into an awe inspiring engineer, driven and getting stuff. I have been addicted to his daily weblog where he shares his experience with various projects he's working on or goes about explaining the basics of electronics in his "easy electrons" series, which in itself is worth the read for the software engineer I am. We're so "removed" from the physical world that it's really helpful, if you're considering starting an electronics project, I recommend to check out his easy electrons blog posts. 
Anyway, what's amazing is not WHAT he does but HOW he goes about it, trying to actually understand what's going on. He doesn't claim to know everything and admits when he doesn't, and you can walk the path with him to get to the point that it all makes sense. It's all about the attitude!
That prompted a purchase from moderndevice to source his electronics goodness and get crackin' with the next project.
Props JCW!
Keep posting, I'd be a total downer if you stop ...

July 25, 2011

Strap in, the "lab" is ramping up

You may not have seen the announcement (at Cloud Identity Summit) but there is a new "department" at UnboundID dubbed UnboundID labs and you can find it here. It may have been somewhat quiet, and there is but one project in there for now, SCIM (Simple Cloud Identity Management) but that doesn't mean we haven't some other stuff on the shelves that are ready to be given some visibility so stay tuned, and in the meantime, check out SCIM, which is probably the fastest a consortium (or a technical group or committee or whatever you want to call that) has ever gone from agreeing on a spec to putting out a reference implementation, all thanks to the awe inspiring engineers who build it.

July 22, 2011

Audit your environment in 10 seconds

So you got servers up and running, how do you make sure their configurations are in sync ?
The Meat
Simple, we provide a tool called ldap-diff tool that allows to compare two trees and it can be used to compare two servers' configurations.
For example:

$ldap-diff --outputLDIF sourcetotargetdiff.ldif --baseDN cn=config --sourceBindDN "cn=directory manager" --sourceBindPassword admin123 --sourcePort 1389 --sourcehost sourceDSIP --targetPort 1389 --targetHost targetDSIP --targetBindDN "cn=directory manager" --targetBindPassword admin123 --searchFilter '(!(objectclass=*replication*))' --numPasses 1 "^userPassword" "+" "*" "^modifyTimestamp" "^modifiersName" "^ds-entry-checksum" "^ds-update-time" "^ds-create-time" "^ds-entry-unique-id" "^creatorsName" "^createTimestamp" "^entryUUID" "^ds-cfg-password" "^pwdChangedTime"

Put that in a loop to iterate across your servers, that's it!