Playing with MongoDB in LA

January 5, 2012 By Leave a Comment

When I first saw NoSQL stores a few years ago I wasn’t that impressed.  They were nice and really fast if you had a bucket load of ram, but it seemed to be a solution to a problem I didn’t have.

My biggest qualm was initially the “eventually consistent” data, and the absolute requirement to have multiple nodes.  I have grown to like the ”eventually consistent” model because 75% of what I build are CMS or CRUD type apps.  With these type of apps you are reading data most of the time and only occasionally writing to the store.

But, for the most part good old MySQL serves as the warhorse as it has with me for the last 12 years (wow, it’s been that long?!).  It does the job, and doesn’t take a lot of admin to keep things running smoothly.

MongoDB – The new cool?

But recently I began to hear about MongoDB, and I’ve really taken a shine to it.  Essentially it is a NoSQL datastore, but stores data as binary JSON (BSON) but seems to straddle the divide between traditional RDBMS & NoSQL stores.  It also has full index support, journaling, really nice support for replication and sharding and pretty much all the things you’d expect of a RDBMS but in a document store with a number of extra bells and whistles attached.

There also seems to be a lot of energy in the project, and it feels like MySQL did when it was the scrappy newcomer (but Mongo has a *much* better website!).

Schema-me-this, schema-me-that

But the one thing I can’t quite understand yet is how to model a document store schema. Document stores can be multi-dimensional, which means that you can store arrays, structs and all manner of data nested within a single document record.  This can be very useful, but traditional normalisation rules do not apply.

I’ve had CODDs normalisation drummed into my brain so much it is just nature to me now; I go all the way to 5NF ’cos I’m hard. ;)  Changing you mindset after 15+ years of BCNF dogma is a tricky thing, and I think it requires the kind of mind-shift that I had to go through when learning OOP having only used functional languages before.

The problem is that there I have not uncovered as yet a wealth of information on what the best way to design these things and as such I’ve not taken the plunge yet with MongoDB as the backbone in a new project.

But yay, enlightenment is a mere 30mins drive away @ LAX

But, the knowledge gods have smiled on me as there is a MongoDB 1day conference in LA on the 19th Jan with a track on just the thing I am craving info on.  I will dutifully report what I learn should I remember anything after we inevitably hit the bar (there will be no repeat of Scotch on the Rocks and the Jagermeister incident).

Watch this space…

Filed Under: MongoDB, Node.js, Oooo shiny!, Ramblings Tagged With: , , , ,

The Quest

December 16, 2011 By 1 Comment

I’ve spoken a few times about my quest for a simpler toolchain, but to keep current with new technologies at the same time.  This is not language face-off, but a reflection on where the industry is headed and the new problems we need to solve in the near future.

ColdFusion, my good friend

I’ve been working with ColdFusion (CFML) as my primary development platform now for about 10 years, and I’ve helped to build many successful sites/engines using CFML like Meno Yearbooks & Raileasy.

I also purposely say CFML now and not ColdFusion since the offering from Railo is frankly just amazing and I now find myself biased towards using Railo above ColdFusion when choosing a platform lately.  I also find myself wondering if ColdFusion will become “Apache ColdFusion” in a few years time as awaits of Flex (for better or worse).

And of the alternatives?

I tried to get into Ruby on Rails, but I just couldn’t.  I really liked the idea, but really disliked the syntax.  Python was great, but just like Ruby it didn’t really solve any problems that I couldn’t solve at least or almost as fast in ColdFusion.  I also had the displeasure recently of playing with Perl and Catalyst, which instantly reminded me why I stopped using Perl back in ’97.

However this week I have stumbled upon Node.js, and am having a great time with it.  It is still young I grant you, but there is something very special about the problems that Node can solve that holds great promise.  Just like Rails finally brought Ruby into the limelight, I think that Node is upon the same cusp (I played with Ruby with a colleague @ Tiscali back in 2001 & and it was novel then, but “had a funny syntax”).

Application servers are fat

This brings me to the thing that has really been bugging me the most about application servers like ColdFusion over this last year.  They were great for what they did and solved massive problems that we faced in web development around 2000.  We needed to build mostly table based, functional websites without being able to assume Javascript would be 100% available.  We had the browser wars where we had to design cross platform and the server did the bulk of the markup.

But fast forward to today and we have a different environment.  Everyone is toeing the HTML5/CSS3/Javascript line, which Adobe conceded in such spectacular fashion recently.  Hell, even Microsoft have thrown in the towel on Silverlight.  The arms race now is about the best HTML5 support and fastest Javascript engine, it isn’t about markup any longer it is mainly about pumping data around.

It’s all about the data baby

ColdFusion can quite competently act as a middleware for data centric apps, but if we are honest it’s main forte is building HTML sites.  But the fact is that we are moving more and more towards a point where the app server sends a initial large HTML payload and then thereafter sending smaller JSON/AJAX packets around or loading partials directly into DIVs with JQuery and the like.

ColdFusion is great, but I am thinking more and more lately that it is totally overkill for what I am trying to do.  That is to build optimized fast mobile applications that suck lots of small amounts of information down, be that JSON or HTML.

To enable the rich experiences that our clients are demanding now we will inevitably (especially on the mobile side) go down the thin client type approach rather than HTML heavy payloads.

I think that Node.js will fit the bill of being a light and nimble middleware for data.
ColdFusion was the middleware that got the data from the database, rendered HTML and threw it at the browser.  Node just throws the data, the browser can do the rest.

Yay, another silver bullet

Well of course it goes without saying that Node isn’t suitable for everything, but I do think that it helps solve a problem that we will encounter more and more in our professional lives.

Filed Under: Adobe, Coldfusion, Flex, HTML5, Javascript, Mobile, Node.js, Railo, Ramblings Tagged With: , , , , , , , , , , ,

Playing with Node.js – the basics…

December 15, 2011 By Leave a Comment

Today has been fun learning Node and finding out how powerful this thing really is. Here are a few simple snippets to illustrate the power of Node:

Simple web server

var http = require('http');
http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
}).listen(3000, "127.0.0.1");
console.log('Server running at http://127.0.0.1:3000/');

That can’t be it surely? Oh yes it is, you now have a very simple webserver sitting on port 3000 just waiting to say hello.

Phhff, that’s just simple and isn’t that impressive

OK, but all that does is say “Hello World” to every request, and that’s not very much use. What if it does something useful like serving static files, essentially a very basic webserver.  That will make it much more complicated since we need to use the connect webserver module, and the huge code for that is:

var connect = require('connect');
var server = connect.createServer(
	connect.logger(),
	connect.static(__dirname + '/html')
).listen(3000);
console.log('Connect server running at http://127.0.0.1:3000/');

That is all that is needed to serve everything in the “html” directory, and log all connections in an Apache log format to the console. The connect module has support for cookies, vhosts, session management and much more.

OK, that’s kinda impressive but is that it?

That’s just the tip of the iceberg, what about a chat server? That must be much harder broadcasting to all the clients and all that. Well it is harder, but only 29 lines harder.

var net = require('net'),
    carrier = require('carrier'),
	connections = [];

var server = net.createServer(function(conn) {
	var username;
	connections.push(conn);

	conn.write('Welcome to the Odyssey Of Mayhem chatroom!\n\n');
	conn.write('Please type your name:');

	conn.on('close', function(){
		var pos = connections.indexOf(conn);
		if(pos >= 0) connections.splice(pos, 1);
	});

	carrier.carry(conn, function(line) {
    	if(!username){
			username = line;
			conn.write('Hello '+username+'! There are '+(connections.length-1)+' other people on the server\n');
			return;
		}

		if(line == 'quit'){
			conn.end();
			return;
		}

		var feedback = username + ': ' + line + '\n';
		connections.forEach(function(one_connection){
			one_connection.write(feedback);
		});
	});
}).listen(3000);
console.log('Simple chat server up on port 3000.');

There you have a simple chat server that remembers your name, and broadcasts your inner thoughts to whoever else dares fire up a telnet client! What was that, “pics or it didn’t happen”?  OK, here you go…

3 terminal windows. 1 for Node, and 2 for chat clients.

Oh, I’m going to have soooooo much fun in the coming weeks. :D

Filed Under: Javascript, Node.js, Oooo shiny! Tagged With: , , , ,

Some Node.js resources

December 15, 2011 By Leave a Comment

I’ve been playing with Node for the last 24 hours and I absolutely love it.  I haven’t been this into a language since I got sucked into Pike almost 10 years ago when I played with Roxen.

Here are some resources that I’ve been perusing:

I’ll be writing some posts detailing some of the cool things that you can do very easily in Node.

Filed Under: Javascript, Node.js, Ramblings Tagged With: , ,

Watch out Lynda.com, here comes the jungle

December 15, 2011 By Leave a Comment

In my quest to devour knowledge about Node I turned to the old faithful, Safari bookshelf.

I’ve not been there in a while, and really I have it so I don’t have to cart my huge collection of O’Reilly tomes around with me.  But to my pleasant surprise O’Reilly are now doing videos too.

Have a watch of this, it is a great introduction to Node and something that Lynda and others should keep an eye on.

Tom Hughes-Croucher on Node:

http://my.safaribooksonline.com/video/programming/javascript/9781449307189

Filed Under: Node.js, Ramblings Tagged With: , ,

Damn you Mr Mclean, damn yoooooouuuuu!

December 14, 2011 By Leave a Comment

The job is on

It all started innocently enough when my good friend and partner in crime Gordon McLean sent me a question:

Do you want to see if you can get an iPhone to send an SMS from a Mac/PC.
Essentially I am looking for a SMS service that can be triggered from
Servoy that uses a standard phone ideally one unlimited free texts via
AT commands.
Dont spend frigging hours on it mind, there is virtually no budget.

Servoy for those who don’t know is a Java based RAD tool to produce desktop/web apps very much in the same way as Filemaker Pro, Visual FoxPro do.  It’s a great tool to build desktop business app quickly.

The Options

So, I wondered off in search of a solution and since the budget was “as tight as a ducks backside”, a KISS approach was called for.  A short while later I fired off a reply saying suggesting the following solutions:
  • Purchase the Prolific Axis serial port plugin for Servoy, and hope it works.
    Costs almost the whole budget however.
  • Write our Servoy plugin utilising the standard Java serial communications API
  • Write a script that will sit along side servoy (but must be installed separately from Servoy).
    This can be in a number of languages like Perl, Python, Ruby or Javascript.

We agreed that the fastest and cheapest way was to build a proxy app that would accept HTTP REST calls, and then communicate with a phone over a serial port.

The quest for a simpler, more productive life

Since I’m in a push to reduce complexity and the number of technologies I finally suggested that we use Node.JS since I dislike Perl, haven’t every got around to working with Python and I’m just not getting in to Ruby that much.  My logic being that otherwise the choices are Perl, Python or Ruby and all use different syntax to Javascript; since we know Javascript, why learn another language?  The response was susinct:

F@*k me, I never thought I would hear you say “use a known technology” ;)
Are you feeling OK !?!

You see, I like shiny and new things – that is why there is an “Ooooo shiny” category to my blog.  I normally learn a few new technologies a year just to keep up to date and have a play.  The bleeding edge is where the cool toys are. ;)

There be gold in them there hills

But, you see despite myself seeking safe ground this time, it would seem that I have uncovered shiny gold!  The more I read about Node, the more I liked and now I am totally smitten.

One key difference is that Node uses an event-loop rather than the norm which is threads.  The problem though is that you need to look at the problem differently since the whole thing is event driven and non-blocking.

Traditional appservers will do an operation, block (wait for IO from database) and then continue with the code.  But while the server is waiting, that process can not do anything else other than wait.  This is why app servers have threads so that there are multiple worker-threads waiting for incoming connections and hopefully no one is kept waiting (assuming we don’t run out of threads).

Upon further reflection it occurred to me that with most of the applications that I’ve worked on over the last decade very rarely was CPU load the major issue.  Most of the time it was either IO (database or disk) or low memory issues.  Indeed the main cause of high CPU loads was when the Java garbage collector “goes nuts” when memory is very low.

So it seems that my inner explorer and lover of shiny things has thwarted my quest to reduce the number of new languages/platforms that I play with.  Oh well, have to go and play with the new toy now. :)

Filed Under: HTML5, Javascript, Node.js, Oooo shiny! Tagged With: , , , , , ,
Follow

Get every new post delivered to your Inbox.

Join 143 other followers